Method, apparatus and electronic equipment for exporting time-series data based on time range partitioning

By adopting a time-range partitioning-based time-series data export method, the performance bottlenecks and resource conflicts in the time-series data export process of enterprise-level information systems have been resolved, improving export efficiency and accuracy, optimizing resource utilization, and reducing operation and maintenance costs.

CN123115137APending Publication Date: 2026-11-17CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611299165.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies suffer from performance bottlenecks, resource conflicts, and architectural limitations in the process of exporting time-series data from enterprise-level information systems, leading to deteriorating query response times, resource contention, and high operation and maintenance costs.

Method used

A time-series data export method based on time range partitioning is adopted. By receiving user requests, parsing the export time range and target table name, generating structured query statements, and dividing the data export task into multiple subtasks that can be executed concurrently, the subtasks are executed using a task queue and structured export files are generated.

Benefits of technology

It improves the efficiency and accuracy of time-series data export, reduces I/O throughput, optimizes resource utilization, lowers operation and maintenance costs, and enables fast query and export.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115137A_ABST
    Figure CN123115137A_ABST
Patent Text Reader

Abstract

This invention discloses a method, apparatus, and electronic device for exporting time-series data based on time range partitioning, comprising: receiving a data export request submitted by a user; parsing the data export request to obtain the export time range, target table name, and export format; determining a set of partitions to be processed based on the export time range and target table name; generating a structured query statement based on the export time range and the set of partitions to be processed, and generating a data export task based on the structured query statement; dividing the data export task into multiple concurrently executable subtasks based on time granularity; executing the subtasks based on the task queue to obtain multiple independent data files; and generating a structured export file based on the independent data files and the export format. This invention improves the efficiency and accuracy of time-series data export and can be applied to the field of data storage technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method, apparatus, and electronic device for exporting time-series data based on time range partitioning. Background Technology

[0002] In modern enterprise information systems, time-series data (such as system logs, transaction records, and user behavior trajectories) typically accumulates continuously using timestamps as the primary key, leading to exponential growth of database tables. Traditional full-table scan export solutions have the following drawbacks when facing data extraction needs in scenarios such as business analysis and regulatory auditing: 1) Performance bottleneck: Linear scanning of massive amounts of data leads to a surge in I / O throughput and non-linear deterioration of query response time; the lack of an effective data location mechanism makes it impossible to optimize using the inherent characteristics of time-series data. 2) Resource conflicts: Export operations compete with online transactions for storage bandwidth and computing resources, causing OLTP performance fluctuations; failure to implement query isolation may trigger storage engine-level lock contention. 3) Architectural limitations: Static data distribution strategies are difficult to adapt to dynamically growing data volumes; maintenance complexity increases sharply with the growth of data volume, resulting in high operation and maintenance costs.

[0003] The above problems urgently need to be addressed. Summary of the Invention

[0004] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0005] Therefore, one objective of this invention is to provide a time-series data export method based on time range partitioning, which improves the efficiency and accuracy of time-series data export.

[0006] Another objective of this invention is to provide a time-series data export device based on time range partitioning.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a method for exporting time-series data based on time range partitioning, including the following steps: Receive a data export request submitted by a user, and parse the data export request to obtain the export time range, target table name and export format; The set of partitions to be processed is determined based on the exported time range and the target table name; A structured query statement is generated based on the export time range and the set of partitions to be processed, and a data export task is generated based on the structured query statement. The data export task is divided into multiple subtasks that can be executed concurrently based on the time granularity. The subtasks are executed based on the task queue to obtain multiple independent data files, and a structured export file is generated based on the independent data files and the export format.

[0008] Furthermore, in one embodiment of the present invention, the step of parsing the data export request to obtain the export time range, target table name, and export format specifically includes: Extract the time interval field, table field, and export format field from the data export request; The time interval field is converted into a timestamp to obtain the exported time range; The target table name is determined based on the table fields, the export format is determined based on the export format fields, and a corresponding export task identifier is assigned to the data export request.

[0009] Furthermore, in one embodiment of the present invention, determining the set of partitions to be processed based on the exported time range and the target table name specifically includes: Read the corresponding target table based on the target table name, and determine the partition time range of each partition of the target table; Traverse each partition of the target table and determine whether there is an intersection between the partition time range and the exported time range; When the partition time range and the exported time range intersect, the partition corresponding to the partition time range is determined as the target partition; A set of partitions to be processed is generated based on the target partition.

[0010] Furthermore, in one embodiment of the present invention, the step of generating a structured query statement based on the export time range and the set of partitions to be processed, and generating a data export task based on the structured query statement, specifically includes: Construct a time condition clause based on the exported time range, and construct a partition filtering clause based on the set of partitions to be processed; The structured query statement is obtained by concatenating the time condition clause and the partition filter clause; Based on the target table, determine the corresponding field types and the field type processing rules for each field type; The data export task is generated based on the structured query statement and the field type processing rules.

[0011] Furthermore, in one embodiment of the present invention, the step of dividing the data export task into multiple concurrently executable subtasks based on time granularity specifically includes: The pagination parameters are determined based on the preset time interval length. Multiple pagination query statements are generated based on the pagination parameters and the structured query statement, and the query partition corresponding to each pagination query statement is determined. Assign a corresponding subtask identifier to each of the pagination query statements; The subtask is constructed based on the subtask identifier, the query partition, the pagination parameters, and the pagination query statement.

[0012] Furthermore, in one embodiment of the present invention, the step of executing the subtask based on the task queue to obtain multiple independent data files, and generating a structured export file according to the independent data files and the export format, specifically includes: The subtasks are sorted by partition and then added to the task queue; Determine the estimated data volume coefficient and the historical average task time of the corresponding partition for each subtask, and determine the load weight of each subtask based on the estimated data volume coefficient and the historical average task time. The priority of each subtask is determined according to the load weight, and the task queue is sorted according to the priority to obtain the scheduling sequence; Determine if the current number of concurrent threads is less than the maximum number of concurrent threads; When the current number of concurrent threads is less than the maximum number of concurrent threads, the subtask is selected from the head of the scheduling sequence and assigned to an idle thread; When the current number of concurrent threads equals the maximum number of concurrent threads, a polling wait begins, and when any thread changes to an idle state, the subtask is selected from the head of the scheduling sequence for allocation. The system receives the independent data files obtained after the execution of each subtask, performs standardization processing on the independent data files, and integrates the standardized independent data files into the structured export file according to the export format.

[0013] Furthermore, in one embodiment of the present invention, the time-series data export method further includes the following steps: Record the task status and execution result of the subtask to obtain subtask log information; The log information of all the subtasks corresponding to the data export request is integrated to generate a data export log.

[0014] On the other hand, embodiments of the present invention provide a time-series data export device based on time range partitioning, comprising: The export request interface module is used to receive data export requests submitted by users and parse the data export request to obtain the export time range, target table name and export format. The partition resolution module is used to determine the set of partitions to be processed based on the exported time range and the target table name; The query statement construction module is used to generate a structured query statement based on the export time range and the set of partitions to be processed, and to generate a data export task based on the structured query statement. The subtask generation module is used to divide the data export task into multiple concurrently executable subtasks based on time granularity. The file output module is used to execute the sub-tasks based on the task queue to obtain multiple independent data files, and generate a structured export file based on the independent data files and the export format.

[0015] On the other hand, embodiments of the present invention provide an electronic device, the electronic device including a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the computer program, when executed by the processor, implements the time-range partition-based timing data export method as described above.

[0016] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the time-range partitioning-based time-series data export method as described above.

[0017] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the time-range partition-based time-series data export method as described above.

[0018] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention receives a user-submitted data export request, parses the request to obtain the export time range, target table name, and export format, determines the set of partitions to be processed based on the export time range and target table name, generates a structured query statement based on the export time range and the set of partitions to be processed, generates a data export task based on the structured query statement, divides the data export task into multiple concurrently executable subtasks based on time granularity, executes the subtasks based on the task queue to obtain multiple independent data files, and generates a structured export file based on the independent data files and the export format. This invention determines the set of partitions to be processed and generates a data export task based on the export time range, divides the data export task into multiple subtasks based on time granularity, and achieves rapid query and export of data within a specified time range by concurrently executing each subtask, thus improving the efficiency and accuracy of time-series data export. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the steps of a time-range partition-based time-series data export method provided in an embodiment of the present invention; Figure 2 A diagram illustrating the relationships between rows, pages, extents, segments, and tablespaces in a database; Figure 3 A schematic diagram of instructions for executing subtasks based on a task queue, provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a time-range partition-based time-series data export device provided in an embodiment of the present invention; Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] This invention provides a time-series data export method based on time range partitioning. By using a range partitioning mechanism, data is divided into time periods, and combined with task scheduling and partition positioning technology, the data can be exported quickly and accurately within a specified time range.

[0024] First, the storage structure of the database involved in the embodiments of the present invention will be described.

[0025] In a database, records are stored row by row, but database reads are not performed row by row. Whether reading one row or multiple rows, the entire page containing those rows is loaded. In other words, the basic unit for managing storage space in a database is the page. A page can store multiple rows. Additionally, databases also contain extents, segments, and tablespaces. These are explained in detail below: A tablespace is a logical container in a database used to organize physical storage files. Each tablespace corresponds to one or more physical files (such as .ibd files) and manages the data storage location on disk. Different tables can be placed in different tablespaces according to their purpose (e.g., separating business tables from log tables). Tablespaces can be categorized as follows: System Tablespace, which stores system metadata; File-Per-Table Tablespace, where each table corresponds to a separate physical file; and General Tablespace, where multiple tables share a single file.

[0026] A segment (or partition) includes data segments, index segments, and rollback segments. It consists of one or more extents, and the extents do not need to be contiguous. A segment is the unit of data allocation in a database; a table segment is created when a table is created, and an index segment is created when an index is created.

[0027] An extent is the basic unit of allocated storage space, typically consisting of a group of contiguous pages (e.g., 64 pages). InnoDB defaults to 64 pages per extent (each page is typically 16KB, totaling 1MB). This reduces fragmentation caused by frequent page allocations and provides efficient storage space management.

[0028] Page (also called block): Space used to store rows of data. A page can contain multiple rows. The page size definition varies depending on the database or data engine. SQL Server's page size is 8KB, Oracle supports block sizes of 2KB, 4KB, 8KB, 16KB, 32KB, and 64KB, and InnoDB uses 16KB. The page size can also be defined using the parameter `innodb_page_size`.

[0029] The relationships between the above rows, pages, sections, paragraphs, and tablespaces are as follows: Figure 2 As shown.

[0030] The following describes some of the terms used in the embodiments of the present invention.

[0031] Range partitioning refers to dividing data in a database table into multiple logical partitions based on a continuous range of values ​​(such as time, ID, etc.). This invention uses range partitioning based on time fields (such as date, timestamp), which can divide data into multiple partitions by month, week, or day, improving query and export efficiency.

[0032] A partitioned table is a special type of database table whose data is physically or logically divided and stored across multiple sub-tables or data blocks, each called a "partition". Partitions are transparent to the user, and queries automatically identify which partitions to access.

[0033] The TO_DAYS() function is a date function in MySQL that converts a date to an integer representing the number of days since year zero. In partitioning, it's used to convert dates to numbers, making RANGE partitions more efficient.

[0034] Partition pruning is an optimization technique used by databases to automatically identify and skip irrelevant partitions when executing SQL queries. This invention utilizes this mechanism to access only the partitions corresponding to the target time period during export, thereby significantly improving performance.

[0035] Data Export: This refers to extracting data that meets certain criteria from a database and saving it to an external file format (such as CSV, JSON, Excel, Parquet) for use in scenarios such as backup, analysis, and migration. This invention supports features such as multi-threaded export, resume interrupted downloads, and export compression.

[0036] Concurrent export / multi-threaded export: This refers to the technique of segmenting data and extracting it concurrently to improve export speed. It is often implemented using a "sharding + concurrency" model, such as sharding based on primary key range or time interval.

[0037] Automatic partition maintenance refers to the system's ability to periodically or on-demand generate new partitions and clean up expired partitions. This can be achieved through scheduled tasks or triggers, ensuring that the partition structure is synchronized with data growth.

[0038] Resume Exporting: This means that if the export process is interrupted, you can continue exporting from the point where you stopped, without having to start over. This is achieved by recording the export progress (such as the last ID).

[0039] Partition metadata table: Stores information such as the time period, partition name, and status of each partition, facilitating automatic partition selection and loading by the system. This can greatly improve partition location efficiency.

[0040] Compress export files: Compress the exported results before writing to disk or transmitting them, such as using .zip or .gz formats, to reduce storage space and improve network transmission efficiency.

[0041] A query execution plan is a plan of execution steps generated by the database before executing an SQL statement. Using range partitioning in conjunction with query conditions allows the database to generate a more optimized query plan, improving performance.

[0042] The time-series data export method based on time range partitioning provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a driver status monitoring method based on machine vision, but is not limited to the above forms.

[0043] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0044] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0045] like Figure 1 The diagram shows a flowchart of a time-series data export method based on time range partitioning provided by an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for exporting time-series data based on time range partitioning, specifically including the following steps: S101. Receive the data export request submitted by the user, and parse the data export request to obtain the export time range, target table name and export format. S102. Determine the set of partitions to be processed based on the exported time range and the target table name; S103. Generate a structured query statement based on the export time range and the set of partitions to be processed, and generate a data export task based on the structured query statement. S104. Divide the data export task into multiple subtasks that can be executed concurrently based on the time granularity. S105. Execute subtasks based on the task queue to obtain multiple independent data files, and generate a structured export file according to the independent data files and the export format.

[0046] Specifically, this invention significantly improves the efficiency and system stability of exporting massive amounts of data within a specified time range by introducing a range partitioning mechanism and a modular scheduling architecture. The method flow of this invention is implemented through an export request interface module, a partition parsing module, a query statement construction module, a subtask generation module, a file output module, and an export status log module. Specifically: Export Request Interface Module: Used to receive export parameters submitted by the user, including export time range, target table name, export format (CSV, JSON, etc.) and export task identifier, etc. Partition parsing module: The core component of the system for quickly locating data subsets. Its main task is to accurately calculate the set of partitions to be processed based on the time field range in the export request, so as to minimize the I / O scan range, reduce invalid reads, and improve export efficiency. The query statement construction module is responsible for automatically generating query statements that conform to database specifications based on the exported parameters (such as table name, field list, time range, hit partition set, etc.) passed in from the upstream module. Concurrent scheduling export module: The export task is divided into multiple subtasks that can be executed concurrently according to the time granularity (by day, hour or minute). Each subtask can independently execute SQL and generate an independent data file. It supports scheduling by thread pool or task queue. File output module: After exporting, the result is output as a file in the specified format. It can be concatenated or compressed into a final file by subtasks and supports breakpoint resume and incremental export strategies. Export Status Log Module: Records the status and execution information of each stage of the export task, including task ID, partition name, start and end time, execution status (in progress, completed, failed), etc., for easy subsequent query and management.

[0047] It can be understood that the export request interface module, acting as a front-end gateway, is responsible for receiving export instructions and time range requests from users; the partition parsing module dynamically matches the required partitions based on the time field, improving query and location efficiency; the query statement construction module generates structured query statements based on partition information, supporting concurrent execution; the concurrent scheduling export module adopts a task sharding and thread pool concurrency mechanism to effectively distribute the load and improve export performance; the file output module converts the result set into a structured export file, supporting multiple output formats and file path archiving; and the export status log module records the task execution status, performance indicators, and error information throughout the process, providing good maintainability and anomaly diagnosis capabilities.

[0048] This invention determines the set of partitions to be processed based on the export time range and generates a data export task. The data export task is divided into multiple subtasks based on time granularity. By executing each subtask concurrently, the fast query and export of data within a specified time range is achieved, which improves the efficiency and accuracy of time series data export.

[0049] As an optional implementation, the export time range, target table name, and export format are obtained by parsing the data export request, specifically including: S1011. Extract the time interval field, table field, and export format field from the data export request; S1012. Convert the time interval field to a timestamp to obtain the exported time range; S1013. Determine the target table name based on the table fields, determine the export format based on the export format field, and assign the corresponding export task identifier to the data export request.

[0050] Specifically, users submit data export requests via API or system interface; the export request interface module receives the data export request, extracts the time range field, table field, and export format field; and parses the string-type time parameter into an internal timestamp. , and verify The system checks if the condition is met; if not, it returns an error. The target table name is determined based on the table fields, the export format is determined based on the export format fields, and an export task identifier must be assigned to each data export request. Finally, the request information is written into the export task registration form.

[0051] As a further optional implementation, the set of partitions to be processed is determined based on the export time range and the target table name, specifically including: S1021. Read the corresponding target table according to the target table name, and determine the partition time range of each partition of the target table; S1022. Traverse each partition of the target table and determine whether there is any overlap between the partition time range and the exported time range. S1023. When the partition time range and the exported time range intersect, determine the partition corresponding to the partition time range as the target partition. S1024. Generate a set of partitions to be processed based on the target partition.

[0052] Specifically, by combining the metadata information of the database partitioned table (Range Partitioned Table) and parsing the boundary values ​​of each partition, the target time interval is determined. , Perform a matching calculation and return the partition identifiers of all hit ranges.

[0053] Table T in the database is defined as a range partition based on the time field t: ,

[0054] The query range for the export request is represented as a closed interval:

[0055] For any partition If there is an overlap with the query time period, then the hit condition is met:

[0056] The final set of partitions containing the exportable data is: .

[0057] The specific implementation process is as follows: 1. Partitioned Tables and Metadata Preparation (1) Basic definition of partition table The target table T in the database needs to be predefined as a range-partitioned table based on the time field. The partitioning rules are based on the converted value of the time field, which is converted into an integer number of days using the TO_DAYS() function.

[0058] (2) Obtaining partition metadata First, access the partition metadata table (which stores the core information of each partition) to obtain the metadata of all partitions, including: the partition's unique identifier p_202507; the partition's time range boundaries: start time (partition.start) and end time (partition.end), and transform p_202507's start=2025-07-1600:00:00, end=2025-07-3123:59:59 based on the boundary values ​​defined in the partition definition. The partition statuses are "Active," "Archived," and "Expired," used to filter inaccessible partitions (deleted or expired partitions). To improve efficiency, partition metadata is cached in memory, avoiding frequent access to the database metadata dictionary and enabling fast queries.

[0059] (3) Standardization of the derived time interval Receive the time parameter passed from the upstream export request interface module and perform the following processing: Format conversion: The string time range start_time="2025-07-1600:00:00" and end_time="2025-07-3123:59:59") is parsed into the system's internal unified timestamp format (Unix timestamp) to ensure consistency in time calculation; Validation: Verify the validity of the time interval, which must satisfy start_time≤end_time. If not, return an error message and terminate the subsequent process.

[0060] (4) Calculation of partition hit conditions For each partition in the cache, the intersection of its time boundary value and the exported time interval is determined. For a partition P (time range is [P_start, P_end]) and the exported query interval [T_start, T_end], if P_start≤T_endANDP_end≥T_start, then the partition has an intersection with the target time range and is determined to be a "hit partition".

[0061] (5) Generation of hit partition set After traversing all cached partitions, all partition identifiers that meet the hit criteria are collected to form the final "hit partition set," which is also the set of partitions to be processed. In the example above, the set is {p_202507, p_202508}.

[0062] At the same time, the module will perform secondary filtering on the set: excluding partitions with abnormal status, such as "expired" or "damaged" partitions; and merging adjacent consecutive partitions (such as p_202507 and p_202508 spanning months) to facilitate task allocation by the subsequent concurrent scheduling module.

[0063] (6) Encapsulation of region sets The final set of hit partitions (such as {p_202507, p_202508}) along with information such as the exported task ID and target table name are encapsulated into structured data and passed to the next level module (query statement construction module) as the core basis for generating accurate query statements.

[0064] In addition, key information about the partition resolution process is recorded, including: export task ID, resolution start / end time; original time range, standardized timestamp; number of partitions hit, list of partition names; reasons for excluding partitions that were not hit (no overlap in time, abnormal status). Log information will be written to the export status log module for subsequent task tracking, performance analysis, and problem diagnosis.

[0065] As a further optional implementation, a structured query statement is generated based on the export time range and the set of partitions to be processed, and a data export task is generated based on the structured query statement, specifically including: S1031. Construct a time condition clause based on the exported time range, and construct a partition filtering clause based on the set of partitions to be processed; S1032. Concatenate the time condition clause and the partition filter clause to obtain the structured query statement; S1033. Determine the corresponding field types based on the target table, and determine the field type processing rules for each field type. S1034. Generate a data export task based on the structured query statement and field type processing rules.

[0066] Specifically, the query field set is defined as follows: , Field namespace; if empty or... If the field set is F=ALL(T) (default is all fields), then the field set is F=ALL(T).

[0067] Let the time field be t, then the time condition clause is: Known hit partition set The partitioning filter clause is then: The resulting structured query statement is: .

[0068] Based on the target table, determine the corresponding field types and the field type processing rules for each field type; finally, generate a data export task based on the structured query statement and the field type processing rules.

[0069] The process for determining field type processing rules is as follows: (1) Field type identification and classification Preliminary preparations: The query statement construction module first obtains field type information from the target table's metadata, including but not limited to: Basic data types: String (VARCHAR / TEXT), Number (INT / FLOAT / DECIMAL), Date / Time (DATE / DATETIME / TIMESTAMP), Boolean (BOOL); Complex types: arrays (such as JSON arrays), objects (such as JSON objects), binary data (BLOB).

[0070] (2) Rules for handling field types in CSV format CSV format uses commas as the default delimiter. Special characters in the field content, format consistency, and data boundary issues require careful handling, as detailed below: String type handling: If the string contains a separator (comma), newline character, or quotation mark, it will be automatically enclosed in double quotes; it distinguishes between empty strings ("") and NULL values, exporting NULL values ​​as N (configurable), and keeping empty strings as "".

[0071] Date and time type handling: uniformly convert to a standardized format to avoid parsing confusion caused by differences in the database's native format; use database functions to directly format during the SQL query stage to reduce application layer conversion overhead.

[0072] Numeric and Boolean type handling: Numeric types (such as INT, FLOAT) directly export the raw value to avoid scientific notation (the DECIMAL type retains a specified number of decimal places); Boolean values ​​are converted to 1 (TRUE) or 0 (FALSE) to ensure numerical compatibility with CSV format.

[0073] Binary data processing: For binary types such as BLOB, convert them to strings using the BASE64_ENCODE() function before exporting to avoid binary data damaging the CSV format structure.

[0074] (3) Rules for handling field types in JSON format JSON format supports key-value pairs, nested structures, and type tags. It is essential to ensure that field types are aligned with the JSON specification, as detailed below: Basic type mapping: Numeric types retain their original format (e.g., 123, 3.14); Boolean values ​​are converted to JSON Boolean types (true / false, not strings); NULL values ​​are explicitly marked as null in JSON, not empty strings.

[0075] Date and time type handling: Convert to ISO8601 standard format characters and store as a string type in JSON, while adding a type specification (such as {"created_at":{"value":"2025-07-16T00:00:00Z","type":"datetime"}}, optional configuration).

[0076] Complex type handling: JSON fields in the database (such as MySQL's JSON type) are exported directly in their original structure, preserving nested objects and arrays (such as {"tags":["log","system"]}). Array type fields (such as comma-separated strings generated by GROUP_CONCAT) are converted to JSON arrays (e.g., converting "a,b,c" to ["a","b","c"]).

[0077] String escaping: Perform JSON escaping on special characters (such as quotation marks and backslashes) in the string (e.g., "Hesaid\"Hello\"" is converted to "Hesaid\"Hello\"").

[0078] (4) Format conversion triggering and verification Conversion timing: During the SQL statement generation stage, basic conversion is completed through database functions (such as DATE_FORMAT, BASE64_ENCODE); before the data is written to the file (file output module), the application layer performs secondary conversion (such as CSV quotation mark encapsulation, JSON structure assembly) to ensure a balance between flexibility and performance.

[0079] Validation mechanism: Perform format validation on the converted data. For example, CSV format validation: check that the number of fields in each row is consistent with the table structure and that special characters have been correctly escaped; JSON format validation: verify the syntax correctness of the output content through a JSON parser to avoid nesting errors.

[0080] As a further optional implementation, the data export task can be divided into multiple subtasks that can be executed concurrently based on time granularity, specifically including: S1041. Determine the pagination parameters according to the preset time interval length, generate multiple pagination query statements according to the pagination parameters and structured query statements, and determine the query partition corresponding to each pagination query statement. S1042. Assign a corresponding subtask identifier to each pagination query statement; S1043. Construct a subtask based on the subtask identifier, query partition, pagination parameters, and pagination query statement.

[0081] Specifically, this invention uses a pagination mechanism to split large result sets into small batches of data, enabling incremental export and reducing I / O and memory consumption for a single query.

[0082] The implementation process of the pagination mechanism is as follows: (1) Pagination basis selection: Prioritize pagination based on time fields: Utilize the temporal order of time-series data and split it by time granularity (such as hour or day). For example, split the partition data from "2025-01-01 to 2025-01-31" into 31 sub-pages by day. (2) Assisted pagination based on primary key: For scenarios with high repetition of time fields, pagination based on the range of primary key (such as auto-incrementing ID) can be combined to avoid data duplication or omission (such as WHERE id>last_id LIMITpage_size).

[0083] (3) Calculation of pagination parameters: Receive the user-configured page size (page_size, such as 1000 rows / page), combine it with the estimated data volume of the target partition (obtained through partition metadata), and calculate the total number of pages: total_pages = CEIL(estimated_rows / page_size); Generate pagination offsets: For time-based pagination, calculate each sub-time interval (e.g., [2025-01-01 00:00:00, 2025-01-01 23:59:59]); for primary key pagination, calculate each sub-range (e.g., id BETWEEN 1 AND 1000, id BETWEEN 1001 AND 2000).

[0084] (4) Pagination SQL generation: SQL example for time pagination: SELECT FROM Logs PARTITION (p_202401) WHERE created_at BETWEEN '2025-01-01 00:00:00' AND '2025-01-01 23:59:59'; Supports "cursor pagination": By recording the timestamp or primary key of the last data on the previous page (such as last_created_at, last_id), it avoids the performance issues of traditional LIMIT offset (too large an offset will lead to a full table scan).

[0085] 3. Implementation process of concurrent construction and subtask allocation Based on the pagination results and partition information, the export task is split into subtasks that can be executed concurrently, and load balancing is achieved through a scheduling algorithm.

[0086] The subtask generation process is as follows: Subtasks are generated using "partition + pagination" as the dimension: After each hit partition is split into paginations, each pagination unit is a subtask. For example, partition p_202401 generates 31 subtasks by paginating by day, and each subtask corresponds to one day's data. Subtask metadata includes: task ID, partition, pagination range (time / primary key), SQL statement, estimated data volume, and priority (tasks with smaller partitions or shorter historical execution times have higher priority).

[0087] Rate limiting mechanisms control query rates and resource consumption to prevent export tasks from competing for resources with online services. The implementation process of rate limiting mechanisms is as follows: (1) Limit on the number of concurrent threads: Based on database performance thresholds (such as maximum number of connections and I / O throughput), configure the maximum number of concurrent threads C (e.g., C=10), which is strictly controlled by the resource limitation model (one executable task group ≤ C) of the concurrent scheduling module.

[0088] Query rate limit: (2) Control the query frequency of a single thread: By adding a delay (such as sleep(100ms)) between subtask execution intervals, avoid a large number of SQL requests impacting the database in a short period of time; Limit the query rate of a single partition: For partitions with large amounts of data (such as active partitions), reduce the frequency of their pagination queries and prioritize allocating resources to smaller partitions.

[0089] (3) Database connection pool control: Allocate a separate database connection pool for the export task, set the maximum number of connections (e.g., max_connections=20), and isolate it from the online business connection pool to avoid connection exhaustion.

[0090] Dynamic rate limiting adjustment: By monitoring database performance metrics (such as CPU utilization and I / O wait time), the rate limiting parameters are dynamically adjusted. For example, when the database CPU usage exceeds 80%, the number of concurrent threads is automatically reduced to C / 2.

[0091] As a further optional implementation, multiple independent data files are obtained by executing subtasks based on the task queue, and a structured export file is generated according to the independent data files and the export format, specifically including: S1051. Sort the subtasks by partition and add them to the task queue; S1052. Determine the estimated data volume coefficient of each subtask and the historical average task time of the corresponding partition, and determine the load weight of each subtask based on the estimated data volume coefficient and the historical average task time. S1053. Determine the priority of each subtask based on the load weight, and sort the task queue according to the priority to obtain the scheduling sequence. S1054. Determine whether the current number of concurrent threads is less than the maximum number of concurrent threads; S1055. When the current number of concurrent threads is less than the maximum number of concurrent threads, select a subtask from the head of the scheduling sequence and assign it to an idle thread. S1056. When the current number of concurrent threads equals the maximum number of concurrent threads, enter the polling wait state, and when any thread changes to idle, select a subtask from the head of the scheduling sequence for allocation. S1057. Receive the independent data files obtained after the execution of each subtask, standardize the independent data files, and integrate the standardized independent data files into a structured export file according to the export format.

[0092] like Figure 3 The diagram shown illustrates the instruction for executing subtasks based on a task queue, as provided in an embodiment of the present invention. This embodiment employs a polling + load-aware scheduling algorithm (RR-LBA) for subtask scheduling and execution. The specific process is as follows: 1. Task Selection: Load-aware prioritization ensures that lightweight tasks (with short historical latency) are scheduled first, reducing overall waiting time. The task selection phase implements the following steps: (1) Load weight calculation: Calculate the load weight W for each subtask in the scheduling queue, using the following formula: W = Historical average time × Estimated data volume coefficient (the larger the data volume, the higher the coefficient and the greater the weight).

[0093] For example, a subtask with a historical time of 20 seconds and an estimated length of 100,000 lines has a lower weight than a subtask with a historical time of 50 seconds and an estimated length of 500,000 lines.

[0094] (2) Priority sorting: The scheduling queue is sorted in ascending order by load weight W, with tasks having lower weights having higher priority (being selected first). If a task has no historical data (first execution), it is sorted in descending order by estimated data volume (tasks with smaller data volumes are given priority).

[0095] (3) Task filtering: Filter subtasks with abnormal status (such as those with the associated partition status "expired") and retain only valid tasks with the status "pending scheduling".

[0096] 2. Resource assessment and task allocation: Determine the task allocation strategy based on the current concurrent resource usage (thread pool idle state).

[0097] (1) Determine if concurrent resources are not full: Acquire the current running thread count current_running of the thread pool in real time. If current_running < C (maximum concurrent thread count), it is determined that the resources are not full. For example: C=10, 8 threads are currently running, the resources are not full, and new tasks can be directly allocated.

[0098] (2) Direct allocation when resources are not full: Select the subtask with the highest priority from the head of the sorted to-be-scheduled queue, assign it to an idle thread (select a slot with the status of "idle" from the thread slot array), and update: the subtask status to "running"; the thread slot information (bind the subtask ID, start time, estimated end time); the value of current_running is increased by 1.

[0099] (3) Polling replacement when resources are full: If current_running = C (resources are full), enter the polling waiting phase: 1. Poll and scan the thread slot array thread_slots to monitor the execution status of each thread; 2. When it is detected that a thread has completed execution (the status becomes "idle"), immediately release the thread slot; 3. Select the subtask with the highest priority from the head of the to-be-scheduled queue, assign it to the released slot, update the slot information, and current_running remains unchanged (the old task is completed and released, and the new task is accessed).

[0100] 3. Load-aware dynamic adjustment: During task execution, the scheduling strategy is optimized by monitoring the load status in real time to avoid single-thread overload or resource waste.

[0101] (1) Real-time load monitoring: Collect thread execution data every second: the elapsed time of the current subtask, database I / O usage, and thread CPU usage. If the elapsed time of a certain thread exceeds 150% of its estimated time, or the database I / O usage exceeds a threshold (such as 80%), mark the thread as "high load".

[0102] (2) Dynamic priority update: For a running subtask, if the actual elapsed time far exceeds historical data, update its load weight W, and adjust the priority of subsequent tasks of the same type (for example, extend the historical elapsed time record and reduce its priority).

[0103] 4. Thread slot management and fault tolerance: To ensure efficient reuse of thread resources and task reliability, refined management of thread slots is required: (1) Slot status tracking: The thread slot array thread_slots records the status of each slot in real time: Running: bound subtask ID, start time, estimated end time; Idle: no bound task, can receive new tasks; Abnormal: task execution failed (such as SQL timeout, network interruption), needs to be marked and retried.

[0104] (2) Slot release mechanism: After the subtask is completed normally, the thread slot status is updated to "idle" and resources are released; at the same time, the subtask execution result (such as the path of the exported temporary file and the time taken) is written to the export status log module, and the historical time taken of the task is updated to provide a basis for subsequent scheduling.

[0105] 5. Subtask isolation and fault tolerance: (1) Each subtask is executed independently and has a unique subtask ID. The execution result is written to a temporary file (such as task_123_p202401_page1.tmp) to avoid the failure of a single task affecting the whole. (2) Support for retrying failed subtasks: For failed subtasks (such as SQL timeout), record the reason for failure and re-add them to the task queue. The number of retries shall not exceed 3 (configurable).

[0106] (3) Subtask result integration: After all subtasks are completed, the file output module merges the temporary files in chronological or partition order to generate the final export file.

[0107] As an optional implementation, the time-series data export method further includes the following steps: S201. Record the task status and execution result of the subtask to obtain the subtask log information; S202. Integrate all subtask log information corresponding to the data export request to generate a data export log.

[0108] Specifically, it records the status and execution information of each stage of the subtask, including task ID, partition name, start and end time, execution status (in progress, completed, failed), etc., to facilitate subsequent querying and management. The export status log module is an important auxiliary module in the entire database partition export system, responsible for recording, tracking, auditing, and error analysis, and it runs through all stages of the export lifecycle.

[0109] The implementation process of the present invention has been described above. It can be understood that the embodiments of the present invention determine the set of partitions to be processed based on the export time range and generate a data export task. The data export task is then divided into multiple subtasks based on time granularity. By concurrently executing each subtask, rapid querying and exporting of data within a specified time range is achieved, improving the efficiency and accuracy of time-series data export.

[0110] Compared with the prior art, the embodiments of the invention have the following advantages: 1) Time-series data is partitioned using time range as the primary control condition. During data export, only the relevant time partitions are activated, avoiding inefficient methods such as full table scans or large-scale data filtering. Compared to traditional query export methods based on data source primary keys and single indexes, this solution is more suitable for the characteristics of time-series data and significantly reduces query latency and resource consumption.

[0111] 2) Introducing a multi-level indexing system (e.g., partition-level, time-slice-level, data block-level) enables coarse-to-fine hierarchical positioning, significantly improving data location and retrieval efficiency. The multi-level indexing structure not only speeds up the positioning process but also supports flexible combinations of filtering conditions based on different dimensions (time, type, source, etc.), enabling rapid data export under complex query conditions.

[0112] 3) A streaming read + asynchronous buffering mechanism, combined with an export task scheduling module, ensures high-performance data export without interrupting the normal operation of the business system. Data transmission is organized using a lightweight intermediate structure, facilitating quick switching between different formats (such as CSV, Parquet, JSON), improving the system's scalability and export flexibility.

[0113] like Figure 4 The diagram shown is a structural schematic of a time-series data export device based on time range partitioning provided in an embodiment of the present invention. (Refer to...) Figure 4 This invention provides a time-series data export device based on time range partitioning, comprising: The export request interface module is used to receive data export requests submitted by users, and parse the data export request to obtain the export time range, target table name and export format; The partition resolution module is used to determine the set of partitions to be processed based on the export time range and the target table name; The query statement building module is used to generate structured query statements based on the export time range and the set of partitions to be processed, and to generate data export tasks based on the structured query statements. The subtask generation module is used to divide the data export task into multiple subtasks that can be executed concurrently based on the time granularity. The file output module is used to execute subtasks based on the task queue to obtain multiple independent data files, and generate a structured export file based on the independent data files and the export format.

[0114] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned time-range partitioned timing data export method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0116] like Figure 5 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 5 This invention provides an electronic device, comprising: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to execute the timing data export method based on time range partitioning of the embodiments of this invention. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0117] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] like Figure 6 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 6 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 601, which can be executed by one or more processors to implement the above-described time-range partition-based time-series data export method.

[0119] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0120] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the above-described time-range partition-based time-series data export method.

[0121] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0123] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0124] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0125] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0126] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0129] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0130] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0132] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0133] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A time range partition based time series data derivation method, characterized in that, Includes the following steps: Receive a data export request submitted by a user, and parse the data export request to obtain the export time range, target table name and export format; The set of partitions to be processed is determined based on the exported time range and the target table name; A structured query statement is generated based on the export time range and the set of partitions to be processed, and a data export task is generated based on the structured query statement. The data export task is divided into multiple subtasks that can be executed concurrently based on the time granularity. The subtasks are executed based on the task queue to obtain multiple independent data files, and a structured export file is generated based on the independent data files and the export format.

2. The time range partition-based time series data derivation method of claim 1, wherein, The step of parsing the data export request to obtain the export time range, target table name, and export format specifically includes: Extract the time interval field, table field, and export format field from the data export request; The time interval field is converted into a timestamp to obtain the exported time range; The target table name is determined based on the table fields, the export format is determined based on the export format fields, and a corresponding export task identifier is assigned to the data export request.

3. The time range partition based time series data derivation method of claim 1, wherein, The step of determining the set of partitions to be processed based on the exported time range and the target table name specifically includes: Read the corresponding target table based on the target table name, and determine the partition time range of each partition of the target table; Traverse each partition of the target table and determine whether there is an intersection between the partition time range and the exported time range; When the partition time range and the exported time range intersect, the partition corresponding to the partition time range is determined as the target partition; A set of partitions to be processed is generated based on the target partition.

4. The time range partition-based time series data derivation method of claim 1, wherein, The step of generating a structured query statement based on the export time range and the set of partitions to be processed, and generating a data export task based on the structured query statement, specifically includes: Construct a time condition clause based on the exported time range, and construct a partition filtering clause based on the set of partitions to be processed; The structured query statement is obtained by concatenating the time condition clause and the partition filter clause; Based on the target table, determine the corresponding field types and the field type processing rules for each field type; The data export task is generated based on the structured query statement and the field type processing rules.

5. The time range partition based time series data derivation method of claim 1, wherein, The process of dividing the data export task into multiple concurrently executable subtasks based on time granularity specifically includes: The pagination parameters are determined based on the preset time interval length. Multiple pagination query statements are generated based on the pagination parameters and the structured query statement, and the query partition corresponding to each pagination query statement is determined. Assign a corresponding subtask identifier to each of the pagination query statements; The subtask is constructed based on the subtask identifier, the query partition, the pagination parameters, and the pagination query statement.

6. The method for exporting time-series data based on time range partitioning according to claim 1, characterized in that, The process of executing the subtasks based on the task queue to obtain multiple independent data files, and generating a structured export file based on the independent data files and the export format, specifically includes: The subtasks are sorted by partition and then added to the task queue; Determine the estimated data volume coefficient and the historical average task time of the corresponding partition for each subtask, and determine the load weight of each subtask based on the estimated data volume coefficient and the historical average task time. The priority of each subtask is determined according to the load weight, and the task queue is sorted according to the priority to obtain the scheduling sequence; Determine if the current number of concurrent threads is less than the maximum number of concurrent threads; When the current number of concurrent threads is less than the maximum number of concurrent threads, the subtask is selected from the head of the scheduling sequence and assigned to an idle thread; When the current number of concurrent threads equals the maximum number of concurrent threads, a polling wait begins, and when any thread changes to an idle state, the subtask is selected from the head of the scheduling sequence for allocation. The system receives the independent data files obtained after the execution of each subtask, performs standardization processing on the independent data files, and integrates the standardized independent data files into the structured export file according to the export format.

7. A method for exporting time-series data based on time range partitioning according to any one of claims 1 to 6, characterized in that, The time-series data export method further includes the following steps: Record the task status and execution result of the subtask to obtain subtask log information; The log information of all the subtasks corresponding to the data export request is integrated to generate a data export log.

8. A time-series data export device based on time range partitioning, characterized in that, include: The export request interface module is used to receive data export requests submitted by users and parse the data export request to obtain the export time range, target table name and export format. The partition resolution module is used to determine the set of partitions to be processed based on the exported time range and the target table name; The query statement construction module is used to generate a structured query statement based on the export time range and the set of partitions to be processed, and to generate a data export task based on the structured query statement. The subtask generation module is used to divide the data export task into multiple concurrently executable subtasks based on time granularity. The file output module is used to execute the sub-tasks based on the task queue to obtain multiple independent data files, and generate a structured export file based on the independent data files and the export format.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the computer program is executed by the processor, it implements the time-range partition-based time-series data export method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the time-range partition-based time-series data export method as described in any one of claims 1 to 7.