Satellite telemetry data management method and system based on clickhouse

Abstract

The application provides a satellite telemetry data management system and method based on ClickHouse, which constructs a heterogeneous database layered storage architecture, business data is stored in a relational business database, and telemetry time series data is stored in a ClickHouse database; a latest data materialized view is created, and the latest telemetry value of each device is automatically maintained; raw telemetry data is preprocessed to obtain an alarm preprocessing data table; according to a query data estimation result, when the estimated data volume is greater than a threshold value, streaming transmission is performed. The system and method can efficiently store and manage massive satellite telemetry data, and simultaneously support real-time alarm monitoring and fast query requirements of historical data analysis.

Description

Technical Field

[0001] This invention relates to the field of satellite telemetry data processing technology, specifically to a method for storing, querying, and managing massive amounts of satellite telemetry data, and more specifically to a satellite telemetry data management method and system based on ClickHouse. Background Technology

[0002] Satellite telemetry systems continuously collect various parameter data during satellite operation, including temperature, voltage, current, and attitude angle. These data have the following characteristics:

[0003] 1. Huge amount of data: A single satellite can generate tens of millions of telemetry data points per day, and the amount of data is even larger when multiple satellites are running simultaneously;

[0004] 2. Frequent writes: Telemetry data is continuously written at high frequencies (e.g., 1Hz or higher);

[0005] 3. Complex query modes: It needs to support multi-dimensional queries such as time range queries, satellite filtering, and parameter combination queries;

[0006] 4. High real-time requirements: Alarm monitoring systems need to quickly access the latest data.

[0007] Currently, the following schemes are mainly used for satellite telemetry data management:

[0008] Option 1: Traditional relational databases (such as MySQL and Oracle) suffer from a sharp decline in query performance as the amount of data increases, making it difficult to support fast retrieval of hundreds of millions of data points.

[0009] Option 2: Time-series databases (such as InfluxDB and TimescaleDB) have the problems of complex integration with business systems, limited support for complex queries, and difficulty in handling real-time monitoring and historical analysis at the same time.

[0010] Option 3: Hybrid architecture (MySQL + time-series database), which has problems such as complex data synchronization mechanism, low efficiency of cross-database query and high latency of real-time alarm.

[0011] In other words, the current data management framework cannot fully meet the management needs of satellite telemetry data. Therefore, a data management method that can efficiently store massive amounts of telemetry data, support fast real-time queries, and integrate well with business systems is urgently needed. Summary of the Invention

[0012] The present invention aims to overcome the above-mentioned defects and provide a method and system that can efficiently store and manage massive amounts of satellite telemetry data, while supporting real-time alarm monitoring and rapid querying of historical data analysis. In other words, it can achieve millisecond-level querying while storing hundreds of billions of data.

[0013] This invention provides a satellite telemetry data management system based on ClickHouse, characterized in that it includes a data storage module;

[0014] The aforementioned data storage module includes a relational business database and a ClickHouse-type telemetry time-series database;

[0015] The aforementioned relational business database stores satellite business data;

[0016] The aforementioned ClickHouse-type telemetry time series database stores satellite telemetry time series data;

[0017] The aforementioned ClickHouse-type telemetry time series database includes a data update submodule;

[0018] The aforementioned data update submodule ensures that the ClickHouse type telemetry time series database always executes the latest data materialized view at the front end.

[0019] Furthermore, the ClickHouse-based satellite telemetry data management system provided by this invention is further characterized in that:

[0020] The aforementioned relational business database includes a telemetry data preprocessing submodule;

[0021] The aforementioned telemetry data preprocessing submodule preprocesses the telemetry data to generate an alarm preprocessing table.

[0022] The alarm preprocessing table above only contains data related to alarm monitoring;

[0023] The alarm monitoring data mentioned above is sorted in chronological order.

[0024] Furthermore, the ClickHouse-based satellite telemetry data management system provided by this invention is further characterized in that:

[0025] The sorting key for the alarm preprocessing table is designed as: TimeStamp, TelemCode, DeviceId.

[0026] Furthermore, the ClickHouse-based satellite telemetry data management system provided by this invention is further characterized in that:

[0027] It also includes a data transmission module;

[0028] The aforementioned data transmission module includes a data prediction submodule, a normal transmission submodule, and a streaming transmission submodule;

[0029] Among them, the aforementioned data estimation submodule estimates the total amount of data in the query results based on the query request;

[0030] When the estimated data volume exceeds the preset threshold, the streaming submodule is activated.

[0031] When the estimated data volume is not greater than the preset threshold, the normal transmission submodule is activated.

[0032] Furthermore, the ClickHouse-based satellite telemetry data management system provided by this invention is further characterized in that:

[0033] The data transmission of the above streaming submodule includes the following steps:

[0034] S1. Read data from the database line by line;

[0035] S2. Serialize the data line by line into JSON;

[0036] S3. Write the single-line JSON fragment generated in S2 directly into the output stream;

[0037] S4. Batch refresh the data in the output stream according to the threshold.

[0038] Furthermore, the ClickHouse-based satellite telemetry data management system provided by this invention is further characterized in that:

[0039] It also includes a data query module;

[0040] The data query methods of the above data query module are as follows:

[0041] SS1. Initialize the global timestamp at the beginning of the query process;

[0042] SS2. Query incremental new data by timestamp condition, and filter data whose timestamp is greater than the previous round's global timestamp;

[0043] SS3. Process the incremental data selected in this round;

[0044] SS4. After data processing is complete, update the global timestamp to the maximum timestamp of all data in the current round of results;

[0045] SS5. Iterate through the next round of incremental queries and continue repeating the steps of SS2-SS4.

[0046] Furthermore, this invention also provides a satellite telemetry data management method based on ClickHouse, characterized by comprising the following steps:

[0047] S1. Construct a heterogeneous database hierarchical storage architecture, with business data stored in a relational business database and telemetry time-series data stored in a ClickHouse database;

[0048] S2. Create the latest materialized view of the data and automatically maintain the latest telemetry values ​​for each device;

[0049] S3. Preprocess the raw telemetry data to obtain the alarm preprocessing data table;

[0050] S4. Based on the estimated results of the query data, when the estimated data volume is greater than the threshold, perform streaming transmission.

[0051] Furthermore, the satellite telemetry data management method based on ClickHouse provided by this invention is further characterized in that:

[0052] In S2, the latest data materialized view uses the ReplacingMergeTree engine to sort by device ID and telemetry parameter code, automatically deduplicating and retaining the latest records.

[0053] Furthermore, the satellite telemetry data management method based on ClickHouse provided by this invention is further characterized in that:

[0054] In S3, the alarm preprocessing data table mentioned above only contains the fields that the alarm rules are concerned with, and the sorting key is designed with timestamp priority.

[0055] The above timestamp sorting rules enable incremental query results based on timestamps;

[0056] The method for incremental querying described above is as follows:

[0057] S31. At the beginning of the query process, initialize the global timestamp;

[0058] S32. Query incremental new data based on timestamp conditions, and filter data whose timestamp is greater than the previous round's global timestamp;

[0059] S33. Perform data processing on the incremental data selected in this round;

[0060] S34. After the data processing is complete, update the global timestamp to the maximum timestamp of all data in the current round of results;

[0061] S35. Iterate through the loop to the next round of incremental query, and continue repeating steps S32-S34.

[0062] Furthermore, the satellite telemetry data management method based on ClickHouse provided by this invention is further characterized in that:

[0063] In S4, the specific method for the above streaming transmission is as follows:

[0064] S41. Read data from the database line by line;

[0065] S42. Serialize the data line by line into JSON;

[0066] S43. Write the single-line JSON fragment generated in S42 directly to the output stream;

[0067] S44. Batch refresh the data in the output stream according to the threshold.

[0068] Functions and effects of this invention:

[0069] This invention utilizes ClickHouse columnar storage and compression technology to achieve storage efficiency that is 5-10 times higher than that of traditional databases.

[0070] The solution of this invention has strong scalability, can support the storage of tens of billions of data points, and the query performance does not decrease significantly as the data volume increases.

[0071] The solution of this invention has good real-time performance, and the incremental query mechanism ensures that alarm monitoring can obtain new data in a timely manner.

[0072] The solution of this invention has low resource consumption: streaming transmission supports the export of large amounts of data, and memory usage remains constant. Attached Figure Description

[0073] Figure 1 The database hierarchical structure diagram of this embodiment.

[0074] Figure 2 This embodiment describes a method for creating the latest data materialized view.

[0075] Figure 3 The alarm preprocessing data table design method in this embodiment.

[0076] Figure 4 Flowchart of the data transmission method in this embodiment.

[0077] Figure 5 The incremental query mechanism in this embodiment.

[0078] Figure 6 The telemetry data writing process in Example 1 of this application.

[0079] Figure 7 The latest data query process in Example 2 of this application.

[0080] Figure 8 The historical data export process in Example 3 of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] This embodiment provides a satellite telemetry data management system based on ClickHouse, which includes a data storage module, a data transmission module, and a data query module;

[0083] This data storage module includes a relational business database and a ClickHouse-type telemetry time-series database;

[0084] ClickHouse type telemetry time series database stores satellite telemetry time series data;

[0085] This ClickHouse type telemetry time series database includes a data update submodule, which ensures that the ClickHouse type telemetry time series database always executes the latest data materialized view on the front end.

[0086] Relational business databases store satellite business data;

[0087] This relational business database includes a telemetry data preprocessing submodule, which preprocesses telemetry data to form an alarm preprocessing table. This table contains only alarm monitoring-related data, which is sorted chronologically. The sorting keys for the alarm preprocessing table are: TimeStamp, TelemCode, DeviceId.

[0088] The data transmission module includes a data prediction submodule, a normal transmission submodule, and a streaming transmission submodule;

[0089] The data estimation submodule estimates the total amount of data in the query results based on the query request.

[0090] When the estimated data volume exceeds the preset threshold, the streaming submodule is activated.

[0091] When the estimated data volume is not greater than the preset threshold, the normal transmission submodule is activated.

[0092] The data transmission of the streaming submodule includes the following steps:

[0093] S1. Read data from the database line by line;

[0094] S2. Serialize the data line by line into JSON;

[0095] S3. Write the single-line JSON fragment generated in S2 directly into the output stream;

[0096] S4. Batch refresh the data in the output stream according to the threshold.

[0097] The data query method for this data query module is as follows:

[0098] SS1. Initialize the global timestamp at the beginning of the query process;

[0099] SS2. Query incremental new data by timestamp condition, and filter data whose timestamp is greater than the previous round's global timestamp;

[0100] SS3. Process the incremental data selected in this round;

[0101] SS4. After data processing is complete, update the global timestamp to the maximum timestamp of all data in the current round of results;

[0102] SS5. Iterate through the next round of incremental queries and continue repeating the steps of SS2-SS4.

[0103] Based on the above system, this embodiment also provides a satellite telemetry data management method based on ClickHouse, including the following steps:

[0104] Step 1, as follows Figure 1 As shown, a heterogeneous database hierarchical storage architecture is constructed.

[0105] Specifically, business data and time-series data are stored separately, namely, a business database (MySQL) and a time-series database (ClickHouse). This achieves heterogeneous layered storage, with the two types of databases having completely isolated responsibilities and storing different data separately. The upper layer is uniformly scheduled by the application service layer, and cross-database join queries are not performed at the database level. The data association and assembly are completed by the intermediate application business code, namely the business logic layer.

[0106] The business database (MySQL) stores business data such as configuration, status, and records.

[0107] Examples include: alarm rule configuration: static configuration parameters such as alarm thresholds, trigger conditions, and alarm levels configured by the administrator; alarm status tracking: real-time status data of whether the current alarms of the device / satellite are active, recovered, or pending processing; alarm history record: event log of alarm triggering, processing, and closure; user permission management: system management configurations such as accounts, roles, menu permissions, and data permissions; satellite metadata: basic file information such as satellite number, model, project, equipment location, and parameter definitions. This type of data is small in volume, frequently involves CRUD operations, has high transaction requirements, and has a variable data structure. Relational MySQL is used to adapt to this type of business management data.

[0108] The time-series database (ClickHouse) stores massive amounts of telemetry data.

[0109] Examples include: historical telemetry data: massive amounts of raw time-series sampling data such as voltage, temperature, and power consumption periodically reported by various satellite sensors; latest data view: a real-time latest indicator view generated based on the raw telemetry data, used to display the satellite's instantaneous operating status on the page; and preprocessed data table: an intermediate preprocessed time-series table after aggregation, noise reduction, and statistical analysis of the raw telemetry data. This type of data is massive, write-intensive, rarely updated or deleted, and primarily aggregated and queried along the time dimension. Using MySQL for storage would overwhelm its performance due to the sheer volume of data. Therefore, in this embodiment, ClickHouse is used for compressed storage and rapid statistical querying of the massive time-series data.

[0110] Since MySQL and ClickHouse are two completely different database engines, cross-database joins cannot be directly performed in SQL statements. Therefore, the joins are implemented at the business logic layer. For example, the logic might be as follows: First, retrieve satellite metadata, alarm rules, and alarm event numbers from MySQL; then, use the obtained satellite ID, alarm start and end times, etc., as query conditions to retrieve historical satellite telemetry data for the corresponding time period from ClickHouse; finally, combine the two query results and return the complete ID, alarm, and corresponding telemetry data to the front end.

[0111] In this design, storing business data in MySQL ensures transaction reliability and flexible access, while storing massive time-series data in ClickHouse ensures both storage cost and aggregate query performance. Furthermore, when telemetry data surges in the future, only ClickHouse needs to be expanded, and when business data increases, only MySQL needs to be expanded, ensuring that they operate independently without interference.

[0112] Step Two, as follows Figure 2 As shown, a new data materialized view is created, which uses the ReplacingMergeTree engine.

[0113] Specifically, satellites continuously report periodic telemetry metrics (time-series measurement data such as temperature, voltage, and power consumption). This data first enters ClickHouse's telemetry history table `telemetry_values`, which is the main history table of ReplacingMergeTree. This saves the original data for each report and permanently stores it in the full time-series log. When new data is added to the history table, ClickHouse automatically triggers an update to the latest materialized view `telemetry_latest_values` without requiring additional scheduled tasks. The script synchronizes, updating the view synchronously upon data entry. This can be achieved by using the ReplacingMergeTree merging tree engine. When merging duplicate primary key data in the background partitioning, it accurately identifies multiple records with the same primary key combination, automatically deleting old historical values ​​and retaining only the latest records. In other words, it automatically overwrites / replaces old data with newly written data, achieving the business goal of retaining only the latest records. For example, when inserting data into the parent table (the history table `telemetry_values`), the data automatically flows synchronously into the bound materialized view `telemetry_latest_values`.

[0114] The latest materialized view of data is grouped and deduplicated by Device ID and Telemetry Parameter Code (TelemCode). This means that for the same measurement point parameter on the same device, the view always retains only the latest reported data. Furthermore, based on the data processing scheme of the aforementioned merging tree engine, all reported records for the same device and the same telemetry parameter are physically arranged together in the disk partition, achieving ordered data results and improving the efficiency of the engine's search process.

[0115] The advantages of this design are that the historical table stores a full and massive amount of time-series data, which can be used for historical backtracking, statistical analysis, etc. The materialized view only stores the latest instantaneous values ​​of all measurement points on all devices. The front-end dashboard can directly query the view to check the current operating status of the device in real time, without scanning massive amounts of historical data, thus significantly improving query speed. The ReplacingMergeTree engine can automatically eliminate old data based on the version field, thereby overcoming the problem of excessive redundancy caused by traditional engines retaining multiple data entries and unclear priorities. Moreover, it eliminates the need for business code to write to two tables, Kafka synchronization, and scheduled scripts. The latest view is synchronized and refreshed the moment upstream telemetry data falls into this historical table, ensuring real-time performance and system stability.

[0116] Based on the above characteristics, when querying, add the "FINAL" key field (this field can be customized according to actual needs, such as: fin, end, etc.) to ensure data consistency.

[0117] Step 3, as follows Figure 3As shown, an alarm preprocessing data table is designed.

[0118] The core idea is to preprocess and filter the raw telemetry data to create an alarm preprocessing table, `alarm_check_values`. This preprocessing table is stored in order based on simplified fields and time, and provides incremental queries to the alarm monitoring service. It acts as a data trimming and buffering layer between the raw data and the alarm service, significantly reducing the amount of data queried by the alarm monitoring service.

[0119] Specifically, raw telemetry data contains numerous fields, such as basic device attributes, redundant extended fields, and backup acquisition parameters. Many of these fields are irrelevant to alarm rule validation. Therefore, the preprocessing stage involves field filtering and pruning, retaining only the core fields essential for alarm rule calculation and threshold determination. This field simplification reduces the size of individual data entries. By compressing the data storage volume of individual entries at the data source level, disk usage is reduced. Furthermore, alarm service queries do not require loading useless fields, directly reducing the amount of data in the query state.

[0120] The preprocessed table, after preprocessing and filtering, contains only the fields necessary for alarm monitoring. Its sorting design is as follows: TimeStamp (data acquisition timestamp), TelemCode (telemetry indicator code), and DeviceId (unique device number). Data is allocated in the database according to "time-measurement point-device," achieving the characteristic of storage sorted by timestamp. Therefore, when time-series data from the same device and measurement point are physically stored consecutively, the database can quickly and continuously scan when querying single-device data by time range, skipping irrelevant data and further reducing the amount of data scanned.

[0121] Furthermore, due to the characteristic of data being stored in a time-stamp sorted manner, each query of the alarm monitoring service only needs to obtain the incremental data of the latest time interval based on the timestamp of the previous query, without having to scan the entire historical data table. Therefore, the amount of data queried for alarm monitoring is significantly reduced.

[0122] The advantage of this design is that it filters the original redundant telemetry data through preprocessing, stores it in a structured time-series format, and enables incremental on-demand querying for alarm services. As a result, it achieves a lightweight, time-series, and incrementally queryable transformation of the original massive telemetry data into alarm query data.

[0123] Step 4, as follows Figure 4 As shown, a streaming transmission is constructed under a large data volume.

[0124] This step describes a data transmission scenario in query mode:

[0125] Upon receiving a query request, first estimate the total data volume of the query results (a common estimation method is: estimated data volume = estimated number of rows × average row size, where the number of rows is quickly obtained using EXPLAIN (MySQL / PG), and the row size is a configured value or cached average). Then, compare the estimated volume with a preset threshold.

[0126] When the estimated data volume exceeds the preset threshold, the streaming mode is enabled.

[0127] When the estimated data volume is no greater than the preset threshold, switch to normal query mode, that is, retrieve and return all data at once.

[0128] The streaming mode includes the following steps:

[0129] S1. Read data from the database line by line.

[0130] By calling the database driver's native streaming query interface, the traditional approach of loading all data into memory at once is abandoned. Streaming cursors from the database client (such as MySQL streaming cursors, PostgreSQL cursor queries, JDBC's `etchSize=Integer.MIN_VALUE`, etc.) are used, and the database server returns row data in batches. The application retrieves individual records row by row from the cursor, so the full amount of data is not loaded into memory instantly, thus avoiding large result sets at the data source level.

[0131] S2. Serialize the data line by line into JSON.

[0132] After obtaining the data from a single database row, each data row is serialized into a separate JSON fragment. This avoids serializing the entire collection at once, preventing the generation of extremely large JSON strings. Each record is JSONified as soon as it is read, forming independent, fragmented JSON fragments. There are no extremely large intermediate objects throughout the process, effectively controlling the instantaneous memory usage.

[0133] S3. Write the single-line JSON fragment mentioned above directly to the output stream.

[0134] S4. Batch refresh by threshold, such as: refresh once every 1000 rows.

[0135] The system counts and tracks the number of single-line JSON fragments written. Once a threshold is reached, a batch refresh is performed to persist the buffered data. Because memory only caches a single-line object and a buffer of 1000 lines of data, the resident memory limit remains fixed regardless of whether the final result is 100,000 or 10 million lines. Memory usage does not increase with the total amount of queried data, completely solving the problem of high memory usage in large-scale data export / query.

[0136] Step 5, as Figure 5As shown, this embodiment can achieve incremental query results.

[0137] Specifically, it includes the following steps:

[0138] S1. At the beginning of the query process (e.g., system startup, the first time the task runs), initialize the global timestamp. That is, the global timestamp is a unique progress marker for the entire chain, used to distinguish between processed data and incremental data to be processed.

[0139] S2. Query incremental new data based on timestamp conditions, filter data with timestamps greater than the previous global cursor, and only pull data newly added to the database after the last synchronization; thus filtering out all historical data before the global timestamp, avoiding repeated reading of existing data from the query level; combined with the above streaming query strategy, large batches of incremental data can also be read row by row, preventing the problem of excessive memory usage for a large amount of data at once.

[0140] S3. Data processing, namely, cleaning, storing, calculating, and forwarding the incremental data selected in this round;

[0141] S4. After all data processing is complete, update the global timestamp to ensure that the cursor only advances after successful processing and to prevent data loss. The method for updating the global timestamp is as follows: extract the maximum timestamp of all data in the current query results and update the global cursor to this maximum value. This ensures that the global timestamp is monotonically increasing, that is, the new timestamp must be greater than the old timestamp of the previous round. The cursor will only advance forward and will not roll back.

[0142] S5. Iterate through the loop until the next round of incremental query updates the timestamp, then repeat steps S2-S4.

[0143] The advantage of this design is that it uses only a single global variable to mark the synchronization timestamp (TimeStamp) throughout the entire process. Historical data with timestamps less than or equal to this global timestamp will never be filtered again. This naturally avoids duplicate consumption and calculations in the query conditions and replaces traditional pagination IDs and multi-field cursors, achieving a highly simplified logic. After each query, the timestamp is updated to the maximum timestamp of the current round, eliminating the need to update the cursor row by row. The maximum value is retrieved and updated uniformly after the entire batch of data is processed, reducing the number of database configuration updates and improving synchronization performance.

[0144] The following will demonstrate the application scenarios of the above methods in several modes.

[0145] Application Example 1: Telemetry Data Writing Process

[0146] like Figure 6 As shown, the specific telemetry data writing process is as follows:

[0147] Step 1: Telemetry data arrives at the system;

[0148] Step 2: Parse the data and extract fields such as device ID, telemetry parameter code, timestamp, and value;

[0149] Step 3: Write the data to the ClickHouse telemetry history table;

[0150] Step 4: The materialized view is automatically updated to maintain the latest data;

[0151] Step 5: If the data falls within the alarm monitoring range, write it to the preprocessing table.

[0152] Application Example 2: Latest Data Query Process

[0153] like Figure 7 As shown, the specific process for querying the latest data is as follows:

[0154] Step 1: Receive the query request, specifying the list of device IDs and the list of telemetry parameter codes;

[0155] Step 2: Construct an SQL query to retrieve data from the latest materialized view;

[0156] Step 3: Use the FINAL keyword to ensure data consistency;

[0157] Step 4: Return the query results.

[0158] Application Example 3: Historical Data Export Process

[0159] like Figure 8 As shown, the specific historical data export process is as follows:

[0160] Step 1: Receive the export request and determine the data volume;

[0161] Step 2: If the data volume does not exceed the threshold, enable the normal query mode to query and return the data in one go;

[0162] If the data volume exceeds the threshold, enable streaming mode to proceed to step 3;

[0163] Step 3: Create a database streaming reader;

[0164] Step 4: Read the data line by line and serialize it into JSON;

[0165] Step 5: Write directly to the HTTP response stream;

[0166] Step 6: Refresh the output every 1000 lines;

[0167] Step 7: Close the stream after reading is complete.

Claims

1. A satellite telemetry data management system based on ClickHouse, characterized in that: Includes a data storage module; The data storage module includes a relational business database and a ClickHouse telemetry time series database. The relational business database stores satellite business data; The ClickHouse type telemetry time series database stores satellite telemetry time series data; The ClickHouse type telemetry time series database includes a data update submodule; The data update submodule ensures that the ClickHouse type telemetry time series database always executes the latest data materialized view at the front end.

2. The ClickHouse-based satellite telemetry data management system as described in claim 1, characterized in that: The relational business database includes a telemetry data preprocessing submodule; The telemetry data preprocessing submodule preprocesses the telemetry data to form an alarm preprocessing table. The alarm preprocessing table only contains data related to alarm monitoring; The alarm monitoring data is sorted in chronological order.

3. The ClickHouse-based satellite telemetry data management system as described in claim 1, characterized in that: The sorting key of the alarm preprocessing table is designed as: TimeStamp, TelemCode, DeviceId.

4. A satellite telemetry data management system based on ClickHouse as described in claim 1, characterized in that: It also includes a data transmission module; The data transmission module includes a data prediction submodule, a normal transmission submodule, and a streaming transmission submodule; The data estimation submodule estimates the total amount of data in the query results based on the query request. When the estimated data volume exceeds the preset threshold, the streaming submodule is activated. When the estimated data volume is not greater than the preset threshold, the normal transmission submodule is activated.

5. A satellite telemetry data management system based on ClickHouse as described in claim 4, characterized in that: The data transmission of the streaming submodule includes the following steps: S1. Read data from the database line by line; S2. Serialize the data line by line into JSON; S3. Write the single-line JSON fragment generated in S2 directly into the output stream; S4. Batch refresh the data in the output stream according to the threshold.

6. A satellite telemetry data management system based on ClickHouse as described in claim 1, characterized in that: It also includes a data query module; The data query method of the data query module is as follows: SS1. Initialize the global timestamp at the beginning of the query process; SS2. Query incremental new data by timestamp condition, and filter data whose timestamp is greater than the previous round's global timestamp; SS3. Process the incremental data selected in this round; SS4. After data processing is complete, update the global timestamp to the maximum timestamp of all data in the current round of results; SS5. Iterate through the next round of incremental queries and continue repeating the steps of SS2-SS4.

7. A satellite telemetry data management method based on ClickHouse, characterized in that, Includes the following steps: S1. Construct a heterogeneous database hierarchical storage architecture, with business data stored in a relational business database and telemetry time-series data stored in a ClickHouse database; S2. Create the latest materialized view of the data and automatically maintain the latest telemetry values ​​for each device; S3. Preprocess the raw telemetry data to obtain the alarm preprocessing data table; S4. Based on the estimated results of the query data, when the estimated data volume is greater than the threshold, perform streaming transmission.

8. The satellite telemetry data management method based on ClickHouse as described in claim 7, characterized in that: In S2, the latest data materialized view uses the ReplacingMergeTree engine to sort by device ID and telemetry parameter code, automatically deduplicating and retaining the latest records.

9. A satellite telemetry data management method based on ClickHouse as described in claim 7, characterized in that: In S3, the alarm preprocessing data table only contains the fields that the alarm rules care about, and the sorting key is designed with timestamp priority; The timestamp sorting rules enable incremental query results based on timestamps; The incremental query method is as follows: S31. At the beginning of the query process, initialize the global timestamp; S32. Query incremental new data based on timestamp conditions, and filter data whose timestamp is greater than the previous round's global timestamp; S33. Perform data processing on the incremental data selected in this round; S34. After the data processing is complete, update the global timestamp to the maximum timestamp of all data in the current round of results; S35. Iterate through the loop to the next round of incremental query, and continue repeating steps S32-S34.

10. A satellite telemetry data management method based on ClickHouse as described in claim 7, characterized in that: In S4, the specific method of streaming transmission is as follows: S41. Read data from the database line by line; S42. Serialize the data line by line into JSON; S43. Write the single-line JSON fragment generated in S42 directly to the output stream; S44. Batch refresh the data in the output stream according to the threshold.

Images