Mass telemetry data feature value processing method based on time fragmentation and double compression

Abstract

A massive telemetering data characteristic value processing method based on time fragmentation and double compression comprises the following steps: receiving satellite telemetering original stream data in real time, performing formatting analysis, extracting a baseband timestamp, a telemetering code and an engineering value field, and generating a standardized telemetering engineering value frame; fragmenting the telemetering engineering value frame based on a timestamp, determining a triple composed of the timestamp, a date to which the timestamp belongs and a telemetering code as a storage sub-partition key, and dynamically allocating data to a corresponding storage partition according to the key for storage in a disk; carrying out historical archiving on the stored telemetering engineering value frames by adopting double-compression processing; constructing a shared bulletin board task model, and registering all to-be-calculated feature tasks to a shared bulletin board queue by taking a combination of a telemetering code, a processing time slice and a processing identifier as a bulletin board key value; and the thread pool obtains task entries in the shared bulletin board queue in parallel based on a shared lock queue mechanism, and reads satellite telemetering point information from the shared memory.

Description

Feature value processing method for massive telemetry data based on time segmentation and dual compression Technical Field

[0001] This invention relates to a method for processing feature values ​​of massive telemetry data based on time slicing and dual compression, belonging to the field of spacecraft telemetry data processing. Background Technology

[0002] Satellite telemetry is core data for satellite tracking and control, serving as a key indicator for assessing satellite status and a primary source of data for data exchange between ground control stations and the satellite. Furthermore, feature extraction from satellite telemetry data enables crucial subsequent operations such as loop-to-loop comparison, trend prediction, health assessment, and risk analysis.

[0003] Traditional satellite telemetry processing systems process satellite source code frame by frame according to a time frame sequence, decoding the data engineering value of the current frame or multiple frames. When faced with massive amounts of data, traditional telemetry data processing systems have the following technical problems: 1) huge data storage space consumption and high storage costs; 2) low data processing efficiency and serious response delays; 3) difficulty in ensuring data integrity, consistency and reliability; 4) poor system scalability, making it difficult to cope with the continuous growth of data volume; 5) low efficiency in historical data query and difficulty in feature value extraction.

[0004] Traditional synchronous blocking processing methods can no longer meet the real-time processing needs of massive telemetry data, and a new technical solution is urgently needed to solve the above problems. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a method for processing feature values ​​of massive telemetry data based on time slicing and dual compression, which solves the problems of wasted storage space, low processing efficiency, and difficulty in extracting feature values ​​of massive data over long time scales in the processing of massive telemetry data.

[0006] The technical solution of this invention is: a telemetry data feature value processing system based on time-slicing and dual compression technology, comprising: S1, receiving raw satellite telemetry stream data in real time, performing formatted parsing, extracting baseband timestamp, telemetry code, and engineering value fields, and generating standardized telemetry engineering value frames; S2, sliced ​​the telemetry engineering value frames based on the timestamp, determining the triplet composed of the timestamp, the date, and the telemetry code as the storage sub-partition key, and dynamically allocating data to the corresponding storage partition for disk storage according to the key; S3, performing dual compression processing on the disk-stored telemetry engineering value frames for historical archiving; S4, constructing a shared bulletin board task model, using a combination of telemetry code, processing time slice, and processing identifier as... The bulletin board key value registers all feature tasks to be calculated into the shared bulletin board queue; S5, the thread pool retrieves task entries from the shared bulletin board queue in parallel based on the shared lock queue mechanism, and reads satellite telemetry point information from shared memory; if there are unprocessed task entries, the distributed memory lock is acquired, the bulletin information within the current time interval is extracted, and its status is updated; S6, input data is loaded according to the task type; S7, based on the input data, feature value calculation is performed concurrently using a multi-threaded pool, and the calculation results are persisted to the feature value database in real time; S8, after the calculation is completed, the distributed memory lock is acquired, the status of the corresponding task entry in the shared bulletin board is updated, the processing time is reset to the current time, and the lock is released to complete the task closed-loop processing.

[0007] Preferably, the telemetry engineering value frames that have been stored on disk are subjected to dual compression processing for historical archiving. Specifically, the timestamps are incrementally encoded to remove redundant time information; the engineering value fields are converted into the underlying binary format and compressed using entropy encoding; and then the compressed historical data blocks are written to the archive storage area according to the time fragmentation principle.

[0008] Preferably, when converting the engineering value field to the underlying binary format, a custom compression algorithm or a general entropy coding algorithm is used.

[0009] Preferably, when loading input data according to the task type: if the task is to calculate minute-level feature values, the data of the previous time slice is extracted from the real-time telemetry data cache; if the task is to calculate hourly, daily, or yearly feature values, the corresponding input data is loaded from the stored historical feature data.

[0010] Preferably, the persistent storage adopts a streaming data ingestion method: multiple insert requests are concatenated into a single streaming session, and submitted to the database in batches through an asynchronous streaming write mechanism.

[0011] Preferably, when multiple threads concurrently perform feature value calculations, the calculation types include: mean, range, and variance.

[0012] Preferably, the method further includes a data reading step: in response to a query request, the feature value database is queried in batches by time slice, and the query results are aggregated and returned in the form of streaming data blocks. The query process is executed through a long connection maintained by a connection reuse pool.

[0013] Preferably, the method runs on a non-blocking event-driven I / O model, using kernel multiplexing technology to listen to message bus events and database I / O events, allowing business threads to handle other tasks during I / O waiting periods.

[0014] Compared with existing technologies, this invention has the following advantages: (1) The dual compression strategy achieves overall compression, significantly reducing the storage cost of telemetry historical data; (2) The time sharding-partitioning mechanism limits the query range to a very small subset, reducing retrieval latency; (3) The "shared bulletin board" parallel computing model enables the throughput to increase linearly with the node scale, breaking through the single-machine bottleneck, and can be used to process feature values ​​in a layer-by-layer progressive manner, which can make the high-dimensional feature value calculation uniform; (4) Streaming links and connection multiplexing reduce frequent short connections, significantly reducing system CPU usage and jitter. Non-blocking I / O combined with kernel multiplexing improves I / O concurrency and data read / write efficiency. Attached Figure Description

[0015] Figure 1 is a structural diagram of the telemetry regrouping system of the present invention. Detailed Implementation

[0016] This invention implements a feature value processing method for massive telemetry data based on time-slicing and dual compression. It proposes a telemetry partitioned storage and telemetry dual compression method, a fast and efficient concurrent processing method for telemetry feature values, and a fast and efficient strategy for telemetry data access. This solves problems such as wasted storage space, low processing efficiency, and difficulty in feature value extraction over long time scales in massive telemetry data processing.

[0017] The specific technical solution is as follows: The system receives satellite telemetry data streams in real time via a message bus. First, it performs formatted parsing according to the telemetry decoding procedure, extracting fields such as baseband timestamp, telemetry code, and engineering value, and generating a telemetry engineering value frame in a unified format. Then, a timestamp-based fragmentation strategy is adopted, using the timestamp, the associated date, and the telemetry code together to determine the storage sub-partition key, achieving balanced partitioning of massive real-time telemetry data.

[0018] During the historical archiving phase, the system performs "double compression" on the telemetry engineering value frames that have been stored on disk: the timestamp field is compressed using incremental encoding to remove redundant time information, and the engineering value field is compressed using entropy compression based on binary format to generate high-density historical data blocks. The compressed historical blocks are still written to the archiving area in time-segmented format to ensure accurate retrieval later.

[0019] To enhance concurrent processing capabilities, the system proposes a "shared bulletin board" task model. Using "telemetry code + processing time slice + processing identifier" as the bulletin board key, all feature value tasks to be calculated are queued and processed uniformly. The thread pool concurrently pulls entries from a shared lock queue and performs parallel computation of feature values ​​from multiple telemetry points using a divide-and-conquer strategy, allowing for horizontal scaling to linearly improve throughput.

[0020] The system first reads the telemetry point information for all satellites and checks if the relevant information already exists in the shared memory. If it does, the system initializes the shared memory, saving the telemetry code, processing time slice, and processing identifier. Then, the system loops and waits to acquire the distributed memory lock. Upon successful acquisition, it iterates through the bulletin board, retrieving unprocessed bulletin information within the current time interval. The system updates the bulletin board information and releases the shared memory lock. Next, if the feature value is a minute feature value, it retrieves the real-time telemetry data from the previous minute; if it is an hour, day, or day feature value, it retrieves the feature value data from the previous time interval. The system concurrently calculates the telemetry feature values ​​using a multi-threaded pool and persists them. Finally, it waits for the distributed memory lock again, updates the bulletin board data, resets the processing time to the current time, and releases the shared memory lock.

[0021] The system rewrote the database insertion logic and introduced a streaming data format to encapsulate the streaming process. Specifically, the write path employs a strategy of concatenating multiple insert requests into a single streaming session, leveraging the asynchronous nature of the streaming data format to achieve efficient batch writing. This approach not only improves data writing efficiency but also reduces the overhead and memory jitter caused by frequent connection establishment.

[0022] In the design of the read path, we aggregate batch query results by time slice and return the data in the form of streaming blocks. Through a long-lived connection multiplexing pool, the system maintains a small number of persistent connections, further reducing connection costs. At the I / O layer, the system fully adopts a non-blocking event-driven model, utilizing kernel multiplexing technology to monitor the message bus. This ensures that the system's business threads are almost never idle while waiting for I / O, enabling high-concurrency read and write operations.

[0023] Example: This example discloses a method for processing feature values ​​of massive telemetry data based on time slicing and dual compression. It is a satellite telemetry data processing method based on a streaming link and shared bulletin board model, suitable for real-time reception, calculation, and archiving of satellite telemetry and control data. The system achieves high throughput and low latency telemetry data processing capabilities through end-to-end streaming data links, dual compression storage mechanisms, and distributed parallel computing structures.

[0024] For example, when the telemetry source code of a satellite is received at a certain point in time, the format is as shown in Table 1, which serves as the data source for the entire system.

[0025] Table 1. Telemetry data for Satellite 1

[0026] The system is responsible for receiving satellite telemetry data in real time via the message bus and performing formatted parsing according to the telemetry decoding procedure. The system extracts the baseband timestamp, telemetry code, and engineering value fields from the raw stream data to generate a unified "telemetry engineering value frame" data structure.

[0027] The system employs a timestamp-based sharding strategy, using a triplet of timestamp, date, and telemetry code to define the storage sub-partition key. This key enables dynamic and balanced partitioning of telemetry data, effectively supporting high-concurrency storage and retrieval of massive amounts of data.

[0028] During the historical archiving phase, the system employs a dual compression strategy for the telemetry engineering value frames already stored on disk to increase data density and reduce storage costs: Differential encoding is used for continuous time fields to remove redundant time information and reduce storage footprint in the time dimension. Numerical engineering value fields are converted to an underlying binary format, and a custom compression algorithm or general entropy encoding is used to achieve high-density historical data blocks. After compression, the system still writes historical blocks to the archiving area according to the time-sharding principle to support subsequent precise retrieval at the time granularity.

[0029] To meet the concurrent computation requirements of high-dimensional feature values, the system introduces a "shared bulletin board" task model: a triple of "telemetry code + processing time slice + processing identifier" is used as the bulletin board key; all feature tasks to be computed are registered in a unified bulletin board queue; a thread pool acquires task entries in parallel based on a shared lock queue mechanism; the system reads telemetry point information for all satellites from shared memory (Redis, a memory-level database). If there are unprocessed entries, the system acquires the distributed memory lock, retrieves the bulletin information within the current time interval, and updates the status. When the task is a sub-feature value (minute-level), the system retrieves the data from the previous time slice from the real-time telemetry data cache; when the task is an hourly, daily, or yearly feature value, the system loads the input from the stored historical feature data. The multi-threaded pool concurrently computes feature values ​​(mean, range, variance, etc.), and the calculation results are persisted to the database immediately. After the calculation is complete, the system acquires the distributed memory lock again, updates the bulletin board status, resets the processing time to the current time, and releases the lock, thus achieving task closure.

[0030] The system restructured the database insertion logic and introduced a streaming data format to achieve streaming link encapsulation: multiple insertion requests are chained into a single streaming session; batch submission is achieved through an asynchronous streaming write mechanism to achieve efficient data insertion; and the CPU overhead and memory jitter caused by frequent connection establishment are significantly reduced.

[0031] During the read phase, the system aggregates the results of batch queries based on time slices and sends them back in streaming blocks. By introducing a connection multiplexing pool, the system maintains only a small number of long-lived connections, effectively reducing connection costs and request latency. The I / O layer fully adopts a non-blocking event-driven model, utilizing kernel multiplexing technology to monitor the message bus and database I / O events. This mechanism ensures that business threads have almost no idle cycles while waiting for I / O, thereby achieving high-concurrency, low-latency data transmission and processing.

[0032] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0033] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for processing feature values ​​of massive telemetry data based on time segmentation and dual compression, characterized in that... include: S1. Receive raw satellite telemetry stream data in real time, perform formatted parsing, extract baseband timestamp, telemetry code and engineering value fields, and generate standardized telemetry engineering value frames; S2. The telemetry engineering value frames are fragmented based on timestamps. The triple consisting of timestamp, date, and telemetry code is used as the storage sub-partition key. Data is dynamically allocated to the corresponding storage partitions for disk storage based on this key. S3. The telemetry engineering value frames that have been stored on disk are archived using dual compression. S4. A shared bulletin board task model is constructed, using a combination of telemetry code, processing time slice, and processing identifier as the bulletin board key value. All feature tasks to be calculated are registered in the shared bulletin board queue. S5. The thread pool obtains task entries in the shared bulletin board queue in parallel based on the shared lock queue mechanism and reads satellite telemetry point information from shared memory. If there are unprocessed task entries, the distributed memory lock is acquired, the bulletin information in the current time interval is extracted, and its status is updated. S6. Load input data according to task type; S7. Based on the input data, use a multi-threaded pool to perform feature value calculation concurrently, and store the calculation results in the feature value database in real time. S8. After the calculation is completed, acquire the distributed memory lock, update the status of the corresponding task entry in the shared bulletin board, reset the processing time to the current time, release the lock, and complete the task closed-loop processing.

2. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: For telemetry engineering value frames that have been stored on disk, a dual compression process is used for historical archiving. Specifically, the timestamps are incrementally encoded to remove redundant time information; the engineering value fields are converted into the underlying binary format and compressed using entropy encoding; and then the compressed historical data blocks are written to the archive storage area according to the time-segmentation principle.

3. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 2, characterized in that: When converting engineering value fields to the underlying binary format, a custom compression algorithm or a general entropy coding algorithm is used.

4. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: When loading input data based on task type: if the task is to calculate minute-level feature values, the data from the previous time slice is extracted from the real-time telemetry data cache; if the task is to calculate hourly, daily, or yearly feature values, the corresponding input data is loaded from the stored historical feature data.

5. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: The persistent storage uses a streaming data ingestion method, which involves concatenating multiple insert requests into a single streaming session and submitting them to the database in batches through an asynchronous streaming write mechanism.

6. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: When multiple threads concurrently perform feature value calculations, the calculation types include: mean, range, and variance.

7. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: The method also includes a data reading step: in response to a query request, the feature value database is queried in batches by time slice, and the query results are aggregated and returned in the form of streaming data blocks. The query process is executed through a long connection maintained by a connection reuse pool.

8. The method for processing feature values ​​of massive telemetry data based on time slicing and dual compression according to claim 1, characterized in that: The method runs on a non-blocking event-driven I / O model, using kernel multiplexing technology to listen for message bus events and database I / O events, allowing business threads to handle other tasks while waiting for I / O.

Images