Telemetry information collection and storage method based on lock-free writing and double-buffer snapshot mechanism

Abstract

This invention proposes a telemetry information acquisition and storage method based on lock-free writing and a double-buffered snapshot mechanism. The method includes: predefining a telemetry field table, generating channel enumeration, field description tables, and total record length; the business process writing business telemetry data lock-free through a shared memory double buffer; the acquisition and storage process periodically performing snapshot flipping to obtain consistent business data snapshots, while simultaneously acquiring SoC and FPGA telemetry data, merging and encoding them into headless fixed-length binary records according to the field description table; asynchronously writing the records to a file, and managing synchronization, round-robin, and circular queues according to a preset strategy. This invention eliminates mutual exclusion overhead in multi-threaded environments by writing business telemetry data lock-free through a shared memory double buffer, and completely solves the data tearing problem in traditional solutions through the snapshot flipping mechanism; the unified merging of multi-source telemetry data into headless fixed-length binary records improves storage space utilization; finally, file synchronization, round-robin, and quantity management ensure data reliability.

Description

Technical Field

[0001] This invention relates to the field of data acquisition and storage technology, and in particular to a method for acquiring and storing telemetry information based on lock-free writing and a double-buffered snapshot mechanism. Background Technology

[0002] During flight missions, UAV trackers need to simultaneously run multi-threaded algorithms such as data stream processing, target tracking, target detection, and fusion control, and interact at high speed with co-processing units such as FPGAs (Field-Programmable Gate Arrays). System debugging, performance optimization, and flight accident debriefing typically rely on the recording and analysis of large amounts of telemetry data. This telemetry data mainly includes three categories: Business telemetry: such as the intermediate state of each algorithm, execution time, processing results, etc.; SoC (System-on-a-Chip) telemetry: such as CPU / GPU / NPU / DDR frequency, temperature, voltage, core utilization, and other system health status information; FPGA telemetry: such as hardware coprocessing status information like the status of FPGA internal registers and operating parameters.

[0003] In existing technologies, embedded devices such as drone trackers have the following common problems in telemetry data acquisition and storage: (1) High overhead of multi-threaded writing: In multi-threaded business programs, multiple threads need to write data to the telemetry system at the same time. Traditional methods usually use mutex locks or queue forwarding to ensure data security, which leads to frequent blocking or jittering of business threads and a significant decrease in real-time performance. (2) Data consistency issues are prominent: When the acquisition process reads business data in shared memory, if the business thread writes at the same time, it is easy to have the data "read halfway and then overwritten", resulting in the acquisition data being torn and incomplete, and unable to be used for accurate analysis; (3) Low storage format efficiency: Due to the large number of telemetry fields and their relatively fixed layout, if a text format (such as JSON, CSV) or a binary format with a header structure is used, on the one hand, the storage overhead is large and it occupies limited storage resources; on the other hand, the upper computer parsing is complicated and affects the efficiency of data post-processing. (4) Limited storage media lifespan: Embedded devices commonly use eMMC as the storage media, which has a limited write lifespan. If the write synchronization frequency is too high or the file management is improper, it can easily lead to write amplification, accelerate the aging of the storage media, and affect the long-term reliability of the device. (5) Difficulty in file management: Some embedded devices are not powered by RTC (real-time clock) batteries and cannot obtain accurate system time. If the files are named according to timestamps, it will lead to duplicate file names or confusion, making it difficult to achieve automated management and data backtracking.

[0004] In summary, existing technologies lack a low-overhead, lock-free, consistent, easily scalable, and long-term operational telemetry acquisition and storage solution, making it difficult to meet the high real-time performance, high reliability, and long-lifespan storage requirements of complex embedded systems such as UAV trackers. Summary of the Invention

[0005] This invention proposes a telemetry information acquisition and storage method based on lock-free writing and double-buffered snapshot mechanism, which solves the problems of high multi-threaded writing overhead and poor data consistency in existing telemetry data acquisition and storage methods.

[0006] The technical solution of this invention is implemented as follows: The first aspect of this invention provides a method for acquiring and storing telemetry information based on lock-free writing and a double-buffered snapshot mechanism, comprising the following steps: A predefined telemetry field table is used to generate a channel enumeration for identifying each telemetry field, a field description table containing attribute information of each telemetry field, and the total length of a single telemetry record. The business process writes business telemetry data to the current write buffer without locking through the shared memory double buffer; The data collection and storage process periodically performs snapshot flipping to obtain consistent business data snapshots from shared memory; The data acquisition and storage process acquires SoC telemetry data and FPGA telemetry data within the same cycle, and merges and encodes the business data snapshot, SoC telemetry data and FPGA telemetry data according to the field description table to obtain a headless fixed-length binary record that conforms to the total record length. Store binary records and perform file synchronization, rotation, and quantity management according to preset strategies.

[0007] Specifically, the telemetry field table includes at least several of the following: channel number, enumeration symbol, field name, data type, byte length, and byte offset. A field definition file is automatically generated based on the telemetry field table, and the channel enumeration, field description table, and total record length are generated during compilation.

[0008] Specifically, the lock-free writing of business telemetry data by the business process includes: reading the current write buffer index, atomically incrementing the write count of the corresponding buffer by 1, confirming that the current write buffer index has not changed, atomically writing the telemetry value to the corresponding channel position in the shared memory according to the channel enumeration, and then atomically decrementing the write count by 1.

[0009] Furthermore, before writing, the business process performs type and range checks on the written value based on the field description table. If the type is invalid or the value is out of bounds, the write is rejected and an error is returned.

[0010] Specifically, the snapshot flipping process of the acquisition and storage process includes: flipping the write buffer index to switch the business process to another buffer, waiting for the write count of the original buffer to return to zero or time out, and then copying the original buffer as a business data snapshot.

[0011] Furthermore, the waiting timeout threshold is set. If the timeout occurs, the system returns to a timeout state but still outputs a snapshot, without interrupting the subsequent acquisition and storage process.

[0012] Specifically, the acquisition of FPGA telemetry data includes: reading a fixed-length data block at once via UART, adjusting the byte order of the data block into groups of n bytes, and writing it into the corresponding channel according to the field description table; when the FPGA reading fails continuously to the threshold, temporarily shutting down the FPGA data source and attempting to reconnect, while continuing to acquire SoC telemetry data and service data snapshots.

[0013] Specifically, methods for storing binary records and performing file synchronization, rotation, and quantity management according to preset strategies include: The binary record is pushed into a lock-free queue, and then written to the file in an append manner after being dequeued asynchronously by an independent storage thread. File synchronization is performed according to the preset synchronization cycle, forcibly writing file data to the storage medium; Create new files in the storage medium in a rotating manner according to the preset segment duration, and name the new files with an incrementing sequence number; Files are managed using a circular queue in the specified directory, retaining a maximum of a preset number of files. When the preset number is exceeded, the file with the smallest sequence number is automatically deleted.

[0014] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the acquisition and storage method.

[0015] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the acquisition and storage method.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention achieves lock-free writing of business telemetry data through a double buffer of shared memory, eliminating the mutual exclusion overhead in a multi-threaded environment and ensuring that the real-time performance of business threads is not affected by telemetry acquisition; it obtains a consistent view of business telemetry data through a snapshot flipping mechanism, which completely solves the problem of data tearing in traditional solutions; multi-source telemetry data are merged and encoded into headless fixed-length binary records in the same period according to a unified field description table, which not only improves the utilization of storage space but also simplifies the parsing process of the host computer; finally, file synchronization, rotation and quantity management are performed through preset strategies, which effectively extends the life of storage media while ensuring data reliability; (2) The present invention uses the telemetry field table as the only data source, automatically generates channel enumeration, field description table and total record length, and expands it during compilation to form a unified description structure in the program; this mechanism ensures that the writing rules on the business side are strictly consistent with the encoding rules on the acquisition side, avoids the misalignment errors that may be introduced by manual maintenance of field definitions, and at the same time makes the addition, deletion, modification and query operations of telemetry fields do not require modification of business code, which significantly improves the scalability and maintainability of the system; (3) This invention achieves lock-free concurrent writing in a multi-threaded environment by introducing a double-buffer index and an independent write count for each buffer. The business thread atomically increments the write count before writing and atomically decrements it after writing. Combined with the index double confirmation mechanism before and after writing, it not only ensures the safety of multi-threaded writing but also avoids the context switching overhead and thread blocking problems caused by traditional mutex locks, thus minimizing the impact of the telemetry acquisition system on the performance of the business program. (4) This invention switches the business write to the new buffer by flipping the write buffer index and waits for the original buffer to be copied as a whole after the write count is zero to obtain a snapshot of the business data; at the same time, a timeout protection mechanism is introduced, which still outputs a snapshot as complete as possible and continues the subsequent process when the wait timeout occurs, ensuring that the acquisition link will not be interrupted due to waiting while ensuring data consistency. (5) The present invention adopts a dual-thread mode of “encoding and enqueuing by the acquisition thread and asynchronous dequeuing and disk writing by the storage thread”, which decouples data encoding from file writing and avoids the storage I / O operation from blocking the acquisition process; at the same time, combined with timed synchronization, timed rotation and circular queue management strategies, the write amplification effect is effectively controlled under the premise of ensuring that data is not lost, and the life of storage media such as eMMC is extended; the use of incremental sequence number to name the file completely gets rid of the dependence on the real-time clock, so that automated file management can be achieved even without RTC devices. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the telemetry information acquisition and storage method based on lock-free writing and double-buffered snapshot mechanism of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the process of lock-free writing of a business process to shared memory in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the process of reading a consistent snapshot of the acquisition process in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram illustrating the asynchronous storage and file lifecycle management process in an embodiment of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] Reference Figure 1 The first aspect of this invention provides a method for acquiring and storing telemetry information based on lock-free writing and a double-buffered snapshot mechanism, comprising the following steps: Step 1: Standardize the definition of telemetry fields A predefined telemetry field table is used to generate a channel enumeration for identifying each telemetry field, a field description table containing attribute information of each telemetry field, and the total length of a single telemetry record.

[0024] Specifically, the telemetry field table, serving as the sole data source, includes at least the channel number, enumeration symbol, field name, data type, byte length, and byte offset. A field definition file is automatically generated based on this field table and is generated by macro expansion during compilation. Channel enumeration assigns a unique numerical identifier to each telemetry field, and the business side uses this enumeration value to locate the write position when making calls. The field description table contains information such as the data type, byte length, and byte offset of each field, which is used for type checking during writing and layout during encoding. The total record length, i.e. the fixed number of bytes in a single complete telemetry record, is used for type checking, range checking, and fixed-length packing by describing the channel index field during program runtime.

[0025] This invention uses the telemetry field table as a single data source to automatically generate a unified description structure within the program, ensuring that the writing rules on the business side and the encoding rules on the acquisition side are strictly consistent, avoiding misalignment errors that may be introduced by manually maintaining field definitions; at the same time, it enables the addition, deletion, modification and query operations of telemetry fields to only require modification of the field table without changing the business code, which significantly improves the scalability and maintainability of the system.

[0026] Step 2: Lock-free writing of business processes to shared memory

[0027] The business process writes business telemetry data to the current write buffer without locking through the shared memory double buffer.

[0028] Specifically, the acquisition and storage process first creates shared memory and writes consistent metadata, including: identifier, version number, number of channels, and field pattern checksum; and initializes double buffers, write buffer indexes, and write counts.

[0029] Specifically, such as Figure 2 As shown, when the business process writes business telemetry data without locks, it operates according to the following steps: First, it reads the current write buffer index and atomically increments the write count of that buffer, indicating that a thread is writing; then, it reads the current write buffer index again and compares it with the previously read index; if the two read indices are inconsistent, it means that the acquisition process performed a snapshot flip after the first index read, causing the write buffer to be switched. At this time, it is necessary to roll back the previously incremented write count (at this time, the original write buffer needs to be snapshot read, and writing can no longer be done in the original write buffer, so the write count of the original buffer needs to be rolled back to avoid unnecessary waiting), and restart the entire writing process to ensure that data is always written to the currently active write buffer. If the two read indices are the same, it means that the buffer has not been switched, and it is safe to atomically write the telemetry value to the corresponding channel position in shared memory; after the writing is completed, it atomically decrements the write count of that buffer; this process allows multiple business threads to concurrently write to different channels without using mutex locks.

[0030] Furthermore, before writing, the business process performs type and range checks on the written value according to the field description table: It performs corresponding numerical range checks based on the data type defined in the field. For example, unsigned 8-bit integers are limited to 0 to 255, signed 8-bit integers are limited to -128 to 127, unsigned 16-bit integers are limited to 0 to 65535, signed 16-bit integers are limited to -32768 to 32767, unsigned 32-bit integers are limited to 0 to 4294967295, and signed 32-bit integers are limited to -2147483648 to 2147483647. If a type mismatch, channel number out of bounds, invalid field, or value exceeding the allowed range is found, an error code is returned and the write operation is rejected; only values ​​that pass the validity check are allowed to be written to shared memory.

[0031] This invention achieves lock-free concurrent writing in a multi-threaded environment through dual-buffer indexing and independent write counting for each buffer. This avoids the context switching overhead and thread blocking issues associated with traditional mutex locks, minimizing the performance impact of the telemetry acquisition system on business applications. Simultaneously, the type and range checks before writing effectively prevent illegal data generated by program errors from entering the telemetry records, ensuring the validity and reliability of the acquired data.

[0032] Step 3: Reading Consistency Snapshots of the Acquisition Process

[0033] The data collection and storage process periodically performs snapshot flipping to obtain consistent business data snapshots from shared memory.

[0034] Specifically, such as Figure 3 As shown, the acquisition thread triggers a snapshot flip operation at a fixed period (e.g., 33ms). When the acquisition storage process performs a snapshot flip, it switches the business process to another buffer by flipping the write buffer index, so that the business write is immediately switched to the new buffer. Then wait for the write count of the original buffer to return to zero, that is, wait for all business threads that are performing write operations in the old buffer to complete; wait for the set timeout threshold (e.g., 5ms) to avoid infinite waiting due to individual thread exceptions; When the write count reaches zero or a timeout occurs, the original buffer is copied to the current cycle cache as a snapshot of the business data. If a timeout occurs, a timeout status is returned, but a snapshot that is as complete as possible is still output to avoid interrupting the subsequent acquisition and storage process and to ensure the continuity of the acquisition chain.

[0035] This invention switches business writes to a new buffer by flipping the write buffer index, and waits for the original buffer to be copied in its entirety after the write count reaches zero to obtain a snapshot of the business data. This ensures a consistent view of the read data without locking, completely solving the data tearing problem of "being overwritten halfway through a read" in traditional solutions. Simultaneously, a timeout protection mechanism is introduced to ensure that the acquisition link is not interrupted due to waiting, achieving a balance between high reliability and high availability.

[0036] Step 4: Acquisition and Merging of Multi-Source Telemetry Data

[0037] The data acquisition and storage process collects SoC telemetry data and FPGA telemetry data within the same cycle, and merges and encodes the business data snapshot, SoC telemetry data, and FPGA telemetry data according to the field description table to obtain a headless fixed-length binary record that conforms to the total record length.

[0038] Specifically, within a fixed period, a service snapshot is first acquired, followed by the collection of SoC telemetry data: temperature, CPU / GPU / NPU / DDR frequency, core utilization, voltage, etc., which are then written to the cache by channel. FPGA telemetry data is then collected: a fixed-length data block (e.g., 160 bytes) is read at once via UART (Universal Asynchronous Receiver / Transmitter). After adjusting the byte order of the data block into groups of 4 bytes, it is written to the corresponding channel according to the field description table. The service snapshot, SoC telemetry, and FPGA telemetry are then merged into a complete cache record for this period.

[0039] The acquisition of FPGA telemetry data also includes a fault tolerance mechanism: when the FPGA read fails continuously to a preset threshold (e.g., 10 times), the FPGA data source is temporarily shut down and subsequent reconnection is attempted, while SoC telemetry data and service data snapshots continue to be acquired, and other sources continue to acquire data without interrupting the overall link.

[0040] This invention completes the acquisition and merging of multi-source telemetry data within the same cycle, ensuring the time correlation between data sources. Encoding data into headless, fixed-length binary records using field description tables improves storage space utilization and simplifies the host computer parsing process. The fault tolerance and automatic recovery mechanism of the FPGA data source effectively isolates the impact of peripheral anomalies on the overall acquisition chain, enabling the system to continuously record critical telemetry data even when some data sources fail, greatly improving the system's robustness and field adaptability.

[0041] Step 5: Asynchronous Storage and File Lifecycle Management

[0042] Store binary records and perform file synchronization, rotation, and quantity management according to preset strategies, such as... Figure 4 As shown, the specific methods include: The encoded binary record is pushed into a lock-free queue with a single producer and single consumer (default capacity 2048). It is then dequeued asynchronously by an independent storage thread and written to a file in an append-only manner. This decouples data encoding from file writing, avoiding the blocking of the acquisition process by storage I / O operations.

[0043] File synchronization is performed according to a preset synchronization cycle: file data is forcibly written to the storage medium (such as eMMC) at preset intervals (e.g., 1 second) to ensure that data within one synchronization cycle is lost at most in the event of an unexpected power outage, thus achieving a balance between reliability and write lifespan.

[0044] Create new files in rotation on the storage medium according to preset segment durations: for example, create a new file every 5 hours. The file name is named with an incrementing sequence number and does not depend on the real-time clock (RTC). The segment duration of a single file can be configured as needed to avoid the management difficulty caused by a single file being too large.

[0045] Files are managed using a circular queue in a specified directory: a maximum of a preset number of files (e.g., 10) are retained, and the file with the smallest sequence number is automatically deleted when the number of files exceeds the limit, thus achieving automatic management of storage capacity.

[0046] This invention employs a dual-thread model of "encoding and queuing by the acquisition thread, and asynchronous dequeueing and disk writing by the storage thread," avoiding storage I / O blocking the acquisition process. Combined with timed synchronization, timed rotation, and circular queue management strategies, it effectively controls write amplification while ensuring no data loss, extending the lifespan of storage media such as eMMC. The use of incrementing sequence numbers for file naming completely eliminates dependence on real-time clocks, enabling automated file management even without RTC devices. The circular queue mechanism ensures orderly reclamation of storage space, preventing system anomalies caused by disk fullness.

[0047] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the acquisition and storage method.

[0048] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the acquisition and storage method.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A telemetry information collection and storage method based on lock-free write and double-buffered snapshot mechanism, characterized in that, Includes the following steps: A predefined telemetry field table is used to generate a channel enumeration for identifying each telemetry field, a field description table containing attribute information of each telemetry field, and the total length of a single telemetry record. The business process writes business telemetry data to the current write buffer without locking through the double buffer of shared memory. The business process writes business telemetry data without locking, which includes: reading the current write buffer index, atomically incrementing the write count of the corresponding buffer by 1, confirming that the current write buffer index has not changed, atomically writing the telemetry value to the corresponding channel position of shared memory according to the channel enumeration, and then atomically decrementing the write count by 1. The data acquisition and storage process periodically performs snapshot flipping to obtain consistent business data snapshots from shared memory. The snapshot flipping includes: flipping the write buffer index to switch the business process to another buffer, waiting for the write count of the original buffer to return to zero or time out, and then copying the original buffer as a business data snapshot. The data acquisition and storage process acquires SoC telemetry data and FPGA telemetry data within the same cycle, and merges and encodes the business data snapshot, SoC telemetry data and FPGA telemetry data according to the field description table to obtain a headless fixed-length binary record that conforms to the total record length. Store binary records and perform file synchronization, rotation, and quantity management according to preset strategies. Specific methods include: The binary record is pushed into a lock-free queue, and then written to the file in an append manner after being dequeued asynchronously by an independent storage thread. File synchronization is performed according to the preset synchronization cycle, forcibly writing file data to the storage medium; Create new files in the storage medium in a rotating manner according to the preset segment duration, and name the new files with an incrementing sequence number; Files are managed using a circular queue in the specified directory, retaining a maximum of a preset number of files. When the preset number is exceeded, the file with the smallest sequence number is automatically deleted.

2. The method of claim 1, wherein, The telemetry field table includes at least several of the following: channel number, enumeration symbol, field name, data type, byte length, and byte offset. A field definition file is automatically generated based on the telemetry field table, and the channel enumeration, field description table, and total record length are generated during compilation.

3. The method of claim 1, wherein the method further comprises: Before writing, the business process performs type and range checks on the written value based on the field description table. If the type is invalid or the value is out of bounds, the write is rejected and an error is returned.

4. The method of claim 1, wherein, The waiting timeout threshold is set. If the timeout occurs, the system returns to the timeout status but still outputs a snapshot, without interrupting the subsequent acquisition and storage process.

5. The method of claim 1, wherein, The acquisition of FPGA telemetry data includes: reading a fixed-length data block at once via UART, adjusting the byte order of the data block into groups of n bytes, and writing it into the corresponding channel according to the field description table; when the FPGA read fails continuously to the threshold, temporarily shutting down the FPGA data source and attempting to reconnect, while continuing to acquire SoC telemetry data and service data snapshots.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the acquisition and storage method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the acquisition and storage method as described in any one of claims 1 to 5.

Images