High-speed data storage method and system based on whole second frame boundary and two-stage cache

Abstract

The invention provides a high-speed data storage method and system based on a whole-second frame boundary and a two-stage cache, and relates to the technical field of reliable storage of high-speed data, and the method comprises the following steps: receiving an external whole-second event signal, generating a unique whole-second frame boundary token based on a rising edge of the whole-second event signal, and storing the unique whole-second frame boundary token in the two-stage cache; the current UTC timestamp and the incremental global frame number are synchronously latched, and a unified whole-second time reference is established; writing and carrying the acquired data by adopting a two-stage cache; and after the data carrying time window is ended, entering a medium writing time window, and writing the generated complete data frames in the second-level cache into a physical storage medium in batches in sequence to complete reliable and persistent storage of the data. According to the invention, through an accurate time sequence scheduling and data handling mechanism, the high efficiency and consistency of data acquisition and storage in a high sampling rate environment are ensured.

Description

Technical Field

[0001] This invention relates to the field of high-speed reliable data storage technology, specifically to a high-speed data storage method and system based on whole-second frame boundaries and two-level caching. Background Technology

[0002] In the design of semi-airborne transient electromagnetic receiving systems, to effectively acquire electromagnetic field changes of UAVs at high flight speeds, the receiving system typically employs a high sampling rate for data acquisition. In high-speed data receiving systems, the system sampling rate is in the hundreds of kilobytes per second, the ADC resolution is usually greater than 24 bits, and the instantaneous throughput of the acquisition link is very high. However, the write rate of the storage medium is usually limited by jitter and file system overhead, which can easily lead to problems such as bus contention, write jitter, and half-frame residue, resulting in data packet loss, jump points, and other phenomena, seriously affecting the integrity of the acquired data.

[0003] Current data storage technologies in semi-airborne transient electromagnetic receiving systems generally adopt a ring buffer plus background writing method. However, these solutions have the following drawbacks: first, they lack a unified frame boundary identifier at the whole second level; second, they fail to effectively handle the semantics of whole second data transfer; and third, they lack mutual exclusion timing scheduling for DMA, data transfer, and media writing processes, making it difficult to maintain zero packet loss and data consistency during long-term operation.

[0004] Therefore, existing high-throughput, high-sampling-rate data storage technologies in semi-airborne transient electromagnetic receiving systems suffer from packet loss and data inconsistency issues. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a high-speed data storage method and system based on whole-second frame boundaries and a two-level buffer. Through precise timing scheduling and data transfer mechanisms, it ensures the efficiency and consistency of data acquisition and storage in high-sampling-rate environments.

[0006] According to some embodiments, the present invention adopts the following technical solution: High-speed data storage methods based on whole-second frame boundaries and two-level buffering include: Receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. A two-level cache is used to write and move the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, in the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing timestamp and frame number is generated in the second-level cache. After the data transfer window ends, the media writing window begins, where complete data frames generated in the secondary cache are written sequentially in batches to the physical storage medium, thus completing the reliable persistent storage of the data.

[0007] According to some embodiments, the present invention adopts the following technical solution: A high-speed data storage system based on whole-second frame boundaries and two-level buffering includes: The boundary establishment module is configured to: receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. The data caching module is configured to use a two-level cache for writing and moving the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, within the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing a timestamp and frame number is generated in the second-level cache. The data persistence module is configured to: after the data transfer window ends, enter the media write window, and sequentially write the complete data frames generated in the secondary cache into the physical storage medium in batches to complete the reliable persistent storage of data.

[0008] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the high-speed data storage method based on whole-second frame boundaries and a two-level cache.

[0009] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the high-speed data storage method based on whole-second frame boundaries and a two-level cache.

[0010] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the high-speed data storage method based on whole-second frame boundaries and two-level buffering.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Precise frame boundary generation and time synchronization mechanism: This invention generates precise whole-second frame boundaries and numbers through an external time source (such as 1PPS rising edge of a whole second), and ensures that a unique and auditable frame boundary is generated every second without intervening in the system-level timekeeping and frequency calibration through minimum pulse width discrimination, digital jitter reduction and anti-re-entry protection window technology. This mechanism provides a unified time reference and precise trigger signal for high-speed data acquisition systems, ensuring the time consistency and precise synchronization of data.

[0012] (2) Two-level cache structure and uninterrupted data transfer mechanism: This invention adopts a two-level cache structure of L1 (high-speed access cache) and L2 (framed preparation cache), combined with mutual exclusion window technology, to ensure that the data of the previous second is transferred from the first-level cache to the second-level cache in one go and without interruption within a dedicated transfer window. This mechanism synchronously generates a complete frame header containing timestamp, frame number, validity bitmap and multi-layer check. Through a strict data submission mechanism, it eliminates half-frame residue and cross-frame mixed writing problems, ensures data consistency and integrity under high sampling rate conditions, and ensures zero packet loss and jump-point-free data transmission. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0014] Figure 1 This is a flowchart of the method in Example 1. Figure 2 The flowchart shows the three mutually exclusive time windows in Example 1. Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0016] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0017] Example 1 One embodiment of the present invention provides a high-speed data storage method based on whole-second frame boundaries and a two-level buffer. Through precise timing scheduling and data transfer mechanisms, it ensures the efficiency and consistency of data acquisition and storage in high-sampling-rate environments. This method is suitable for high-sampling-rate data storage such as semi-aerospace transient electromagnetic (SATEM) data. Figure 1 As shown, it includes: S1: Receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. S2: A two-level cache is used to write and move the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, in the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing timestamp and frame number is generated in the second-level cache. S3: After the data transfer window ends, the media write window begins, and the complete data frames generated in the secondary cache are written sequentially in batches to the physical storage medium to complete the reliable persistent storage of the data.

[0018] This involves three aspects of technology: 1. Whole second frame boundary Using the rising edge of an external time source (such as GNSS 1PPS) as a reference, after cross-clock domain synchronization, digital jitter reduction, and anti-reentry window processing, a unique integer-second frame boundary token is generated. This token is then simultaneously latched with a UTC timestamp and the global frame number is incremented, thus establishing a unified integer-second time reference for the entire data link. This mechanism supports enabling local timekeeping logic when external signals are abnormal, ensuring continuous generation of frame boundaries and continuous numbering.

[0019] 2. Two-level cache A collaborative structure of Level 1 (L1) and Level 2 (L2) caches is adopted: L1 is a circular cache located in system memory, which continuously receives acquisition data packets via DMA through the AXI4-Stream interface; L2 is a linear cache area with pre-allocated frame slots, used to aggregate L1 data by second and encapsulate it into complete data frames; after triggering at the whole second boundary, the data of the previous second is transferred from L1 to L2 in a dedicated transfer window, and a frame header containing timestamp, frame number, validity bitmap and verification information is generated synchronously, realizing lossless conversion of data from streaming reception to framed preparation.

[0020] 3. Timing scheduling To eliminate bus contention and ensure data integrity and reliability, the "data write window," "data transfer window," and "media write window" are strictly staggered. This ensures that operations within each window are independent in the time domain, thereby achieving stable data storage with zero packet loss, no jump points, and recoverability. This technical solution, through precise timing scheduling and data transfer mechanisms, guarantees the efficiency and consistency of data acquisition and storage in high sampling rate environments.

[0021] Specifically, a mutual exclusion time window mechanism strictly isolates three types of operations—data writing, data transfer, and media writing—in time: a data writing time window is set at the beginning of the second to prioritize uninterrupted data input; subsequently, the data transfer time window completes the uninterrupted transfer of data from L1 to L2 from the previous second; finally, the media writing time window is opened during the remaining time of the second to sequentially write the complete data frames ready in L2 to the storage medium; the length of each window is configurable or adaptively adjustable to ensure that each operation is independent in the time domain, thereby achieving stable data storage with zero packet loss, no jump points, and recoverability. This technical solution, through precise timing scheduling and data transfer mechanisms, ensures the efficiency and consistency of data acquisition and storage in high sampling rate environments.

[0022] This method employs a complete chain of "frame boundary establishment, two-level caching, frame encapsulation, and persistent writing." Its core idea is based on three mutually exclusive time windows. Driven by the full-second frame boundary (FBT), it strictly staggers three types of high-frequency operations within the same second—DMA flowing into the first-level cache, transferring full-second data to the second-level cache, and peripheral batch writing (SDIO / eMMC)—in time. This is supplemented by write batch processing and write-back indexing to ensure zero-packet-loss, jump-point-free, and recoverable continuous acquisition and writing to the storage medium under 512kSPS and 24-bit conditions. The three mutually exclusive time windows are: (1) Data writing window (initial second stage) Only the acquisition end is allowed to write the data of the next second to the first-level buffer via DMA; Disallow write operations to the storage medium; In this phase, the priority is to ensure that the data acquisition link is not blocked.

[0023] (2) Data transfer window (followed by) Move the entire data from the previous second from the L1 cache to the corresponding slot in the L2 cache; The frame header must be finalized during the transfer process, and this process must not be interrupted. Storage write operations are not allowed during this phase to ensure that bus resources are concentrated on data transfer.

[0024] (3) Medium writing window (remaining time in this second) Write the complete full-second frames that have been formed in the secondary cache to the storage medium in a multi-block sequential write manner; The acquisition link continues to work, but runs at normal priority to avoid conflicts with write operations; It supports writing multiple consecutive data blocks at once, improving write efficiency.

[0025] The time window, or time window configuration, is configurable or adaptive. The length and boundaries of the mutually exclusive time window can be dynamically adjusted or configured via registers based on external 1PPS signal quality, media write performance, system load, etc. The default ratio T0 / T1 / T2 ≈ 1.5% / 1% / 97.5%, and the actual ratio is adaptively fine-tuned by the controller based on the media and load. Figure 2 As shown, specifically: T0 - Data write window (0–15ms): During this period, only DMA writes to the L1 cache are permitted. The primary objective of this phase is to prioritize processing bursts of data at the beginning of each new second while prohibiting external storage writes to ensure a smooth transition of cached data.

[0026] T1 - Data Transfer Window (15–25ms): Within this time window, the system moves the data from the previous second from the first-level cache to the second-level cache in its entirety, and finalizes the frame header of the full-second data frame. This operation cannot be interrupted to ensure data consistency and integrity.

[0027] T2 - Media write window (25ms–1s): This phase performs sequential multi-block write operations on completed whole-second data frames, writing the data in batches to the storage medium. During this period, the DMA continues to receive data for the current second at its normal priority, but avoids conflicts with storage write operations.

[0028] Based on the above, the specific implementation of S1-S3 will be explained in detail: S1. Receive an external whole-second event signal, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference.

[0029] Furthermore, after receiving the external integer second event signal, the external integer second event signal is subjected to cross-clock domain synchronization, digital dejittering, and anti-reentry window processing to eliminate signal metastability, glitches, and repeated triggering, ensuring that only one valid integer second frame boundary token is generated per second.

[0030] Specifically, this step aims to provide a unified, unique, and auditable whole-second time reference for the entire data caching, transfer, and persistence process, and to generate a unique identifier (i.e., number) for the data frame at the boundary of the natural second, so that the continuity, consistency, and recoverability of the data can be maintained even under high-speed acquisition and complex operating conditions. Event signals enter the local clock domain, and cross-clock domain metastability and glitches are eliminated through dual-trigger synchronization and debouncing processing. With the help of programmable minimum pulse width discrimination and anti-re-entry window, it is ensured that only one valid trigger is generated per natural second, avoiding duplicate frame blocking and frame errors.

[0031] Once a valid whole-second event is confirmed, the following key actions are performed within the same clock cycle: (1) Issue the frame boundary token for this second as the sole trigger for the first-level cache to close the data window of the previous second and the second-level cache to start the framing process; (2) Increment the frame number so that each second of data has an independent and continuous number identifier; (3) Synchronously latch the UTC timestamp corresponding to this second; (4) Reset the local subsecond counter to provide a starting point for the intra-frame relative time base.

[0032] To improve the time reliability of the method, this step further performs statistical analysis on the intervals of consecutive whole-second events and outputs quality indicators such as "time valid", "period abnormal", "edge lost" and "repeated edge" in real time. When the external time signal is temporarily missing or unstable, the monotonic second synthesis logic can be enabled to generate temporary whole-second boundaries by extrapolating the most recent time interval to ensure that the subsequent buffering and writing process is not interrupted. When the external signal is restored, the external time source is automatically switched back and the continuity of frame numbering is maintained.

[0033] Through the above design, this step constructs a whole-second frame boundary mechanism with uniqueness, stability, and fault adaptive capability. S2. A two-level cache is used to write and move the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, in the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing timestamp and frame number is generated in the second-level cache.

[0034] Furthermore, the first-level cache in the dual-level cache is a circular cache structure used for continuously receiving data packets; the second-level cache is a linear cache area with pre-allocated fixed frame slots, with each frame slot corresponding to a complete data frame of one second.

[0035] Furthermore, the complete data frame also includes a validity bitmap and a multi-layer check field. The validity bitmap is used to identify the integrity status of each data packet within that second, and the multi-layer check field is used to verify the correctness of the frame header and payload data. This step can be summarized as follows: Collected data is written to the first-level buffer (L1) in small packets via the AXI4-Stream interface. When the data packet threshold (N) per second is reached, an uninterrupted data transfer operation is performed within a predetermined transfer window, transferring the entire data from the previous second to the second-level buffer (L2). In the second-level buffer, a complete frame header containing a UTC timestamp, frame number, validity bitmap, and multi-layer checksum is generated for each second of data. The specific implementation is explained from two aspects: 1. Two-level caching and data movement The PL / PS collaborative implementation based on ZYNQ SoC involves the PL side continuously writing sampled data to the PS side's DDR3 L1 cache (high-speed access area) via AXI4-Stream+DMA. The PS side pre-allocates fixed frame slots in the L2 cache (frame preparation area), aggregates microframes in L1 by second, completes metadata, and then hands them over to the memory manager for batch writing to the medium (SDIO / eMMC). AXI4-Lite is used for the configuration and status readback of control registers such as DMA, transporters, and water level thresholds. Its implementation steps are as follows: 1.1 Establish a two-level cache structure (1) On the PL side, the collected data is divided into small data packets of fixed size and continuously written to the memory area of ​​the processing system via a high-speed bus to form a level 1 cache. The level 1 cache adopts a ring structure and can continuously receive data without interruption.

[0036] (2) Divide the memory on the PS side into a secondary cache area to store the complete data frames after being aggregated by second, with each second corresponding to a fixed cache slot.

[0037] 1.2 Close level 1 cache data by second When an external integer second event (e.g., a 1PPS rising edge) is detected: (1) Immediately stop the write interval in the first-level cache of the previous second to form the "complete data interval of the previous second" and fix the start and end addresses of the data in the cache for this second; (2) Record the start position, end position, number of data packets and verification status of the previous second, and generate a cache snapshot of the second, which will be used as the basis for subsequent whole-second transfer and frame processing.

[0038] 1.3 Move the entire data from the previous second to the secondary cache. Within a dedicated time window after the arrival of an entire second: (1) The PS side completes the batch transfer from the L1 cache to the L2 cache in one go within the mutual exclusion window according to the start and end positions and the number of data packets recorded in the cache snapshot. The DMA method is preferred for high-speed transfer. If the DMA fails, it will automatically switch to the backup transfer method. For example, the processor will complete the data copy from the L1 cache to the L2 cache in blocks by memory access instructions, or the segmented transfer method with reduced bandwidth will be enabled to ensure that the transfer process is not interrupted.

[0039] (2) An offset table is established during the transportation process to record the storage position and order of each data packet in the whole second frame; (3) Mark missing or damaged data packets to form a validity bitmap, which facilitates subsequent tracking and data repair.

[0040] High-speed handling is prioritized for material handling operations, and can be automatically switched to backup mode in abnormal situations to ensure that the material handling process is completed.

[0041] 1.4 Generate a complete full-second data frame After the second-level cache has completed the transfer of all data from the previous second, generate a complete full-second data frame using the following steps: (1) Calculate the checksum of the whole second data to verify the integrity of the data for that second; (2) Write frame header information, including second identifier, timestamp, number of data packets, total length, offset table position, validity mark, etc., to form the core description area of ​​the frame; (3) The frame header is updated by writing once, so that the external system can only see the two states of "incomplete" or "fully available", thereby avoiding reading part of the frame data.

[0042] 1.5 Batch write to storage media and reclaim cache During the media writing window, perform the following steps: (1) Write the completed whole-second frames in the secondary cache to the storage medium in batches; (2) After successful writing, release the cache slot corresponding to that second to prepare for the next whole second data frame; (3) Update the writing progress and statistics.

[0043] 1.6 Water Level Control and Abnormal Handling To ensure the cache does not overflow and to adapt to fluctuations in storage media performance: (1) Set high and low watermarks for the primary and secondary caches; (2) When the primary buffer approaches a high water level, the system increases the transport priority and notifies the acquisition end to temporarily limit the flow; it automatically recovers after the water level drops. (3) If the storage medium speed decreases, the data aggregation cycle can be temporarily extended to improve the efficiency of a single write operation; (4) During power failure recovery, only the second-level cache and the most recently written storage area need to be checked to quickly locate the last complete frame without a full disk scan.

[0044] 2. Framed encapsulation After the “whole-second frame boundary / dual-level buffer” and before the batch writing to the storage medium, the electromagnetic waveform loads aggregated in the previous second and the attitude / positioning subframes of the same second are encapsulated into whole-second data frames according to a unified specification, and key metadata and verification information are supplemented.

[0045] 2.1 Receiving Input Data (1) Receive basic time information for whole seconds, including frame number, UTC seconds, whole second trigger flag and time quality flag.

[0046] (2) Receive the previous second electromagnetic data reference, attitude / positioning data reference, and validity bitmap marking missing packets from the cache system.

[0047] 2.2 Establishing Core Information for the Frame Header Based on the input whole second time and frame number, create the core part of the frame header for the new frame and write: Frame identifier and version information; The timestamp corresponding to this second; This second's data packet quantity and length placeholder; Quality flag placeholder.

[0048] 2.3 Organize integer second load data (1) The location of the electromagnetic data in memory is directly bound to the current frame as the payload, without changing the data content; (2) Embed the same-second attitude / positioning data into the payload according to the predetermined format; (3) Attach a validity bitmap to the frame to indicate whether the data packets within that second are missing or corrupted.

[0049] 2.4 Supplementing Metadata Optional metadata, such as gain, operating mode, and environmental status, can be written to the extended area of ​​the frame header as needed. This part uses a variable-length structure, allowing the parser to selectively read it.

[0050] 2.5 Calculate the checksum and close the frame (1) Calculate the actual data length and number of data packets in this frame and fill them back into the frame header; (2) Calculate the frame header checksum and write it to the specified position in the frame header; (3) Calculate the overall load verification value and write it to the end of the frame to form a complete and verifiable whole-second frame.

[0051] 2.6 Output whole-second frames and index information (1) Send the encapsulated whole-second frame into the subsequent write queue; (2) Generate index information, including frame number, UTC seconds, and the offset position and length of this frame in the file, to facilitate quick location later.

[0052] 2.7 Anomalies and Quality Labeling If there are abnormal events such as whole-second events, missing data packets, or missing attitude / positioning information in this second, the data itself will not be modified. Instead, it will be marked in the quality field of the frame to ensure data traceability and facilitate post-processing repair.

[0053] S3. After the data transfer window ends, the media writing window begins, and the complete data frames generated in the secondary cache are written sequentially in batches to the physical storage medium to complete the reliable persistent storage of the data.

[0054] Furthermore, the data writing window, data transport window, and media writing window are mutually exclusive in time and are configured to be executed sequentially by a timing scheduler to avoid bus access conflicts and operational interference. Furthermore, the data writing is implemented through the AXI4-Stream interface, the data transfer is performed using DMA or a dedicated transfer device, and the media writing is implemented through the SDIO or eMMC interface. Specifically, it securely and efficiently writes data generated upstream in whole-second frames to the physical medium. Designed for engineering scenarios involving continuous, high-throughput, sequential append writes in semi-aerospace transient electromagnetic (SATEM) missions, it achieves long-term, zero-packet-loss / data-loss-free sequential batch writing to general-purpose block storage media (SD / eMMC / SSD, etc.) through low-overhead block organization, batch commit optimization, and sequential commit strategies. It also provides power-off consistency and rapid recovery capabilities. (1) Receive whole-second data frames Receive whole-second data frames from upstream, including frame header, payload, tail check and corresponding index information, and write them into a buffer in memory, waiting for batch submission.

[0055] (2) Organize the segment files in sequence Multiple whole-second data frames are arranged sequentially in time to form a segment file; the segment file adopts a contiguous storage structure, and each frame is arranged in the order of "frame header - payload - checksum". The file is switched in a rolling manner after reaching a set capacity or time.

[0056] (3) Batch writing to storage media Write whole-second frames from the buffer to the storage medium in batches using a sequential write method, specifically including: Perform necessary block alignment on the data to be written; Merge multiple whole-second frames into a single commit to reduce the number of write operations; Sequential append writing is used to avoid the performance degradation caused by random writes.

[0057] (4) Validate after writing and append index After writing is complete, the write result is verified according to the check fields in the frame header and payload, and the corresponding frame index information is appended to the end of the file to quickly locate the position of the frame in the storage medium.

[0058] (5) Rapid recovery after power outage Several index mirrors are maintained at the beginning and end of the segment file.

[0059] After a power outage or abnormal shutdown, the system does not need to scan the entire file. It can quickly find the last complete frame through the index and resume writing from that position.

[0060] (6) Exception handling and degradation writing If the storage medium speed decreases or write fails: Retry writing or increase the batch write size; If necessary, temporarily suspend the submission of new frames and wait for the media to recover; Upon receiving a shutdown command, immediately perform a synchronous write and safely close the file.

[0061] The following describes the application scenarios for high sampling rate data storage, such as semi-airborne transient electromagnetic (SATEM): On an unmanned aerial vehicle (UAV)-borne semi-airborne transient electromagnetic (SATEM) exploration platform, the receiving coil flies with the platform over the target area, undertaking the task of acquiring secondary field signals after the transmission is turned off. The data acquisition module mainly consists of a coil, a low-noise analog front-end, a 24-bit high sampling rate ADC (typically 512kSPS), a programmable logic SoC (PL / PS), and local storage, simultaneously acquiring IMU / GNSS attitude and positioning information in parallel. The entire link establishes a unified time reference based on external integer-second events (such as GNSS 1PPS), requiring the secondary field signal and attitude / positioning data to be naturally aligned to "integer seconds" and continuously recorded over a long period. During data acquisition, the coil signal is differentially amplified and anti-aliasing filtered before being continuously quantized by the ADC, forming a stable data stream of approximately 1.5–2.3 MB / s (including alignment and metadata). Attitude / positioning data is stamped to generate subframes using the same integer-second reference.

[0062] The PL end packages the data collected per second by the AD7760 into 200 10KB data packets, which are then sequentially transmitted to the segmented L1 cache in the PS end's DDR3 via AXI DMA. Each received packet triggers an interrupt, and after accumulating 200 packets, all data is moved to a logically isolated L2 cache. Subsequently, the system concatenates this data with the attitude and positioning information for each second to form the final data frame, and writes it to the SD card using a unified write function, i.e., writing 1 second of data at a time, ensuring that the data for the current second is highly consistent with the current attitude, positioning, and time information. The L1 and L2 caches are both implemented in DDR3, and functional isolation is achieved through address management and data transfer logic to maintain the independence of write and write data resources and prevent write-write conflicts.

[0063] Example 2 One embodiment of the present invention provides a high-speed data storage system based on whole-second frame boundaries and a two-level buffer, comprising: The boundary establishment module is configured to: receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. The data caching module is configured to use a two-level cache for writing and moving the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, within the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing a timestamp and frame number is generated in the second-level cache. The data persistence module is configured to: after the data transfer window ends, enter the media write window, and sequentially write the complete data frames generated in the secondary cache into the physical storage medium in batches to complete the reliable persistent storage of data.

[0064] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the high-speed data storage method based on whole-second frame boundaries and two-level buffering.

[0065] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, they implement the high-speed data storage method based on whole-second frame boundaries and two-level cache.

[0066] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute the high-speed data storage method based on whole-second frame boundaries and two-level buffering.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A high-speed data storage method based on whole-second frame boundaries and two-level buffering, characterized in that, include: Receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. A two-level cache is used to write and move the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, in the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing timestamp and frame number is generated in the second-level cache. After the data transfer window ends, the media writing window begins, where complete data frames generated in the secondary cache are written sequentially in batches to the physical storage medium, thus completing the reliable persistent storage of the data.

2. The high-speed data storage method based on whole-second frame boundaries and two-level buffering as described in claim 1, characterized in that, The method further includes: after receiving the external integer second event signal, performing cross-clock domain synchronization, digital dejittering, and anti-reentry window processing on the external integer second event signal to eliminate signal metastability, glitches, and repeated triggering, ensuring that only one valid integer second frame boundary token is generated per second.

3. The high-speed data storage method based on whole-second frame boundaries and two-level buffering as described in claim 1, characterized in that, The two-level buffer consists of a ring buffer structure for continuously receiving data packets in the first level; and a linear buffer with pre-allocated fixed frame slots, where each frame slot corresponds to a complete data frame of one second.

4. The high-speed data storage method based on whole-second frame boundaries and two-level buffering as described in claim 1, characterized in that, Within each full second, the data writing window, data transport window, and media writing window are mutually exclusive in time and are configured to be executed sequentially by the timing scheduler to avoid bus access conflicts and operational interference.

5. The high-speed data storage method based on whole-second frame boundaries and two-level buffering as described in claim 1, characterized in that, The complete data frame also includes a validity bitmap and a multi-layer check field. The validity bitmap is used to identify the integrity status of each data packet within that second, and the multi-layer check field is used to verify the correctness of the frame header and payload data.

6. The high-speed data storage method based on whole-second frame boundaries and two-level buffering as described in claim 1, characterized in that, The data writing is implemented through the AXI4-Stream interface, the data transfer is performed using DMA or a dedicated transfer device, and the media writing is implemented through the SDIO or eMMC interface.

7. A high-speed data storage system based on whole-second frame boundaries and a two-level buffer, characterized in that, include: The boundary establishment module is configured to: receive external whole-second event signals, generate a unique whole-second frame boundary token based on the rising edge of the whole-second event signal, and synchronously latch the current UTC timestamp and the incrementing global frame number to establish a unified whole-second time reference. The data caching module is configured to use a two-level cache for writing and moving the collected data. Specifically, based on the whole-second frame boundary token, within each whole second, firstly, during the data writing window, the collected data is continuously written to the first-level cache in the form of fixed-size data packets; when the next whole-second boundary is detected, the data range of the previous second in the first-level cache is immediately closed, and a corresponding cache snapshot is generated; then, within the data moving window immediately following the whole-second boundary, the data of the previous second is moved from the first-level cache to the second-level cache according to the cache snapshot, and a complete data frame containing a timestamp and frame number is generated in the second-level cache. The data persistence module is configured to: after the data transfer window ends, enter the media write window, and sequentially write the complete data frames generated in the secondary cache into the physical storage medium in batches to complete the reliable persistent storage of data.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the high-speed data storage method based on whole-second frame boundaries and two-level buffer as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the high-speed data storage method based on whole-second frame boundaries and two-level cache as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the high-speed data storage method based on whole-second frame boundaries and two-level buffer as described in any one of claims 1-6.

Images