A high speed storage system and method for optical fiber data
By adapting the interface conversion module to fiber optic communication cables and utilizing the FPGA platform and NUMA topology sensing technology for data capture and storage, the system solves the problems of data loss and low fault location efficiency in high-bandwidth fiber optic data storage in traditional systems. It achieves high-speed storage and analysis with universal hardware and definable software, and is suitable for fields such as aerospace and smart grids.
Patent Information
- Application Number
- CN202610759727.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-29
- Publication Date
- 2026-08-25
AI Technical Summary
Traditional data acquisition systems face challenges when dealing with high-bandwidth fiber optic data, including difficulties in data tracing, insufficient evidence for post-analysis, low efficiency in fault location, inability to store high-speed data in a timely manner, insufficient storage architecture and interface bandwidth, high memory access latency in NUMA multi-channel servers, and the inability of single-threaded synchronous write mode to utilize the parallel channel capabilities of NVMe SSDs, leading to data loss and system instability.
An interface conversion module is used to adapt to optical fiber communication cables of different specifications. Data capture and preprocessing are performed using an FPGA platform. A ring buffer is allocated in combination with NUMA topology sensing technology to perform lock-free synchronous and large-granularity asynchronous disk writing. The actions of each module are coordinated through software modules to achieve overall data flow control and visualization.
It achieves high-speed and reliable fiber optic data storage and intelligent analysis, improves hardware compatibility and processing efficiency, ensures data integrity and fault location accuracy, reduces hardware procurement and maintenance costs, and adapts to the needs of multiple application fields.
Smart Images
Figure CN122633111A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of measurement and instrumentation technology, and more specifically, to a high-speed storage system and method for fiber optic data. Background Technology
[0002] With the widespread application of fiber optic sensing technology in key fields such as aerospace, smart grids, and large-scale infrastructure, the fiber optic data output by the equipment exhibits significant characteristics such as high bandwidth, multiple channels, strong real-time performance, and extremely high data generation rate of the front-end hardware, which places unprecedented demands on the performance of data acquisition and recording systems.
[0003] Traditional data acquisition systems often focus on real-time monitoring, generally lacking the ability to fully record raw fiber optic data and conduct in-depth post-processing analysis. This leads to difficulties in data tracing, insufficient evidence for post-processing analysis, low efficiency in fault location, and the inability to store high-speed data input from the front end in a timely manner, resulting in data loss. Furthermore, existing technologies face several core technical challenges: the increasing data rates of fiber optics place extremely high demands on storage architecture and interface bandwidth; high cross-node memory access latency in NUMA multi-processors severely impacts data migration efficiency; and single-threaded synchronous write modes cannot utilize the parallel channel capabilities of modern NVMe SSDs, easily causing onboard FIFO overflow or data loss. In addition, the software layer lacks efficient real-time processing and in-depth post-processing capabilities, making it difficult to meet the needs of current complex application scenarios. Summary of the Invention
[0004] The problem addressed by this invention is how to achieve high-speed and reliable storage and intelligent analysis of high-speed fiber optic data, improve hardware adaptability and processing efficiency, and thereby improve data integrity and fault location accuracy.
[0005] To address the above problems, this invention provides a high-speed storage system and method for optical fiber data.
[0006] In a first aspect, the present invention provides a high-speed optical fiber data storage system, comprising: an interface conversion module, a data acquisition module, a data storage module, a data processing module, and a software module.
[0007] The interface conversion module is used to adapt to optical fiber communication cables of different specifications and receive external optical fiber data, and send the external optical fiber data to the data acquisition module;
[0008] The data acquisition module is used to capture and preprocess the external optical fiber data through the FPGA platform to obtain the preprocessed data stream and onboard status information, and transmit the preprocessed data stream to the data storage module, while sending the onboard status information to the software module.
[0009] The data storage module is used to allocate a circular buffer according to NUMA topology-aware technology, receive and cache the data stream in the circular buffer, perform lock-free synchronization and large-granularity asynchronous disk persistence processing on the data stream, obtain the stored data file, and send it to the data processing module.
[0010] The data processing module is used to perform overall data flow control processing on the data file and send the processed data file to the software module.
[0011] The software module is used to receive user operation instructions and interact with the onboard status information to construct a human-computer interaction loop. The software module generates and distributes operation control instructions according to the user operation instructions to control the data acquisition module, data storage module and data processing module to perform corresponding actions, and at the same time receives feedback from each module.
[0012] Optionally, the data acquisition module includes an FPGA acquisition unit, a clock configuration unit, and a power management unit.
[0013] The FPGA acquisition unit is used to receive the external optical fiber signal, perform protocol parsing and buffering on the external optical fiber signal to obtain the data stream and the onboard status information, transmit the data stream to the data storage module via DMA, and send the onboard status information to the software module at the same time.
[0014] The clock configuration unit is used to configure the clock synthesizer, generate a clock signal, and provide a working clock for the FPGA acquisition unit.
[0015] The power management unit is used to provide the core power and adjustable ADJ voltage to the FPGA acquisition unit through a power chip.
[0016] Optionally, the FPGA acquisition unit is also used to receive the binary configuration file sent by the software module, and transmit the binary configuration file to the Flash via the onboard DDR for online re-soldering of the FPGA logic to adapt to different fiber optic communication protocols.
[0017] Optionally, the data storage module includes a NUMA binding unit, a lock-free buffer unit, and an asynchronous disk persistence unit.
[0018] The NUMA binding unit is used to call the GetNumaHighestNodeNumber interface to detect the server's NUMA topology, allocate a preset number of bytes of the circular buffer for each acquisition channel, and use the VirtualAllocExNuma interface to force the physical memory of the circular buffer to be allocated locally on the NUMA node closest to the acquisition card's PCIe slot, while aligning the starting address of the circular buffer.
[0019] The lock-free buffer unit is used to monitor the buffer level using atomic long integer pointers readTotal and writtenTotal, and to perform data read and write pointer updates for lock-free synchronization of the multi-channel data stream;
[0020] The asynchronous disk write unit is used to disable system caching and enable asynchronous I / O mode to initiate a disk write request when the amount of unprocessed data in the circular buffer reaches a preset number of bytes. It also processes the write completion event through the I / O completion port model and auxiliary thread pool to perform large-granular asynchronous disk write processing on the data stream.
[0021] Optionally, the data storage module further includes a thread affinity unit.
[0022] The thread affinity unit is used to bind the data transfer thread of the data acquisition module and the write thread of the asynchronous disk write unit to the processor cores in the corresponding NUMA nodes through the SetThreadAffinityMask interface.
[0023] Optionally, the data processing module includes a parsing unit, a processing unit, and a central control unit.
[0024] The parsing unit is used to read the data file and parse the original optical fiber data in the data file;
[0025] The processing unit is used to perform data playback processing on the parsed data;
[0026] The central control unit is used to coordinate and schedule the data parsing, processing, and playback of the data file, and to send the processed data file to the software module.
[0027] Optionally, the software module includes a control instruction unit and a status monitoring unit.
[0028] The control command unit is used to respond to the user operation command, generate corresponding operation control commands, and send the operation control commands to the data acquisition module, the data storage module, and the data processing module respectively, so as to control each module to perform corresponding actions.
[0029] The control of each module to perform corresponding actions includes: controlling the data acquisition module to enable the corresponding channel and controlling the data storage module to start asynchronous disk writing; and responding to the stop recording command in the user operation command by sending a stop reading signal to the data storage module.
[0030] The status monitoring unit is used to receive the onboard status information fed back by the data acquisition module and the in-depth analysis results and visualization reports generated by the data processing module in real time, and to feed back the onboard status information and in-depth analysis results to the human-computer interaction interface for dynamic updates.
[0031] Optionally, after receiving the stop read signal, the asynchronous disk write unit is further configured to perform tail data processing.
[0032] The tail data processing includes: waiting for the data acquisition module to complete the last DMA transfer, calculating the amount of tail scattered data in the circular buffer, using the HeapAlloc temporary buffer in conjunction with the CopyMemory operation to perform zero-padding alignment to ensure that the last write operation meets the alignment constraint of unbuffered write; and calling the SetFilePointerEx and SetEndOfFile interfaces to precisely truncate the file to the effective data size.
[0033] Optionally, the data storage module is also used to cooperate with the data acquisition module to form a two-level back pressure mechanism. By monitoring the DDR level of the FPGA acquisition unit and the level of the annular buffer, an elastic buffer layer is formed to control the onboard FIFO from overflowing when disk I / O jitter occurs.
[0034] In a second aspect, the present invention provides a high-speed optical fiber data storage method, comprising:
[0035] External fiber optic data is acquired, and the external fiber optic data is captured and preprocessed through the FPGA platform to obtain the preprocessed data stream and onboard status information.
[0036] A circular buffer is allocated based on NUMA topology-aware technology, the data stream is cached in the circular buffer, and lock-free synchronization and large-granularity asynchronous disk persistence processing is performed on the data stream to obtain a data file that has been stored.
[0037] The data file is subjected to overall data flow control processing to obtain the processed data file;
[0038] The system acquires user operation instructions, generates and distributes operation control instructions based on these instructions, controls the execution of corresponding actions, and combines onboard status information with in-depth analysis results for visualization.
[0039] The high-speed optical fiber data storage system and method of this invention adapts to optical cables of different specifications and receives external optical fiber data through an interface conversion module, thereby unifying the physical access standard and ensuring that signals from various devices under test can be smoothly integrated into the system, improving hardware compatibility. The data acquisition module then uses the FPGA platform to capture and preprocess the external optical fiber data, and synchronously outputs the preprocessed data stream and onboard status information reflecting the operating status, providing a foundation for subsequent storage and monitoring. The data storage module allocates a ring buffer based on NUMA topology sensing technology, locking the cache space to the local memory node closest to the acquisition card, and then uses lock-free synchronization and large-granular asynchronous disk writing processing to allow the high-speed data stream to be smoothly written to the disk, avoiding transmission congestion and packet loss. The data processing module performs overall data flow control processing on the stored data file and sends the processed data file to the software module. The software module acts as a central hub to receive user instructions and onboard status information, coordinates the distribution of operation control instructions to each module, coordinates the start and stop of acquisition, storage rhythm and analysis tasks, and provides visualization, ultimately realizing closed-loop management of the entire process from optical fiber signal access to intelligent analysis, thereby improving data integrity and fault location accuracy. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the high-speed optical fiber data storage system in an embodiment of the present invention;
[0041] Figure 2 This is a flowchart illustrating the high-speed storage method for optical fiber data in an embodiment of the present invention. Detailed Implementation
[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0043] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0044] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "optionally" means "optional embodiment". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0045] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0046] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.
[0047] In related technologies, traditional high-speed data acquisition and storage systems exhibit significant deficiencies in the determinism of their storage architecture and system adaptability when dealing with 39.06Gbps-level fiber optic data. Existing data loggers typically only possess unidirectional data throughput capabilities, lacking deep awareness of memory topology and I / O scheduling mechanisms. This often leads to uncontrollable packet loss when facing high burst bandwidth. Especially for solutions based on general-purpose server architectures, the lack of NUMA (Non-Uniform Memory Access) affinity design means DMA (Direct Memory Access) operations frequently cross node boundaries, causing high access latency and bus congestion. Simultaneously, the single-threaded synchronous write mode cannot match the parallel processing capabilities of NVMe SSDs, resulting in storage bottlenecks and data overflow in the buffer. This architectural failure is particularly fatal in high-bandwidth fiber optic testing scenarios. Users are often focused on signal transmission and fail to detect hidden packet loss in the background storage in real time, only discovering file incompleteness when rereading data after the test, causing irreversible test accidents and losses of manpower and resources.
[0048] Furthermore, the fixed hardware solutions of traditional dedicated acquisition equipment are severely incompatible with the ever-changing protocols in industrial environments, resulting in extremely high testing costs and maintenance difficulties. Industrial fiber optic protocols are complex, while traditional solutions typically use a single device with a fixed protocol, making them unable to adapt to changes in project requirements. This deployment method not only incurs high upfront procurement costs and a wide variety of spare parts, but also requires complete hardware replacement for later upgrades, lacking flexibility. More importantly, due to the lack of an FPGA online reconfiguration mechanism, the system cannot quickly respond to the testing requirements of new fiber optic communication standards, causing the testing equipment to become obsolete rapidly. In addition, traditional software architectures often employ a single-process blocking design, lacking lock-free scheduling capabilities for multi-channel data, leading to frequent migration and contention of CPU resources between kernels, further exacerbating system instability and latency jitter.
[0049] Meanwhile, in practical applications, fiber optic data streams exhibit strong continuity and burstiness, while traditional storage solutions lack effective elastic buffering and backpressure mechanisms. When disk I / O experiences a brief slowdown due to momentary jitter (such as file system defragmentation or background system tasks), the front-end acquisition module cannot perceive the back-end pressure and continues to feed data at full speed, causing the onboard FIFO to overflow rapidly. Furthermore, traditional solutions often abruptly cut off the data stream when recording stops, lacking zero-padding and truncation processing for trailing fragmented data, resulting in invalid padding or format errors in the generated data files, increasing the complexity and error rate of subsequent data parsing. In summary, existing technologies have serious systemic deficiencies in the determinism, hardware adaptability, and system robustness of high-speed fiber optic data storage, failing to meet the urgent needs of various fields such as aerospace and power grid monitoring for zero-loss high-bandwidth data recording.
[0050] To address the problems existing in the aforementioned related technologies, this embodiment provides a high-speed storage system and method for optical fiber data.
[0051] Combination Figure 1 As shown in the figure, the high-speed optical fiber data storage system of the present invention provided in this embodiment includes: an interface conversion module, a data acquisition module, a data storage module, a data processing module, and a software module.
[0052] The interface conversion module is used to adapt to optical fiber communication cables of different specifications and receive external optical fiber data, and send the external optical fiber data to the data acquisition module;
[0053] Specifically, this embodiment utilizes an interface conversion module to achieve fiber optic link switching. Depending on the working environment, it matches different types of fiber optic communication cables, solving the problem of traditional testing equipment having closed physical interfaces and being unable to adapt to multiple fiber optic cable models. Standardized external fiber optic data is sent to the data acquisition module. This embodiment breaks down the physical barriers between heterogeneous fiber optic networks and acquisition equipment, achieving "hardware universality + software definition" access capabilities. Through pre-processing physical layer signal conditioning, the purity and stability of input data are ensured, avoiding signal attenuation or reflection caused by interface incompatibility at the source. This significantly reduces hardware procurement and spare parts maintenance costs in multi-scenario testing, and significantly improves the system's environmental adaptability and engineering deployment efficiency. It enables a single device to adapt to multiple fiber optic protocols and application scenarios, eliminating the need for users to purchase dedicated instruments for each protocol, reducing equipment procurement costs and the types of spare parts required. If widely adopted in the industry, it can save significant social resources and financial investment, aligning with the concept of green and economical development.
[0054] The data acquisition module is used to capture and preprocess the external optical fiber data through the FPGA platform to obtain the preprocessed data stream and onboard status information, and transmit the preprocessed data stream to the data storage module, while sending the onboard status information to the software module.
[0055] The data storage module is used to allocate a circular buffer according to NUMA topology-aware technology, receive and cache the data stream in the circular buffer, perform lock-free synchronization and large-granularity asynchronous disk persistence processing on the data stream, obtain the stored data file, and send it to the data processing module.
[0056] The data processing module is used to perform overall data flow control processing on the data file and send the processed data file to the software module.
[0057] The software module is used to receive user operation instructions and interact with the onboard status information to construct a human-computer interaction loop. The software module generates and distributes operation control instructions according to the user operation instructions to control the data acquisition module, data storage module and data processing module to perform corresponding actions, and at the same time receives feedback from each module.
[0058] Specifically, this embodiment utilizes an FPGA platform to achieve high-speed capture and preprocessing of fiber optic data. The preprocessed data stream is output to the data storage module, and onboard status information is sent to the software module. A circular buffer is allocated using NUMA topology sensing technology, and lossless data storage is achieved through lock-free synchronous and large-granularity asynchronous disk persistence. The data processing module then performs overall data stream control processing and sends the processed data file to the software module. The software module receives user operation commands, interacts with onboard status information, and distributes operation control commands. This effectively solves problems such as difficulty in data traceability, insufficient post-event analysis basis, low fault location efficiency, data loss, and bottlenecks in memory access and I / O blocking in traditional systems. It achieves high-speed, frame-loss-free recording, deterministic low-latency disk persistence, multi-card, multi-channel linear expansion, zero data loss guarantee, and in-depth analysis of the entire raw data chain. The hardware is universal, and the software is customizable, making it adaptable to applications in aerospace, smart grids, large-scale infrastructure, and other fields. This reduces industry testing costs, saves social resources, and enhances the safe operation guarantee capabilities of key equipment.
[0059] Optionally, the data acquisition module includes an FPGA acquisition unit, a clock configuration unit, and a power management unit.
[0060] The FPGA acquisition unit is used to receive the external optical fiber signal, perform protocol parsing and buffering on the external optical fiber signal to obtain the data stream and the onboard status information, transmit the data stream to the data storage module via DMA, and send the onboard status information to the software module at the same time.
[0061] The clock configuration unit is used to configure the clock synthesizer, generate a clock signal, and provide a working clock for the FPGA acquisition unit.
[0062] The power management unit is used to provide the core power and adjustable ADJ voltage to the FPGA acquisition unit through a power chip.
[0063] Specifically, in this embodiment, the FPGA acquisition unit receives external fiber optic signals and completes protocol parsing and buffering, generating data streams and onboard status information (including but not limited to FPGA logic operating status, board handle, initialization status, DMA transfer completion status, etc.). Through the DMA high-speed direct data storage module, and simultaneously sending the status back to the software module, high-speed parallel acquisition, hardware preprocessing, and non-blocking data transfer are achieved. This solves the problems of insufficient bandwidth, time-consuming data preprocessing, heavy CPU load, and easy data loss in traditional systems. DMA (Direct Memory Access) refers to a hardware mechanism in a computer system that allows external devices (such as FPGA acquisition cards) to directly exchange data with system memory without CPU intervention. In the traditional method, fiber optic data enters the FPGA; the FPGA notifies the CPU; the CPU reads the FPGA data; and the CPU writes it to memory. This traditional method incurs significant CPU overhead; for example, when faced with a 39.06Gbps data stream, the CPU will be instantly overloaded, leading to data loss. In this embodiment, however, the external fiber optic signal enters the FPGA acquisition unit; the DMA controller takes over; the data is directly transferred from the acquisition unit to the circular buffer in the server host memory; and only the CPU is notified once upon completion.
[0064] In this embodiment, the clock configuration unit configures the clock synthesizer to generate a low-jitter clock signal, providing a stable operating clock for the FPGA acquisition unit. This ensures accurate timing, low jitter, and stable link for high-speed fiber optic data transmission, avoiding data errors and frame drops caused by clock deviations. The power management unit provides a stable core power supply and adjustable ADJ voltage through a power chip, ensuring stable power supply, adjustable voltage, and reliable industrial-grade wide-temperature operation of the FPGA acquisition unit under high load and high-speed transceiver scenarios. This eliminates the technical bottlenecks of FPGA acquisition unit malfunction, data transmission errors, and poor system stability caused by power supply noise and voltage instability. The three components work together to achieve high-speed, stable, and reliable fiber optic data acquisition and preprocessing, supporting the system's high bandwidth, strong real-time, and zero-frame-drop acquisition requirements.
[0065] Optionally, the FPGA acquisition unit is also used to receive the binary configuration file sent by the software module, and transmit the binary configuration file to the Flash via the onboard DDR for online re-soldering of the FPGA logic to adapt to different fiber optic communication protocols.
[0066] Specifically, this embodiment completes the online re-solidification of FPGA logic by transmitting data from onboard DDR to Flash, achieving hardware universality, software definition, and online reconfiguration and upgrade. It can adapt to different fiber optic communication protocols without replacing hardware, solving the problems of traditional equipment protocol rigidity, poor universality, high adaptation costs, and long upgrade cycles. It improves resource utilization and system flexibility, reduces equipment procurement and maintenance costs in multi-protocol scenarios, supports multiple models of equipment to share the same hardware platform, and adapts to testing needs in multiple fields.
[0067] In one embodiment, this embodiment addresses the issue of multiple device models sharing a single device by implementing an FPGA reconfiguration design. FPGA reconfiguration refers to the process of reprogramming an FPGA (Field-Programmable Gate Array) after it has been configured, thereby changing its internal logic structure. This process allows the FPGA to execute different functional tasks at different times, thus improving the FPGA's flexibility and resource utilization. Specifically:
[0068] FPGA reconfiguration is mainly divided into two types: complete reconfiguration and partial reconfiguration.
[0069] Complete refactoring: This involves reprogramming the entire FPGA, essentially replacing all logic configurations within the FPGA. This refactoring method is simple and straightforward, but it will interrupt all current functions of the FPGA.
[0070] Partial refactoring: This involves replacing only a portion of the FPGA design while the rest continues to function normally. Partial refactoring is not considered a special case of full refactoring, as they are essentially similar, differing only in the scope of their impact. Partial refactoring allows for dynamic updates or optimizations of specific parts of the system without affecting the operation of other components.
[0071] FPGA reconfiguration is typically performed through specific interfaces such as JTAG, CAP, SelectMAP, or the Internal Configuration Access Port (ICAP). Partial and full reconfiguration are similar in methodology, both involving reprogramming the FPGA by downloading a new bitstream file.
[0072] This embodiment of FPGA reconfiguration allows multiple designs to share the same FPGA architecture, thereby reducing FPGA area, cost, and system complexity, and improving resource utilization. By dynamically changing the internal logic structure of the FPGA, it can adapt to different application requirements, improving system flexibility and adaptability. Partial reconfiguration technology allows for online upgrades and optimizations of specific parts of the FPGA without interrupting the operation of other parts of the system. This embodiment uses the AXI-QSPI IP core to implement the online reprogramming of the host computer within the FPGA's soft core. The host computer sends the bin file to the DDR, and then the SDK soft core transmits the DDR data to the flash. The hardware interface adopts a universal design, and different usage requirements can be met by modifying the FPGA software; after the host computer reprogramming is complete, only a power-off and restart are needed to load the new FPGA program onto the board.
[0073] Optionally, the data storage module includes a NUMA binding unit, a lock-free buffer unit, and an asynchronous disk persistence unit.
[0074] The NUMA binding unit is used to call the GetNumaHighestNodeNumber interface to detect the server's NUMA topology, allocate a preset number of bytes of the circular buffer for each acquisition channel, and use the VirtualAllocExNuma interface to force the physical memory of the circular buffer to be allocated locally on the NUMA node closest to the acquisition card's PCIe slot, while aligning the starting address of the circular buffer.
[0075] The lock-free buffer unit is used to monitor the buffer level using atomic long integer pointers readTotal and writtenTotal, and to perform data read and write pointer updates for lock-free synchronization of the multi-channel data stream;
[0076] The asynchronous disk write unit is used to disable system caching and enable asynchronous I / O mode to initiate a disk write request when the amount of unprocessed data in the circular buffer reaches a preset number of bytes. It also processes the write completion event through the I / O completion port model and auxiliary thread pool to perform large-granular asynchronous disk write processing on the data stream.
[0077] Specifically, in this embodiment, the NUMA binding unit detects the server's NUMA topology, allocates a preset byte circular buffer to each channel, and binds the memory to the local memory of the nearest NUMA node on the acquisition card's PCIe, achieving memory address alignment. This effectively reduces NUMA cross-node access latency, improves DMA and caching efficiency, and solves the problems of high cross-node memory access latency and low data migration efficiency in traditional multi-channel servers. The lock-free buffer unit monitors the buffer level and updates the read / write pointers through atomic long integer pointers readTotal and writtenTotal, achieving lock-free synchronous processing of multi-channel data streams, avoiding global lock contention, and eliminating thread blocking. This solution addresses the issues of high synchronization overhead and poor scalability in multi-channel concurrent scenarios. The asynchronous disk write unit disables system caching and enables asynchronous I / O mode when the data volume in the buffer reaches a preset value. It uses an I / O completion port model and an auxiliary thread pool to handle write completion events, achieving large-granularity asynchronous disk write, reducing I / O context switching, fully utilizing NVMe parallel bandwidth, and reducing CPU usage. This solves the I / O blocking problem that traditional single-threaded synchronous write mode cannot match the high-speed acquisition speed of the front end and is prone to FIFO overflow and data loss. The combination of these three features enables high-speed, low-latency, lock-free concurrency, and highly reliable massive fiber optic data storage, ensuring lossless and stable data write to disk.
[0078] In one embodiment, this embodiment uses a NUMA node awareness-based scheme for allocating and aligning a circular buffer. During system initialization, GetNumaHighestNodeNumber is called to probe the server's NUMA topology. An independent 16GB circular buffer (BUFFER_SIZE) is allocated to each channel of each acquisition card (Card AE). VirtualAllocExNuma is used to force the allocation of the buffer's physical memory on the local memory of the NUMA node closest to the card's PCIe slot. The ROUND_UP_PTR(Ptr, SECTOR_SIZE) macro is used to force the buffer's starting address to align to the storage sector size (4KB) boundary. This operation not only satisfies the FILE_FLAG_NO_BUFFERING requirement for memory address alignment, avoiding extra copying within the driver, but also significantly improves the efficiency of DMA (Direct Memory Access) and CPU cache.
[0079] This embodiment is based on a read / write separation multi-level parallel processing architecture. The system adopts a producer-consumer model, dividing the process into two levels of operations: "data acquisition thread (ReadThread)" and "data write-to-disk thread (WriteThread)".
[0080] Level 1 Operations (Acquisition and Buffering):
[0081] Poll the PCIeGTH_Card_Channel_Usedw interface to obtain the write level of the onboard DDR in the onboard status information. When the accumulated data exceeds the preset threshold (CHUNK_SIZE=4MB), trigger a DMA read operation, move the data to the aforementioned NUMA local circular buffer, and atomically update the read pointer readTotal.
[0082] Second-stage pipeline (asynchronous disk loading):
[0083] The disk write thread monitors the difference between readTotal and writtenTotal. When the amount of unprocessed data in the circular buffer reaches the trigger value (WRITE_SIZE=512MB), an asynchronous write request is initiated.
[0084] In another embodiment, the DDR level in the onboard status information is obtained by polling the PCIeGTH_Card_Channel_Usedw interface. DMA is triggered only when the data accumulates to CHUNK_SIZE (4MB) to avoid frequent CPU interruptions. The target address (host memory) for DMA transfer must be processed by the ROUND_UP_PTR macro and aligned to the 4KB sector boundary to meet the FILE_FLAG_NO_BUFFERING requirement. DMA writes the data directly to the local memory of the NUMA node closest to the PCIe slot, avoiding latency caused by cross-node transfers.
[0085] Optionally, the data storage module further includes a thread affinity unit.
[0086] The thread affinity unit is used to bind the data transfer thread of the data acquisition module and the write thread of the asynchronous disk write unit to the processor cores in the corresponding NUMA nodes through the SetThreadAffinityMask interface.
[0087] Specifically, in this embodiment, the thread affinity unit uses the SetThreadAffinityMask interface to bind the data transfer thread of the data acquisition module and the write thread of the asynchronous disk writing unit to the processor core of the corresponding NUMA node, respectively. This eliminates cache invalidation and memory access latency jitter caused by random migration of threads across nodes, avoids resource contention and cross-bus congestion when the acquisition card DMA data is written to memory and the CPU processes data and writes it to disk, effectively solves the technical problems of unstable thread scheduling, poor response determinism, and low CPU cache hit rate under the NUMA architecture, ensures deterministic low latency in high-speed data stream processing, and improves the system's concurrent processing efficiency and operational stability.
[0088] In one embodiment, this implementation uses a low-CPU-occupancy write algorithm based on I / O completion port and thread affinity. In CreateFileA, it combines FILE_FLAG_NO_BUFFERING (bypassing the system cache and eliminating secondary cache copy overhead) and FILE_FLAG_OVERLAPPED (enabling asynchronous I / O). Each write thread independently creates an I / O completion port (IOCP) and a corresponding auxiliary thread IOCPThread to handle cleanup and status confirmation after the write operation, decoupling I / O waiting from business logic. SetThreadAffinityMask explicitly binds the "acquisition thread" and "write thread" to the corresponding NUMA node's processor core. This embodiment eliminates cache invalidation and memory access latency jitter caused by random thread migration between NUMA nodes, ensuring deterministic response to high-speed data streams.
[0089] Optionally, the data processing module includes a parsing unit, a processing unit, and a central control unit.
[0090] The parsing unit is used to read the data file and parse the original optical fiber data in the data file;
[0091] The processing unit is used to perform data playback processing on the parsed data;
[0092] The central control unit is used to coordinate and schedule the data parsing, processing, and playback of the data file, and to send the processed data file to the software module.
[0093] Specifically, this embodiment performs protocol decoding and format standardization output on the original optical fiber data, so as to achieve complete restoration of the original data, unified format, and easy cross-platform reuse, and solve the problems of data format incompatibility, incomplete parsing, and difficulty in tracing the source in traditional systems.
[0094] Optionally, the software module includes a control instruction unit and a status monitoring unit.
[0095] The control command unit is used to respond to the user operation command, generate corresponding operation control commands, and send the operation control commands to the data acquisition module, the data storage module, and the data processing module respectively, so as to control each module to perform corresponding actions.
[0096] The control of each module to perform corresponding actions includes: controlling the data acquisition module to enable the corresponding channel and controlling the data storage module to start asynchronous disk writing; and responding to the stop recording command in the user operation command by sending a stop reading signal to the data storage module.
[0097] The status monitoring unit is used to receive the onboard status information fed back by the data acquisition module and the in-depth analysis results and visualization reports generated by the data processing module in real time, and to feed back the onboard status information and in-depth analysis results to the human-computer interaction interface for dynamic updates.
[0098] Specifically, the control command unit responds to the start recording command, enabling the data acquisition module to enable the channel and the data storage module to start asynchronous disk writing; responds to the stop recording command, issuing a stop reading signal, realizing one-click start and stop of the acquisition and storage process, precise timing linkage, and process automation, solving the problems of cumbersome manual step-by-step operation, inconsistent start and stop timing, and easy operation leading to incomplete or redundant data storage in traditional systems; the status monitoring unit receives onboard status information, in-depth analysis results, and visualization reports and presents them on the interface, realizing real-time visualization of the entire link status, immediate fault detection, and intuitive presentation of analysis results, solving the technical pain points of traditional systems such as opaque status, delayed anomaly detection, obscure analysis results, and inability to grasp system operation and data quality in real time, forming a closed-loop human-computer interaction management system, improving system usability, operation and maintenance efficiency, and operational reliability.
[0099] Optionally, after receiving the stop read signal, the asynchronous disk write unit is further configured to perform tail data processing.
[0100] The tail data processing includes: waiting for the data acquisition module to complete the last DMA transfer, calculating the amount of tail scattered data in the circular buffer, using the HeapAlloc temporary buffer in conjunction with the CopyMemory operation to perform zero-padding alignment to ensure that the last write operation meets the alignment constraint of unbuffered write; and calling the SetFilePointerEx and SetEndOfFile interfaces to precisely truncate the file to the effective data size.
[0101] Specifically, this embodiment solves the technical problems of inconsistent alignment of fragmented data at the end of traditional systems, invalid and redundant files, excessive disk space usage, and insufficient data precision. It ensures compliant data file formats, data integrity, and efficient space utilization, thereby improving storage reliability and disk utilization.
[0102] In one embodiment, after the acquisition stop command (stopReading) is issued, the system executes a deterministic termination algorithm: waits for the acquisition thread to complete the last DMA transfer and updates the final read pointer; the disk write thread calculates the amount of scattered data at the end that is less than WRITE_SIZE; a "zero-padding alignment" strategy using a HeapAlloc temporary buffer plus CopyMemory is adopted to ensure that the last WriteFile still meets the alignment requirements for unbuffered writing; and SetFilePointerEx and SetEndOfFile are used to precisely truncate the file to the effective data size, eliminating the use of invalid padding data to occupy disk space.
[0103] Optionally, the data storage module is also used to cooperate with the data acquisition module to form a two-level back pressure mechanism. By monitoring the DDR level of the FPGA acquisition unit and the level of the annular buffer, an elastic buffer layer is formed to control the onboard FIFO from overflowing when disk I / O jitter occurs.
[0104] Specifically, this embodiment solves the technical problem that instantaneous disk I / O fluctuations in traditional systems can easily cause front-end FIFO overflow and data loss, effectively tolerates short-term disk performance fluctuations, achieves zero data loss guarantee, and enhances the stability and reliability of the system under complex operating conditions.
[0105] Combination Figure 2 As shown, this embodiment of the invention also provides a high-speed storage method for optical fiber data, including:
[0106] External fiber optic data is acquired, and the external fiber optic data is captured and preprocessed through the FPGA platform to obtain the preprocessed data stream and onboard status information.
[0107] A circular buffer is allocated based on NUMA topology-aware technology, the data stream is cached in the circular buffer, and lock-free synchronization and large-granularity asynchronous disk persistence processing is performed on the data stream to obtain a data file that has been stored.
[0108] The data file is subjected to overall data flow control processing to obtain the processed data file;
[0109] The system acquires user operation instructions, generates and distributes operation control instructions based on these instructions, controls the execution of corresponding actions, and combines onboard status information with in-depth analysis results for visualization.
[0110] The high-speed optical fiber data storage method of the present invention has the same advantages over the prior art as the aforementioned high-speed optical fiber data storage system, and will not be repeated here.
[0111] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.
Claims
1. A high-speed optical fiber data storage system, characterized in that, The system includes: an interface conversion module, a data acquisition module, a data storage module, a data processing module, and a software module. The interface conversion module is used to adapt to optical fiber communication cables of different specifications and receive external optical fiber data, and send the external optical fiber data to the data acquisition module; The data acquisition module is used to capture and preprocess the external optical fiber data through the FPGA platform to obtain the preprocessed data stream and onboard status information, and transmit the preprocessed data stream to the data storage module, while sending the onboard status information to the software module. The data storage module is used to allocate a circular buffer according to NUMA topology-aware technology, receive and cache the data stream in the circular buffer, perform lock-free synchronization and large-granularity asynchronous disk persistence processing on the data stream, obtain the stored data file, and send it to the data processing module. The data processing module is used to perform overall data flow control processing on the data file and send the processed data file to the software module. The software module is used to receive user operation instructions and interact with the onboard status information to construct a human-computer interaction loop. The software module generates and distributes operation control instructions according to the user operation instructions to control the data acquisition module, data storage module and data processing module to perform corresponding actions, and at the same time receives feedback from each module.
2. The high-speed optical fiber data storage system according to claim 1, characterized in that, The data acquisition module includes an FPGA acquisition unit, a clock configuration unit, and a power management unit. The FPGA acquisition unit is used to receive the external optical fiber signal, perform protocol parsing and buffering on the external optical fiber signal to obtain the data stream and the onboard status information, transmit the data stream to the data storage module via DMA, and send the onboard status information to the software module at the same time. The clock configuration unit is used to configure the clock synthesizer, generate a clock signal, and provide a working clock for the FPGA acquisition unit. The power management unit is used to provide the core power and adjustable ADJ voltage to the FPGA acquisition unit through a power chip.
3. The high-speed optical fiber data storage system according to claim 2, characterized in that, The FPGA acquisition unit is also used to receive the binary configuration file sent by the software module, and transmit the binary configuration file to the Flash via the onboard DDR for online re-soldering of the FPGA logic to adapt to different fiber optic communication protocols.
4. The high-speed optical fiber data storage system according to claim 1, characterized in that, The data storage module includes a NUMA binding unit, a lock-free buffer unit, and an asynchronous disk persistence unit. The NUMA binding unit is used to call the GetNumaHighestNodeNumber interface to detect the server's NUMA topology, allocate a preset number of bytes of the circular buffer for each acquisition channel, and use the VirtualAllocExNuma interface to force the physical memory of the circular buffer to be allocated locally on the NUMA node closest to the acquisition card's PCIe slot, while aligning the starting address of the circular buffer. The lock-free buffer unit is used to monitor the buffer level using atomic long integer pointers readTotal and writtenTotal, and to perform data read and write pointer updates for lock-free synchronization of the multi-channel data stream; The asynchronous disk write unit is used to disable system caching and enable asynchronous I / O mode to initiate a disk write request when the amount of unprocessed data in the circular buffer reaches a preset number of bytes. It also processes the write completion event through the I / O completion port model and auxiliary thread pool to perform large-granular asynchronous disk write processing on the data stream.
5. A high-speed optical fiber data storage system according to claim 4, characterized in that, The data storage module also includes a thread affinity unit. The thread affinity unit is used to bind the data transfer thread of the data acquisition module and the write thread of the asynchronous disk write unit to the processor cores in the corresponding NUMA nodes through the SetThreadAffinityMask interface.
6. A high-speed optical fiber data storage system according to claim 1, characterized in that, The data processing module includes a parsing unit, a processing unit, and a central control unit. The parsing unit is used to read the data file and parse the original optical fiber data in the data file; The processing unit is used to perform data playback processing on the parsed data. The central control unit is used to coordinate and schedule the data parsing, processing, and playback of the data file, and to send the processed data file to the software module.
7. A high-speed optical fiber data storage system according to claim 1, characterized in that, The software module includes a control command unit and a status monitoring unit. The control command unit is used to respond to the user operation command, generate corresponding operation control commands, and send the operation control commands to the data acquisition module, the data storage module, and the data processing module respectively, so as to control each module to perform corresponding actions. The control of each module to perform corresponding actions includes: controlling the data acquisition module to enable the corresponding channel and controlling the data storage module to start asynchronous disk writing; and responding to the stop recording command in the user operation command by sending a stop reading signal to the data storage module. The status monitoring unit is used to receive the onboard status information fed back by the data acquisition module and the in-depth analysis results and visualization reports generated by the data processing module in real time, and to feed back the onboard status information and in-depth analysis results to the human-computer interaction interface for dynamic updates.
8. A high-speed optical fiber data storage system according to claim 7, characterized in that, After receiving the stop read signal, the asynchronous disk write unit is also used to perform tail data processing. The tail data processing includes: waiting for the data acquisition module to complete the last DMA transfer, calculating the amount of tail scattered data in the circular buffer, using the HeapAlloc temporary buffer in conjunction with the CopyMemory operation to perform zero-padding alignment to ensure that the last write operation meets the alignment constraint of unbuffered write; and calling the SetFilePointerEx and SetEndOfFile interfaces to precisely truncate the file to the effective data size.
9. A high-speed optical fiber data storage system according to claim 8, characterized in that, The data storage module is also used to cooperate with the data acquisition module to form a two-level back pressure mechanism. By monitoring the DDR level of the FPGA acquisition unit and the level of the annular buffer, an elastic buffer layer is formed to control the onboard FIFO from overflowing when disk I / O jitter occurs.
10. A high-speed storage method for optical fiber data, characterized in that, The method includes: External fiber optic data is acquired, and the external fiber optic data is captured and preprocessed through the FPGA platform to obtain the preprocessed data stream and onboard status information. A circular buffer is allocated based on NUMA topology-aware technology, the data stream is cached in the circular buffer, and lock-free synchronization and large-granularity asynchronous disk persistence processing is performed on the data stream to obtain a data file that has been stored. The data file is subjected to overall data flow control processing to obtain the processed data file; The system acquires user operation instructions, generates and distributes operation control instructions based on these instructions, controls the execution of corresponding actions, and combines onboard status information with in-depth analysis results for visualization.