A data state driving-based DAS data adaptive scheduling processing system and method

Abstract

This invention discloses a data state-driven adaptive scheduling and processing system and method for DAS data. The system includes a host computer and a DAS acquisition terminal. The host computer and the DAS acquisition terminal establish a RoCE link via a 10 Gigabit Ethernet network. The host computer is equipped with a data processing terminal and a display and control module. The data processing terminal is communicatively connected to the DAS data acquisition terminal. The data processing terminal performs data reading operations based on whether the data state meets preset conditions, calculates the data collected by the DAS data acquisition terminal, and sends the processing results to the display and control module. The method includes the following steps: S1, link initialization; S2, state query and adaptive data retrieval; S3, data processing; S4, result output. Compared with existing technologies, this invention has the advantages of good data availability, high system stability, and strong real-time processing capabilities.

Description

Technical Field

[0001] This invention belongs to the field of fiber optic sensing data processing technology, and in particular to a data state-driven DAS data adaptive scheduling and processing system and method. Background Technology

[0002] Distributed fiber optic vibration sensing systems (DAS) generate gigabytes of data per second per channel in long-distance, high-resolution scenarios. However, the processing capacity and system load of the data processing end fluctuate dynamically, making it difficult to guarantee the reliability, consistency, and availability of data during transmission under the existing data transmission mode that mainly relies on the acquisition end to actively push data. This, in turn, affects the stability of real-time analysis and processing.

[0003] Specifically, the existing technology has the following technical problems: 1. The acquisition end cannot perceive the real-time processing status of the data processing end when sending data, resulting in a lack of effective control over data transmission behavior; 2. The data processing end cannot determine whether the received data is the latest valid data or complete data during the receiving stage; 3. In high-bandwidth continuous acquisition scenarios, data transmission and data processing are coupled, which can easily lead to data loss or processing abnormalities; 4. When it is necessary to repeatedly analyze data at a specific time or location, there is a lack of a controllable data access mechanism. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a data state-driven DAS data adaptive scheduling and processing system, which achieves advantages such as good data availability, high system stability, and strong real-time processing capabilities.

[0005] The technical solution of this invention is as follows: A data state-driven adaptive scheduling and processing system for DAS data includes a host computer and a DAS acquisition terminal. The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit Ethernet. The host computer is equipped with a data processing terminal and a display and control module. The data processing terminal is communicatively connected to the DAS data acquisition terminal. The data processing terminal performs data reading operations based on whether the data state meets preset conditions, calculates the data collected by the DAS data acquisition terminal, and sends the processing results to the display and control module.

[0006] Preferably, the data processing terminal includes a status query module, a data processing module, and an anomaly and duplicate query module. The status query module polls the data refresh status of the DAS acquisition terminal using a data status query protocol. The data processing module has a built-in multi-threaded data processing algorithm. During data refresh, it reads data from remote memory through the RDMA data protocol to realize real-time calculation and processing of the acquired data. The anomaly and duplicate query module judges the abnormal situation of the received data and performs duplicate queries to realize interruption when data is abnormal and repeated access and analysis of the data after recovery.

[0007] This invention also provides a data state-driven adaptive scheduling and processing method for DAS data, which employs the aforementioned data state-driven adaptive scheduling and processing system for DAS data and includes the following steps: S1. Link Initialization: The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit network and perform initialization. S2. Status Query and Adaptive Data Retrieval: The host computer software uses the data status query protocol to poll the data refresh status of the DAS acquisition terminal; based on the status query result, if the data has been refreshed, the data is read from the remote memory according to the "address offset" through the RDMA data protocol. S3. Data Processing: The pulled data enters the analysis thread pool and uses a multi-threaded data processing algorithm to perform data processing operations such as intensity phase calculation, short-time energy calculation, and FFT transformation. S4. Output Results: Display the processing results on the host computer.

[0008] Preferably, the status query and adaptive data retrieval in step S2 includes the following steps: (1) The acquisition end writes the acquired data into the preset data storage area and updates the corresponding data status information and data version identifier; (2) The data processing end sends a data status query request to the acquisition end to obtain the current data status information; (3) The acquisition terminal returns data status information and the corresponding data version identifier; (4) The data processing terminal determines whether the current data meets the preset data access conditions based on the data status information and data version identifier; (5) When the access conditions are met, the data processing terminal performs a data reading operation based on the preset logical data base address and address offset; (6) If the access conditions are not met, terminate or delay the data reading operation.

[0009] Preferably, in step (1), the data status information is represented by a status code, a flag bit, or other equivalent method.

[0010] Preferably, in step (1), the data version identifier is implemented using a serial number, timestamp, or other combination of methods.

[0011] Preferably, in step (5), the data reading operation is implemented through Remote Direct Memory Access (RoCE) or by using iWARP or other communication methods that support remote memory access.

[0012] This invention utilizes RoCE technology (RDMA technology based on converged Ethernet) to construct a physically-aware asynchronous fetch mechanism, achieving zero-copy transmission, hard verification of data consistency, and adaptive flow control for DAS service logic. It features the following characteristics: 1. A data access control mechanism driven by data status, with control logic that dynamically initiates RDMA read requests by the data processing end based on the data refresh status; 2. The data access process design, led by the data processing end, enables adaptive scheduling of DAS data; 3. The data version identifier participates in the data access decision-making mechanism, which uses a combination of fixed logical data base address and address offset to complete data location; 4. The decoupled design of the data status query mechanism and the data reading mechanism means that the acquisition end and the processing end do not need to use PCIe data transmission, and the distance between them is completely limited by the distance of the network cable, enabling remote processing.

[0013] The beneficial effects of this invention are as follows: 1. To achieve controllability of the data access process and improve data availability, the processing end can autonomously schedule hardware resources through software instructions according to its own processing capabilities and business needs; 2. Reduce the risk of data processing terminal anomalies under high load scenarios. RoCE's hardware retransmission mechanism ensures physical-level reliability of data transmission. 3. Supports on-demand access and repeated analysis of data from the acquisition terminal. By calculating the offset through software, it enables random retrieval of any data from the acquisition terminal's DDR at the same acquisition time. 4. Improve the overall stability and real-time processing capability of the system. By leveraging the characteristics of RoCE, the CPU computing power originally used by the DAS device for protocol stack parsing is released to the core demodulation algorithm. 5. It is suitable for real-time transmission and processing of high-speed continuous data. The closed-loop feedback mechanism completely eliminates situations such as "data misalignment" or "invalid analysis" caused by PC system lag, and at the same time, it has a problem repair method to reduce the risk of data errors. Attached Figure Description

[0014] Figure 1 This is a module connection diagram of a data state-driven DAS adaptive data scheduling and processing system according to the present invention. Figure 2 This is a flowchart illustrating the process of step S2, status query, and adaptive data reading in a data state-driven DAS data adaptive scheduling and processing method according to the present invention. Detailed Implementation

[0015] The present invention will be further described below with reference to specific embodiments and accompanying drawings: DAS systems are widely used in fiber optic sensing, structural monitoring, and other fields. They are characterized by high data sampling rates, strong data continuity, and high real-time requirements. The system typically uses an FPGA as the front-end data acquisition and preprocessing unit, and a PC as the back-end data receiving and analysis unit. For DAS systems with technical specifications of 100km measurement distance at a 1kHz sampling rate or 50km measurement distance at a 2kHz sampling rate, continuous acquisition is performed once per second. The data volume calculation formula is: measurement distance / spatial resolution * sampling rate * number of data bytes * number of channels, i.e., 50000m / 4m * 2000 * 32bit * 1, which translates to 1GB. For DAS systems, ensuring the accuracy of GB-level data per second is crucial.

[0016] In engineering practice, to reduce communication protocol overhead and increase effective data bandwidth, DAS acquisition systems and PC analysis units typically communicate using PCIe or simple UDP protocols. However, the separation of acquisition and analysis equipment, continuous data acquisition, and large data volumes (e.g., 100 million bytes per second) can easily lead to data loss or misalignment issues in high-load real-time scenarios. Even using Ethernet jumbo frames (MTU ≥ 9000 bytes) can reduce the number of frames for large data transmissions, lower frame header overhead, and improve transmission efficiency. Since the UDP protocol itself does not provide retransmission, acknowledgment, or flow control mechanisms, transmission instability still exists. Traditional TCP / IP protocol stacks consume significant CPU resources and incur considerable latency when processing large data volumes, affecting the real-time execution of demodulation algorithms.

[0017] In existing technologies, one approach uses local PCIe transmission, suitable for short-distance, small-volume data parsing; the other involves the acquisition end continuously encapsulating the acquired data and actively sending it over the network at a fixed rate, while the data processing end passively receives and processes the data sent by the acquisition end. In these solutions, the DAS acquisition end cannot perceive the processing status of the PC processing end, and the PC processing end cannot determine whether data loss has occurred or whether the current data is valid, thus easily leading to data unavailability during system load fluctuations. Furthermore, if the processing module needs to repeatedly process other data, it cannot back up the received data; it can directly obtain any data at the current processing moment from the DAS acquisition end via RDMA.

[0018] Therefore, the shortcomings of the existing technology are: 1. The data transmission process lacks an effective data access control mechanism; 2. The data processing end cannot determine whether the data meets the processing conditions before accessing the data; 3. The data transmission and data processing processes are strongly coupled, resulting in poor overall system stability; 4. It does not support on-demand access and repeated analysis of the data collected from the acquisition end.

[0019] This invention provides a data state-driven DAS data adaptive scheduling and processing system. Through a data access process led by the data processing end, combined with a data state determination mechanism and a data version consistency management mechanism, it achieves controllable access and reliable transmission of high-speed continuous data from the acquisition end, thereby improving data availability, system stability and real-time processing capabilities.

[0020] (a) Explanation of technical terms: 1. DAS: Distributed Fiber Optic Vibration Sensing System; 2. PCIe: High-speed serial computer expansion bus standard; 3. RDMA: Direct Remote Memory Access; 4. RoCE: Remote Direct Memory Access Protocol for Converged Ethernet; 5. Data version identifier: Identification information used to identify the update status of data frames in order to determine data consistency; 6. Address offset: The offset position of data reading relative to the preset fixed storage address. (II) Specific Implementation Examples: like Figure 1 As shown in the figure, an embodiment of the present invention provides a data state-driven DAS data adaptive scheduling and processing system, including a host computer and a DAS acquisition terminal. The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit network. The host computer is equipped with a data processing terminal and a display and control module. The data processing terminal is communicatively connected to the DAS data acquisition terminal. The data processing terminal performs data reading operations based on whether the data state meets preset conditions, calculates the data collected by the DAS data acquisition terminal, and sends the processing results to the display and control module.

[0022] Specifically, the data processing terminal includes a status query module, a data processing module, and an exception and duplicate query module. The status query module polls the data refresh status of the DAS acquisition terminal using a data status query protocol. The data processing module has a built-in multi-threaded data processing algorithm. During data refresh, it reads data from remote memory through the RDMA data protocol to realize real-time calculation and processing of the acquired data. The exception and duplicate query module judges the abnormal situation of the received data and performs duplicate queries to realize the functions of interruption when data is abnormal and repeated access and analysis of data after recovery.

[0023] This invention provides a data state-driven adaptive scheduling and processing method for DAS data, employing the aforementioned data state-driven adaptive scheduling and processing system for DAS data, and includes the following steps: S1. Link Initialization: The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit network and perform initialization. S2. Status Query and Adaptive Data Retrieval: The host computer software uses the data status query protocol to poll the data refresh status of the DAS acquisition terminal; based on the status query result, if the data has been refreshed, the data is read from the remote memory according to the "address offset" through the RDMA data protocol. S3. Data Processing: The pulled data enters the analysis thread pool and uses a multi-threaded data processing algorithm to perform data processing operations such as intensity phase calculation, short-time energy calculation, and FFT transformation. S4. Output Results: Display the processing results on the host computer.

[0024] like Figure 2 As shown, the present invention, based on data status indicators, includes the following steps for the status query and adaptive data retrieval operation in step S2: (1) The acquisition end writes the acquired data into the preset data storage area and updates the corresponding data status information and data version identifier; (2) The data processing end sends a data status query request to the acquisition end to obtain the current data status information; (3) The acquisition terminal returns data status information and the corresponding data version identifier; (4) The data processing terminal determines whether the current data meets the preset data access conditions based on the data status information and data version identifier; (5) When the access conditions are met, the data processing terminal performs a data reading operation based on the preset logical data base address and address offset; (6) If the access conditions are not met, terminate or delay the data reading operation.

[0025] Specifically, in step (1), the data status information can be represented by status codes, flag bits, or other equivalent methods.

[0026] Specifically, in step (1), the data version identifier can be implemented using a serial number, timestamp, or other combination of methods.

[0027] Specifically, in step (5), the data reading operation can be implemented through Remote Direct Memory Access (RoCE) technology (RDMA based on converged Ethernet), or iWARP or other communication methods that support remote memory access can be used, which reduces the system overhead during data transmission and ensures that the data processing end can adaptively schedule any required data at each acquisition moment.

[0028] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent modifications made based on the above embodiments are all within the scope of protection of the present invention.

Claims

1. A data state-driven DAS adaptive data scheduling and processing system, characterized in that, It includes a host computer and a DAS acquisition terminal. The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit network. The host computer is equipped with a data processing terminal and a display and control module. The data processing terminal is communicatively connected to the DAS data acquisition terminal. The data processing terminal performs data reading operations based on whether the data status meets preset conditions, calculates the data collected by the DAS data acquisition terminal, and sends the processing results to the display and control module.

2. The DAS data adaptive scheduling and processing system based on data state driving according to claim 1, characterized in that: The data processing terminal includes a status query module, a data processing module, and an anomaly and duplicate query module. The status query module polls the data refresh status of the DAS acquisition terminal using a data status query protocol. The data processing module has a built-in multi-threaded data processing algorithm. During data refresh, it reads data from remote memory through the RDMA data protocol to realize real-time calculation and processing of the acquired data. The anomaly and duplicate query module judges the abnormal situation of the received data and performs duplicate queries to realize interruption when data is abnormal and repeated access and analysis of the data after recovery.

3. A data state-driven adaptive scheduling and processing method for DAS data, characterized in that, Includes the following steps: S1. Link Initialization: The host computer and the DAS acquisition terminal establish a RoCE link through a 10 Gigabit network and perform initialization. S2. Status Query and Adaptive Data Retrieval: The host computer software uses the data status query protocol to poll the data refresh status of the DAS acquisition terminal; based on the status query result, if the data has been refreshed, the data is read from the remote memory via the RDMA data protocol. S3. Data Processing: The retrieved data enters the analysis thread pool, and various data processing operations are performed using a multi-threaded data processing algorithm. S4. Output Results: Display the processing results on the host computer.

4. The data state-driven DAS data adaptive scheduling processing method according to claim 3, characterized in that, The status query and adaptive data retrieval in step S2 include the following steps: (1) The acquisition end writes the acquired data into the preset data storage area and updates the corresponding data status information and data version identifier; (2) The data processing end sends a data status query request to the acquisition end to obtain the current data status information; (3) The acquisition terminal returns data status information and the corresponding data version identifier; (4) The data processing terminal determines whether the current data meets the preset data access conditions based on the data status information and data version identifier; (5) When the access conditions are met, the data processing terminal performs a data reading operation based on the preset logical data base address and address offset; (6) If the access conditions are not met, terminate or delay the data reading operation.

5. The data state-driven DAS data adaptive scheduling processing method according to claim 4, characterized in that: In step (1), the data status information is represented by a status code or a flag bit.

6. The DAS data adaptive scheduling processing method based on data state driving according to claim 4, characterized in that: In step (1), the data version identifier is implemented using a serial number or timestamp.

7. The data state-driven DAS data adaptive scheduling processing method according to claim 4, characterized in that: In step (5), the data reading operation is performed via Remote Direct Memory Access (RoCE) or using the iWARP communication method.

Images