A large-scale epics data archiving method based on time series database

Abstract

This invention relates to the field of experimental data storage technology for nuclear fusion devices, and discloses a method for large-scale EPICS data archiving based on a time-series database. The method involves acquiring process variables in an event-driven acquisition mode through an embedded channel access protocol data acquisition module and writing them into buffered memory channels. Through group aggregation and batch persistence, the process variables are grouped into fixed-size groups, each group corresponding to an aggregation coroutine. The latest event data is read non-blockingly from the buffered memory channels of each group at fixed intervals. Metadata mapping is used to convert the process variable names into device paths and measurement value names in the time-series database, assembling them into a sparse wide table. Null value columns in the sparse wide table are filtered, and the data is then batch-written into a distributed time-series database through a connection pool. This invention achieves stable high-throughput writing and low-latency retrieval of large-scale experimental data in the EPICS system, and provides an efficient data access interface for subsequent artificial intelligence training and inference platforms.

Description

Technical Field

[0001] This invention relates to the field of experimental data storage technology for nuclear fusion devices, and specifically to a large-scale EPICS data archiving method based on a time-series database. Background Technology

[0002] EPICS (Experimental Physics and Industrial Control System) is a collection of open-source software tools, libraries, and applications widely used to build distributed soft real-time control systems, serving scientific instruments such as particle accelerators, telescopes, and other large-scale scientific experimental facilities.

[0003] The EPICS system can broadcast and transmit physical quantities within a local area network in the form of process variables (PVs). Reliable archiving and rapid retrieval of this high-frequency time series data are crucial for real-time monitoring, fault diagnosis, and physical analysis of the equipment. However, current EPICS data archiving technologies suffer from the following main shortcomings: Archiving solutions based on relational databases (such as the EPICS Archiver system based on MySQL) suffer from high query latency when dealing with large-scale time-series data, and lack columnar compression and automatic partitioning mechanisms for time-series data, resulting in low storage space utilization.

[0004] While file system serialization-based archiving solutions (such as the EPICS Archiver Appliance data archiving system and the Channel Archiver data archiving system) improve instantaneous write throughput through multi-level memory buffering, they lack the database's native aggregation calculation and cross-device joint query capabilities, and cannot directly support complex query scenarios such as downsampling and multi-dimensional correlation analysis.

[0005] The existing EPICS data archiving system lacks efficient streaming buffering and batch aggregation mechanisms. Process variables acquired at the acquisition layer are passed to the storage end in a fine-grained manner, resulting in the database facing high-frequency small transaction writes, and the write throughput drops sharply as the number of process variables increases.

[0006] The existing archiving system lacks integration capabilities with artificial intelligence inference platforms. Mainstream EPICS data archiving solutions store data in proprietary file formats or relational databases. When using artificial intelligence platforms for model training and inference, format conversion and data migration are required, which increases the complexity of the data pipeline and reduces end-to-end inference efficiency. Summary of the Invention

[0007] To address the aforementioned technical challenges, this invention provides a method for large-scale EPICS data archiving based on a time-series database. This method constructs a layered pipeline architecture comprising event-driven acquisition, streaming buffering, group aggregation, and batch persistence. First, an embedded channel access protocol data acquisition module stably acquires massive amounts of high-frequency process variables (PVs) in an event-driven manner and writes them to a buffered memory channel. Second, through group aggregation and batch persistence modules, combined with a metadata mapping mechanism, the PVs are dynamically converted into the native storage structure of the time-series database, and then evenly routed to the underlying distributed time-series database cluster using a sparse wide-table batch write approach. This invention achieves stable, high-throughput archiving writes of hundreds of thousands of PVs and, relying on the columnar storage and native aggregation computing capabilities of the time-series database, significantly reduces the retrieval latency of long-span historical data. Simultaneously, it provides a high-efficiency, migration-free data access interface for artificial intelligence training and inference platforms.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for large-scale EPICS data archiving based on time-series databases, including The process variables are acquired through the embedded channel access protocol data acquisition module in an event-driven acquisition mode and written to a buffered memory channel. By grouping aggregation and batch persistence, process variables are grouped into fixed sizes, with each group corresponding to an aggregation coroutine. The latest event data is read non-blockingly from the buffered memory channels of each group at fixed intervals. Metadata mapping is used to convert process variable names into device paths and measurement value names in the time series database, and then assembled into sparse wide tables. Filter out null columns in the sparse wide table and write them to the distributed time-series database in batches using a connection pool.

[0009] In one embodiment, the embedded channel access protocol data acquisition module is implemented in the following way: The EPICS system's channel access protocol client library is embedded across languages ​​into the main process runtime environment, and the calling interface for creating channel access and monitoring subscriptions is encapsulated. Initialize the channel access protocol context and create independent input / output threads, and manage all TCP connections with the input / output controller through the input / output multiplexing event loop; Create a buffered memory channel for each process variable to be archived, register a monitoring subscription, and establish a hash mapping table of process variable names to the buffered memory channels; When event data for a process variable arrives, the input / output thread actively parses it and triggers a callback function. The callback function looks up the corresponding buffered memory channel through a hash map and writes the event data in a non-blocking manner. If the buffered memory channel is full, the event data is discarded and the discard counter is incremented.

[0010] In one embodiment, the step of embedding the EPICS system's channel access protocol client library across languages ​​into the main process runtime environment and encapsulating the call interface for creating channel access and monitoring subscriptions specifically includes: By using a cross-language interoperability mechanism, the EPICS system's channel access protocol client library is embedded into the main process runtime environment of the archiving system. Protocol layer wrapper functions are defined to encapsulate the call interfaces for creating channel access and monitoring subscriptions. Application layer exported functions are defined as cross-language bridging entry points for channel access protocol callback functions, enabling the channel access protocol stack and the business logic of the archiving system to run in the same process address space.

[0011] In one embodiment, the initialization of the channel access protocol context and the creation of independent input / output threads, and the management of all TCP connections with the input / output controller through an input / output multiplexing event loop, specifically include: The channel access protocol context is initialized and preemptive callback mode is enabled, allowing the channel access protocol client library to create an independent input / output thread within the process. This input / output thread runs the input / output multiplexing event loop and manages all TCP connections with the input / output controller. In event-driven acquisition mode, when a process variable arrives, the input / output thread actively parses and triggers the callback function registered for that process variable, eliminating the need for the archiving system to poll the event loop.

[0012] In one embodiment, the step of creating a buffered memory channel for each process variable to be archived, registering a monitoring subscription, and establishing a hash mapping table of process variable names to the buffered memory channel specifically includes: Load the list of all process variable names to be archived from the configuration file, create a buffered memory channel for each process variable as a streaming data buffer; establish access logic channels for each channel and register monitoring subscriptions, and pass the process variable name identifier as user data to the callback function; perform a batch network refresh after each batch of monitoring subscriptions is completed; at the same time, establish a global hash mapping table of process variable names to buffered memory channels.

[0013] In one embodiment, when event data for the process variable arrives, the input / output thread actively parses and triggers a callback function. The callback function looks up the corresponding buffered memory channel through a hash map and writes the event data in a non-blocking manner. If the buffered memory channel is full, the event data is discarded and a discard counter is incremented. Specifically, this includes: When the input / output controller pushes event data for process variables, the event data reaches the input / output thread via the kernel buffer. After the channel access protocol client library parses the channel access protocol binary packet, it triggers a callback function. The callback function extracts the measurement value and timestamp from the event data, performs a time base conversion, and then calls the application layer callback through a cross-language bridge. The application layer callback looks up the corresponding buffered memory channel in the hash map table based on the process variable name and writes the event data in a non-blocking manner. If the buffered memory channel is full, the current event data is discarded and the atomic discard counter is incremented to ensure that the callback function does not block the input / output thread.

[0014] In one embodiment, the step of grouping and batch persistence, which involves grouping process variables into fixed-size groups, with each group corresponding to an aggregation coroutine, specifically includes: All process variables are grouped into fixed sizes, with each group corresponding to an aggregation coroutine. Each aggregation coroutine runs a timer independently, triggering a data aggregation and persistence operation once at a fixed period. The group size determines the number of columns written to the time-series database in each operation.

[0015] In one embodiment, the step of periodically and non-blockingly reading the latest event data from each group of buffered memory channels, using metadata mapping to convert process variable names into device paths and measurement value names in a time-series database, and assembling them into a sparse wide table specifically includes: When each aggregation coroutine is triggered by a timer, it iterates through all buffered memory channels corresponding to process variables in its group and attempts to read the latest event data one by one in a non-blocking manner; it records the values ​​of process variables with event data and marks process variables without event data as empty; it uses the current system time as a unified timestamp to align the event data of all process variables in the same group to the same time point, and automatically converts the process variable names into device paths and measurement value names in the time series database through a metadata mapping mechanism, assembling them into a sparse wide table.

[0016] In one embodiment, filtering null columns in the sparse wide table specifically includes: Before writing to the time series database, the value list of the sparse wide table is traversed, and all columns corresponding to null values ​​are filtered out, so that only process variables with event data are constructed.

[0017] In one embodiment, the step of batch writing to the distributed time-series database via a connection pool specifically includes: A connection is obtained from the connection pool of the time-series database. The batch write interface is called to complete the writing of the sparse wide table in a single remote procedure call. After the writing is completed, the connection is returned to the connection pool.

[0018] Compared with the prior art, the beneficial technical effects of the present invention are: This invention uses a distributed time-series database as a persistent storage layer to replace the traditional relational database in the EPICS system, realizing an archiving system for time-series data, which significantly reduces the number of retrieval events and storage space usage for massive amounts of time-series data.

[0019] Compared to file storage and relational database storage, the time-series database used in this invention natively provides database-level computing capabilities such as aggregation calculation, downsampling query, and cross-device joint query, thereby improving the efficiency of engineering data analysis.

[0020] By introducing streaming buffering and group aggregation mechanisms between the data acquisition layer and the persistence layer, this invention constructs a layered aggregation architecture consisting of event-driven acquisition, buffered memory channels, timed aggregation, and batch writing. At the data acquisition end, the Channel Access (CA) protocol receives data in an event-driven manner and transmits it to the aggregation layer through a non-blocking buffered memory channel, avoiding additional inter-process communication overhead and data serialization redundancy. At the persistence end, the group aggregation mechanism reduces the frequency of database write requests by several orders of magnitude. This improves the system's write throughput and resource utilization efficiency in scenarios with hundreds of thousands of process variables (PVs).

[0021] This invention uses a distributed time-series database that supports standard database access interfaces as a unified storage layer, enabling archived nuclear fusion experimental data to be directly accessed by artificial intelligence training frameworks and inference platforms without conversion or cross-system data migration, thereby improving the availability and inference efficiency of nuclear fusion engineering time-series data in artificial intelligence application scenarios. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system architecture diagram of the present invention; Figure 3 The present invention is shown in the flowchart below, wherein (a) is the main program flowchart and (b) is the flowchart of the aggregate coroutine subroutine. Detailed Implementation

[0023] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] like Figure 1 As shown, a large-scale EPICS data archiving method based on a time-series database in this invention includes the following steps: S1, through the embedded channel access protocol data acquisition module, acquires process variables in event-driven acquisition mode and writes them to the buffered memory channel. S2, through group aggregation and batch persistence, groups process variables into fixed-size groups, each group corresponding to an aggregation coroutine, and reads the latest event data from the buffered memory channels of each group in a non-blocking manner at fixed intervals. It uses metadata mapping to convert process variable names into device paths and measurement value names in the time series database and assembles them into sparse wide tables. S3 filters null columns in sparse wide tables and writes them in batches to a distributed time-series database via a connection pool.

[0025] like Figure 2 , Figure 3 As shown, the present invention will be described in detail below in several parts.

[0026] 1. Streaming data acquisition based on the Channel Access (CA) protocol.

[0027] Establish a cross-language binding layer. Through a cross-language interoperability mechanism, the official EPICS system CA protocol client library is directly embedded into the main process runtime environment of the archiving system. Protocol layer wrapper functions are defined to encapsulate the call interfaces for creating channel access and monitoring subscriptions. Application layer exported functions are defined as the cross-language bridge entry point for protocol callbacks, ensuring that the CA protocol stack and the business logic of the archiving system run within the same process address space.

[0028] Create a CA protocol context and enable event-driven acquisition mode. Initialize the CA protocol context and enable preemptive callback mode, allowing the channel access protocol client library to create an independent input / output (I / O) thread within the process. This I / O thread runs an I / O multiplexing event loop, managing all TCP connections with the input / output controller (IOC). In this mode, when a process variable (PV) arrives, the I / O thread actively parses and triggers the registered callback function, eliminating the need for the archiving system to poll the event loop.

[0029] Register PV subscriptions and establish callback-buffered memory channel mappings. Load the complete list of PV names to be archived from the configuration file, and create a buffered memory channel as a streaming data buffer for each PV. Establish CA logical channels one by one and register monitoring subscriptions, passing the PV name identifier as user data to the callback. After each batch of subscriptions is completed, perform a batch network refresh (i.e., refresh all arriving event data into memory) to balance network round-trip overhead and memory backlog. At the same time, establish a global hash mapping table of PV names to buffered memory channels, initialize it to read-only, and allow the callback function to look up data without locks.

[0030] Event-driven data reception and non-blocking buffering. When the I / O controller pushes PV event data, TCP data arrives at the I / O thread via the kernel buffer. The channel access protocol client library parses the CA protocol binary packet and triggers a callback function. The callback function extracts the measurement value and timestamp from the data structure, performs time base conversion, and then calls the application layer callback through a cross-language bridge. The application layer callback looks up the corresponding buffered memory channel in the hash map table based on the PV name and writes the event data in a non-blocking manner; if the channel is full, the current data is discarded and the atomic discard counter is incremented to ensure that the callback function never blocks the I / O thread.

[0031] 2. Grouping, aggregation, and persistence methods for time-series databases.

[0032] PV grouping and aggregation goroutine scheduling. All PVs are grouped into fixed-size groups, each group corresponding to a lightweight aggregation goroutine. Each aggregation goroutine runs an independent timer, triggering a data aggregation and persistence operation at fixed intervals. The group size determines the number of columns written in each database write operation, i.e., the number of measurements written in a single Remote Procedure Call (RPC).

[0033] Timed aggregation, metadata mapping, and sparse table assembly are employed. Each aggregation coroutine, triggered by a timer, iterates through the buffered memory channels corresponding to all PVs in its group, attempting to read the latest event data one by one in a non-blocking manner. For PVs with data, their values ​​are recorded; PVs without data are marked as empty. The current system time is used as a unified timestamp to align the data of all PVs within the same group to the same point in time. Through a metadata mapping mechanism, PV names are automatically converted into device paths and measurement value names in the time-series database, assembling them into a sparse wide table data structure.

[0034] Sparse column filtering and connection pooling for batch writes. Before writing to the time-series database, the value list of the wide table is traversed, filtering out columns corresponding to all null values. Measurement patterns are constructed only for variables with data, reducing network transmission volume and database storage overhead. Connections are obtained through a database connection pool, and the batch write interface is called to complete the wide table write in a single RPC. After the write is complete, the connection is returned to the connection pool. The connection pool supports a large number of concurrent aggregation coroutine write requests with a small number of connections, and avoids database connection resource exhaustion through a queuing and reuse mechanism.

[0035] 3. Native integration of time-series databases with AI inference platforms.

[0036] The persistence layer uses a distributed time-series database that natively supports integration with mainstream AI training frameworks. Archived data can be directly accessed by the AI ​​platform through the database connector without format conversion, eliminating the intermediate steps of data migration and format conversion in traditional archiving solutions.

[0037] This invention aims to address the high latency issue in long-span retrieval of large-scale, high-frequency real-time data from large scientific devices such as tokamaks, as present in the current EPICS archiving system. Nuclear fusion device data is transmitted to a time-series database real-time archiving framework via the EPICS system's Channel Access (CA) protocol in a PV (PV) manner. This framework employs a hierarchical aggregation architecture to stably acquire and archive large-scale, high-frequency experimental data into the time-series database. First, an embedded channel access protocol data acquisition module acquires the large-scale, high-frequency experimental data. Then, using a group aggregation and batch persistence module and a metadata mapping mechanism, the data is dynamically converted into the native storage structure of the time-series database and evenly routed to the underlying distributed time-series database cluster. This invention achieves stable, high-throughput writing and low-latency retrieval of large-scale experimental data in the EPICS system and provides an efficient data access interface for subsequent artificial intelligence training and inference platforms, significantly improving the storage access efficiency and intelligent analysis capabilities of nuclear fusion device engineering data.

[0038] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0039] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0040] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0041] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0042] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for large-scale EPICS data archiving based on a time-series database, characterized in that, include The process variables are acquired through the embedded channel access protocol data acquisition module in an event-driven acquisition mode and written to a buffered memory channel. By grouping aggregation and batch persistence, process variables are grouped into fixed sizes, with each group corresponding to an aggregation coroutine. The latest event data is read non-blockingly from the buffered memory channels of each group at fixed intervals. Metadata mapping is used to convert process variable names into device paths and measurement value names in the time series database, and then assembled into sparse wide tables. Filter out null columns in the sparse wide table and write them to the distributed time-series database in batches using a connection pool.

2. The method for large-scale EPICS data archiving based on a time-series database according to claim 1, characterized in that, The embedded channel access protocol data acquisition module is implemented in the following way: The EPICS system's channel access protocol client library is embedded across languages ​​into the main process runtime environment, and the calling interface for creating channel access and monitoring subscriptions is encapsulated. Initialize the channel access protocol context and create independent input / output threads, and manage all TCP connections with the input / output controller through the input / output multiplexing event loop; Create a buffered memory channel for each process variable to be archived, register a monitoring subscription, and establish a hash mapping table of process variable names to the buffered memory channels; When event data for a process variable arrives, the input / output thread actively parses it and triggers a callback function. The callback function looks up the corresponding buffered memory channel through a hash map and writes the event data in a non-blocking manner. If the buffered memory channel is full, the event data is discarded and the discard counter is incremented.

3. The method for large-scale EPICS data archiving based on a time-series database according to claim 2, characterized in that, The step of embedding the EPICS system's channel access protocol client library across languages ​​into the main process runtime environment and encapsulating the calling interfaces for creating channel access and monitoring subscriptions specifically includes: By using a cross-language interoperability mechanism, the EPICS system's channel access protocol client library is embedded into the main process runtime environment of the archiving system. Protocol layer wrapper functions are defined to encapsulate the call interfaces for creating channel access and monitoring subscriptions. Application layer exported functions are defined as cross-language bridging entry points for channel access protocol callback functions, enabling the channel access protocol stack and the business logic of the archiving system to run in the same process address space.

4. The method for large-scale EPICS data archiving based on a time-series database according to claim 2, characterized in that, The initialization of the channel access protocol context and the creation of independent input / output threads, along with the management of all TCP connections with the input / output controller through an input / output multiplexing event loop, specifically include: The channel access protocol context is initialized and preemptive callback mode is enabled, allowing the channel access protocol client library to create an independent input / output thread within the process. This input / output thread runs the input / output multiplexing event loop and manages all TCP connections with the input / output controller. In event-driven acquisition mode, when a process variable arrives, the input / output thread actively parses and triggers the callback function registered for that process variable, eliminating the need for the archiving system to poll the event loop.

5. A method for large-scale EPICS data archiving based on a time-series database according to claim 2, characterized in that, The process of creating a buffered memory channel for each process variable to be archived, registering a monitoring subscription, and establishing a hash mapping table of process variable names to the buffered memory channel specifically includes: Load the list of all process variable names to be archived from the configuration file, create a buffered memory channel for each process variable as a streaming data buffer; establish access logic channels for each channel and register monitoring subscriptions, and pass the process variable name identifier as user data to the callback function; perform a batch network refresh after each batch of monitoring subscriptions is completed; at the same time, establish a global hash mapping table of process variable names to buffered memory channels.

6. A method for large-scale EPICS data archiving based on a time-series database according to claim 2, characterized in that, When event data for the process variable arrives, the input / output thread actively parses and triggers a callback function. The callback function looks up the corresponding buffered memory channel through a hash map and writes the event data in a non-blocking manner. If the buffered memory channel is full, the event data is discarded and a discard counter is incremented. Specifically, this includes: When the input / output controller pushes event data for process variables, the event data reaches the input / output thread via the kernel buffer. After the channel access protocol client library parses the channel access protocol binary packet, it triggers a callback function. The callback function extracts the measurement value and timestamp from the event data, performs a time base conversion, and then calls the application layer callback through a cross-language bridge. The application layer callback looks up the corresponding buffered memory channel in the hash map table based on the process variable name and writes the event data in a non-blocking manner. If the buffered memory channel is full, the current event data is discarded and the atomic discard counter is incremented to ensure that the callback function does not block the input / output thread.

7. A method for large-scale EPICS data archiving based on a time-series database according to claim 1, characterized in that, The process variables are grouped into fixed-size groups through group aggregation and batch persistence, with each group corresponding to an aggregation coroutine. Specifically, this includes: All process variables are grouped into fixed sizes, with each group corresponding to an aggregation coroutine. Each aggregation coroutine runs a timer independently, triggering a data aggregation and persistence operation once at a fixed period. The group size determines the number of columns written to the time-series database in each operation.

8. A method for large-scale EPICS data archiving based on a time-series database according to claim 1, characterized in that, The process involves periodically and non-blockingly reading the latest event data from each group of buffered memory channels, using metadata mapping to convert process variable names into device paths and measurement value names in the time-series database, and assembling them into a sparse wide table. Specifically, this includes: When each aggregation coroutine is triggered by a timer, it iterates through all buffered memory channels corresponding to process variables in its group and attempts to read the latest event data one by one in a non-blocking manner; it records the values ​​of process variables with event data and marks process variables without event data as empty; it uses the current system time as a unified timestamp to align the event data of all process variables in the same group to the same time point, and automatically converts the process variable names into device paths and measurement value names in the time series database through a metadata mapping mechanism, assembling them into a sparse wide table.

9. A method for large-scale EPICS data archiving based on a time-series database according to claim 1, characterized in that, The filtering of null columns in the sparse wide table specifically includes: Before writing to the time series database, the value list of the sparse wide table is traversed, and all columns corresponding to null values ​​are filtered out, so that only process variables with event data are constructed.

10. A method for large-scale EPICS data archiving based on a time-series database according to claim 1, characterized in that, The step of batch writing to the distributed time-series database via a connection pool specifically includes: A connection is obtained from the connection pool of the time-series database. The batch write interface is called to complete the writing of the sparse wide table in a single remote procedure call. After the writing is completed, the connection is returned to the connection pool.

Images