A hybrid storage method and system for a laser accelerator control system

Abstract

The application discloses a hybrid storage method and system for a laser accelerator control system. The application comprises a data acquisition layer, a hybrid storage layer, a data cache and retrieval layer, and a data visualization layer. The hybrid storage layer adopts a relational database to store configuration data, a time series database to store process variables and image metadata, and a distributed object storage to store image data in a batch processing mode. The application realizes accurate alignment of images and process variables through a ring buffer and trigger time matching. The application adopts a memory-solid state disk two-level cache and a LFU eviction algorithm to accelerate image retrieval. The application solves the problems of unified storage of heterogeneous data in a laser accelerator, contradiction between high throughput writing and resource consumption, lack of data association, and low retrieval efficiency, and realizes efficient collaborative management and real-time visualization of multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of physical experiment data storage technology, specifically to a data storage method and system for laser accelerator control systems, which is particularly suitable for real-time storage and efficient retrieval of high-throughput, multimodal heterogeneous experimental data.

[0002] Background Introduction Laser accelerators utilize the extremely high electric fields (reaching the GV / m level, three orders of magnitude higher than traditional radio frequency accelerators) generated by the interaction of ultrashort, ultra-intense lasers with plasma. These fields can accelerate charged particles to relativistic energies within centimeter-scale distances, representing a revolutionary approach to overcoming the limitations of large-scale accelerators in terms of scale and economic cost. The particle beams produced by this technology possess unique advantages such as ultra-high peak current, picosecond-level pulse width, and extremely small source size, showing broad application prospects in cutting-edge fields such as cancer treatment (e.g., FLASH radiotherapy), fusion energy diagnostics, material irradiation modification, space environment simulation, and high-energy-density physics. However, as laser accelerators develop towards higher repetition rates and higher precision, the types of data generated by experiments are becoming increasingly complex, and the volume of data is growing dramatically. The experiments generated a wide variety of data, including high-throughput image data from scientific cameras, time-series monitoring data of process variables (such as pressure, flow, and temperature) from various detectors, and structured data such as system configuration and control parameters (including scientific camera images, time-series data of process variables, and system configuration parameters). This heterogeneous data placed stringent demands on the storage system: low-latency reception, high-throughput writing, long-term secure archiving, and efficient and flexible data retrieval capabilities.

[0003] Existing storage solutions are often optimized for single data types, making it difficult to simultaneously meet the storage needs of multiple data types. For example, while all-flash arrays offer high throughput, they are expensive and have poor scalability; mechanical disk arrays have large capacity but cannot support multi-way parallel real-time writes; general-purpose distributed storage systems are not specifically optimized for time-series and image data, resulting in low storage efficiency. Furthermore, existing systems generally lack a unified data classification and association mechanism, and the separate storage of metadata and raw data creates data silos, severely restricting subsequent data association analysis and AI-assisted physical process inversion.

[0004] Therefore, there is an urgent need for a comprehensive solution that can simultaneously meet the requirements for efficient storage and management of image data, time-series monitoring data, and configuration data in laser accelerator control systems. Summary of the Invention

[0005] This invention aims to address the technical challenges faced by existing laser accelerator control systems in handling the real-time storage and efficient retrieval of various heterogeneous data types, including high-throughput image data, time-series monitoring data, and structured configuration data. Specifically, it requires the unified storage and efficient collaborative management of three types of data: high-throughput images, high-frequency time-series data, and structured configuration data. This involves reducing storage resource consumption while ensuring high write performance to achieve long-term archiving of large volumes of data, establishing an association index between image data and time-series monitoring data to support rapid multi-dimensional data retrieval, and improving the access speed of massive image data to meet the dual needs of real-time monitoring and historical backtracking. This invention provides a hybrid storage method and system for laser accelerator control systems, employing a hierarchical hybrid storage architecture that combines a dedicated database with a distributed file system, and introducing a batch processing storage strategy and hierarchical caching mechanism based on HDF5.

[0006] The technical solution of the present invention is as follows: A hybrid storage system for a laser accelerator control system, comprising: Data acquisition layer: used to acquire image data generated by scientific cameras and time-series data of process variables generated by devices based on the EPICS architecture in real time.

[0007] The hybrid storage layer specifically includes: Relational databases are used to store structured configuration data such as device parameters, user permissions, and experiment configurations.

[0008] A time-series database is used to store the time-series data of the process variables and the metadata of the image data (including at least the acquisition time, device identifier, storage index, and file name).

[0009] Distributed object storage is used to store the image data in batch processing: a preset number of images are packaged into a scientific data storage file (such as an HDF5 file) in chronological order, compressed, and then stored in the distributed object storage.

[0010] The unified data indexing module is used to establish a mapping between the metadata of each image and the process variable data at the corresponding time in the time series database.

[0011] Data caching and retrieval layer: This layer includes a two-level caching module consisting of memory and solid-state drives (SSDs) to accelerate access to image data in the distributed object storage. Specifically, this layer includes: a cache index list (recording metadata and storage locations of cached files), a query module (first checking the cache; if a match is found, it is retrieved directly; otherwise, it is downloaded and cached), an eviction module (based on the LFU algorithm, eviction is performed according to access frequency and last access time), and a tiered storage module (storing the most frequently accessed files in the memory cache and caching the rest on the SSD).

[0012] Data visualization layer: Real-time data push and display based on the WebSocket protocol.

[0013] The process variable time series data is stored using a batch write mechanism: multiple process variable data are aggregated according to a time window and then written to the time series database in batches, and the latest value is written to the circular cache each time the data is updated; when the camera is triggered, the process variable data is matched from the circular cache according to the trigger time to achieve accurate alignment between the image and the process variable.

[0014] The present invention also provides a hybrid storage method based on the above system, comprising the following steps: Image acquisition and temporary storage steps: Image data is acquired in real time through the camera data processing module, preprocessed, and then pushed to the front end via WebSocket. At the same time, the original image is losslessly compressed and temporarily stored in the hard disk cache directory.

[0015] Image batch processing and archiving steps: When the number of accumulated images reaches a preset threshold, the temporarily stored batch of images are packaged into a scientific data storage file (such as an HDF5 file) in chronological order, compressed and stored in distributed object storage, and the metadata of each image (acquisition time, device identifier, index in the HDF5 file and file name) is stored in the time series database.

[0016] Process variable acquisition and synchronous caching steps: Process variable data is acquired in real time through the process variable data processing module. After cleaning and standardization, multiple process variable data are aggregated according to time windows and written into the time series database in batches. On the other hand, the latest value is synchronously written into the circular cache every time the data is updated.

[0017] Data association indexing steps: When the camera is triggered, based on the trigger time, process variable data within the preset time window before and after are extracted from the circular buffer, an association index with the current image is established, and the association relationship is stored in the time series database.

[0018] Tiered caching retrieval steps: When a user requests an image, the cache index list is first queried. If the cache is hit, the corresponding scientific data storage file is retrieved directly from the memory cache or SSD cache based on its storage location, and the target image is parsed. If the cache is not hit, the corresponding scientific data storage file is downloaded from distributed object storage, stored in the cache layer, and the cache index is updated. Cache eviction uses the LFU algorithm, prioritizing the eviction of files with the lowest access frequency and the earliest last access time. The top 20% of most frequently accessed files are stored in the memory cache, while the rest are cached on the SSD.

[0019] Visualization steps: The visualization interface displays images and process variable data streams in real time, and provides historical data retrieval and download functions based on multiple dimensions such as time and device.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. Unified storage of multimodal heterogeneous data: Through a hierarchical hybrid storage architecture, the most suitable storage engine is selected for configuration data, time series data and image data respectively, so as to realize fine-grained management of data classification and avoid data silos.

[0021] 2. High throughput and low latency writing: Image data adopts a batch processing HDF5 packaging strategy, merging multiple writes into one, which significantly reduces storage I / O pressure; PV data adopts a batch writing mechanism to reduce database connection overhead; the measured peak write performance can reach 42 GB / s, which meets the real-time storage requirements of parallel data from 42 cameras.

[0022] 3. Data Association and Efficient Retrieval: A time-series association between image metadata and PV data is established through a unified indexing mechanism, supporting fast retrieval by time, device, and other dimensions; a two-level cache of memory and solid-state drive is introduced, combined with the LFU eviction algorithm, so that the image retrieval latency is less than 40 ms when the cache is hit, which greatly improves the efficiency of accessing historical data.

[0023] 4. Resource consumption optimization: The image batch processing strategy eliminates the need for the server to perform compression and storage operations most of the time, effectively reducing CPU and memory usage; the tiered caching design places most of the cached data on solid-state drives, reducing reliance on expensive memory and lowering hardware costs.

[0024] 5. Real-time visualization and remote collaboration: Real-time data push based on WebSocket enables researchers to remotely monitor experimental status, supports multi-user collaborative analysis, and improves experimental efficiency and data utilization. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall architecture of a hybrid storage system for a laser accelerator control system provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the image data storage process in this invention.

[0027] Figure 3 This is a schematic diagram of the access process for laser accelerator monitoring variables (process variables) in this invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only, but the scope of protection of this invention is not limited thereto.

[0029] Please see Figure 1A hybrid storage system for a laser accelerator control system, comprising: Data acquisition layer: This includes the camera big data processing module and the system monitoring data processing module in the beam diagnostic system, which are responsible for acquiring real-time image data and measurement time-series data from the scientific camera and the equipment monitored by the laser accelerator control system, respectively.

[0030] Hybrid storage layer: It adopts a three-layer architecture, including high-speed cache, distributed parallel array and long-term archive storage; among them, configuration data is stored in relational database; time-series data monitored on laser accelerator is stored in time-series database; image data is packaged in HDF5 file format and stored in distributed object storage MinIO; and a unified data index module is set up to establish the association mapping between image metadata and monitoring data.

[0031] Data caching and retrieval layer: Includes a two-level caching module of memory and solid-state drive, and implements cache eviction based on LFU (Least Recently Used) algorithm to accelerate access to image data.

[0032] Data visualization layer: Based on the WebSocket protocol, it realizes real-time data push, and provides synchronous display of images and PV data, as well as historical data query and download functions.

[0033] The image data storage process is as follows: Figure 2 As shown, it specifically includes: The camera data processing module acquires raw images in real time and performs data cleaning, median filtering preprocessing, and ROI (region of interest) cropping. The preprocessed image data is pushed to the front end for display in real time via WebSocket; Meanwhile, the original image is losslessly compressed and temporarily stored in the specified hard disk cache directory; When the number of accumulated images reaches a preset threshold, the system packages a batch of images into an HDF5 file in chronological order, and then compresses the HDF5 file and writes it to the MinIO distributed object storage. The metadata of each image (including acquisition time, camera ID, parameter configuration, index in the HDF5 file, and HDF5 file name) is stored in the InfluxDB time series database and associated with the PV data at the corresponding time point.

[0034] The storage process of monitoring data in the control system is as follows: Figure 3 As shown, it specifically includes: The monitoring data processing module collects data released by the control system in real time, such as process variables (PV) released by Experimental Physics and Industrial Control Systems (EPICS), or variables released based on configuration software and OPC UA / OPC protocol, and performs data cleaning and standardized encapsulation. The pre-processed real-time monitoring data is compared with the configured threshold, and alarms are pushed in real time when the data exceeds the threshold. A batch write mechanism is adopted, which aggregates multiple monitoring data according to time windows and writes them in batches to the time series database (InfluxDB), thereby reducing the database I / O frequency. Each time monitoring data is updated, it is simultaneously written to a circular cache. When the camera is triggered, the monitoring data in the cache is matched according to the trigger time to achieve accurate alignment between the image and the monitoring data.

[0035] The image data caching and retrieval process includes: Build a cache index list to record the metadata of cached HDF5 files (filename, usage frequency, last access time, storage location); When a user requests an image, the system first queries the cached index list: If the target HDF5 file exists in the memory cache, extract the image data directly from memory; If the image exists in the SSD cache, read the HDF5 file from the SSD and extract the image. If a cache miss occurs, the corresponding HDF5 file is downloaded from the file database (MinIO) to the solid-state drive (or memory), and the cache index is updated. The cache eviction process uses the LFU algorithm, prioritizing the eviction of HDF5 files that have the lowest access frequency and the earliest last access time. The memory cache stores the top 20% of the most frequently accessed HDF5 files, while the remaining cached files are stored on the solid-state drive.

[0036] Example: This example uses the Laser Wake Field Accelerator (SIOM-LWFA) as the implementation platform, but the application of the present invention is not limited to this.

[0037] 1. System Deployment Environment The SIOM-LWFA facility is equipped with 42 scientific cameras for acquiring diagnostic images of plasma channels and electron beam spots. The accelerator control system, employing an EPICS architecture, monitors the operational status of various subsystems, including the beam diagnostic system and vacuum system, in real time. During the accelerator's trial run, the total peak data rate from all cameras exceeded 42 Gb / s. The system is deployed on a high-performance server cluster and managed using Kubernetes for containerized orchestration.

[0038] 2. Hybrid Storage Layer Implementation Database configuration: MySQL 8.0 is used to store structured data such as device parameters, user permissions, and experimental configurations.

[0039] Time-series database: Deploy a 3-node InfluxDB cluster to store PV monitoring data (such as flow meter readings, gas coefficients, etc.) and image metadata published by the EPICS architecture. Each measurement is grouped by device function (e.g., "Flow Meter Group" includes target flow, actual flow, etc. PV).

[0040] Object storage: Divided into multiple MinIO sub-clusters according to camera type, with 3 MinIO nodes corresponding to each type of camera, storing image data packets in HDF5 format.

[0041] 3. Batch processing and storage of image data During the experiment, the beam diagnostic system's camera generated 16-bit grayscale images averaging approximately 23.7 MB each at a frequency of 1 Hz. The system was configured to package 100 images (approximately every 100 seconds) into a single HDF5 file. The HDF5 file internally stored the dataset in chronological order of image acquisition time and recorded the index of each image. After packaging, the HDF5 file was compressed (e.g., using the LZ4 algorithm) and uploaded to the corresponding MinIO bucket. Simultaneously, the acquisition time, camera ID, HDF5 filename, and internal index of each image were stored in InfluxDB. Tests showed that the total time for packaging and storing 100 images did not exceed 30 seconds, meaning the system did not need to handle image compression and storage within 70 seconds, significantly reducing resource consumption.

[0042] 4. Monitoring data collection and correlation PV data is acquired synchronously upon camera trigger. The system maintains a circular cache to store all PV update records within the most recent second. When the camera triggers, PV data within 100 ms before and after the trigger time is extracted, correlated with the current image, and the correlation is stored in InfluxDB. PV data is written in batches, aggregated and written to the database every 10 seconds, resulting in a more than 5-fold improvement in write efficiency in actual tests.

[0043] 5. Image caching and retrieval The system is configured with 2 GB of memory cache and 10 GB of NVMe SSD cache (customizable size). Based on historical access statistics (typically, an HDF5 file contains 10-20 images, with a compressed size of around 100MB), the most frequently accessed 20% of HDF5 files (approximately 20 files) are kept resident in memory, while the rest are cached on the SSD. When a user queries an image at a specific moment via the web interface, the system first checks the cache index. If the image is found in memory, it returns a result in microseconds; if it's found on the SSD, it returns a result in milliseconds; if not, it retrieves the image from MinIO and caches it on the SSD. Test comparisons show that: Experimental results show that the image retrieval performance of this invention varies significantly under different caching strategies. Without any caching mechanism, the average time for retrieving a single image is approximately 620 ms. When a caching layer is introduced but a cache miss occurs, requiring data to be retrieved from the underlying MinIO object storage, the average time is still 620 ms (same as above). However, when a cache hit occurs, performance is significantly improved: using SSD as the caching medium, the average time can be reduced to 45 ms; if a memory cache is used, the average time is further reduced to 27 ms. These data fully verify that the multi-level caching mechanism adopted in this invention can effectively reduce image retrieval latency and improve system response speed.

[0044] 6. Visual Interface Implementation The front-end uses the Vue 3 framework, and the back-end uses Django Channels to establish a WebSocket persistent connection. The real-time monitoring page simultaneously displays the camera image stream (after median filtering and ROI cropping) and key PV curves, and supports searching historical images and PV data by time range. Alarm information is displayed in real-time pop-up notifications. Experimenters can remotely access the page via a browser to achieve cross-regional collaborative analysis.

[0045] 7. System Performance Summary The system performed stably in actual operation of SIOM-LWFA, fully meeting the real-time data storage requirements of 42 cameras. The image archiving latency was less than 500 ms, the average retrieval latency when the cache was hit was less than 40 ms, and the PV data write throughput reached 1 million points / second. It supports long-term data secure archiving and efficient reuse.

Claims

1. A hybrid storage system for a laser accelerator control system, characterized in that, include: The data acquisition layer is used to acquire image data generated by scientific cameras and time-series data of process variables generated by EPICS devices in real time. A hybrid storage layer includes a relational database, a time-series database, a distributed object storage, and a unified data indexing module. The relational database stores configuration data; the time-series database stores the time-series data of the process variables and the metadata of the image data; the distributed object storage stores the image data in batch processing, where a preset number of images are packaged chronologically into a scientific data storage file, compressed, and then stored in the distributed object storage; the unified data indexing module establishes a mapping between the metadata of each image and the corresponding time-series data of the process variables in the time-series database. The data caching and retrieval layer includes a two-level caching module consisting of memory and solid-state drive, used to accelerate access to image data in the distributed object storage; and The data visualization layer uses the WebSocket protocol to implement real-time data push and display. The process variable time series data is stored using a batch write mechanism: multiple process variable data are aggregated according to a time window and then written to the time series database in batches, and the latest value is written to the circular cache each time the data is updated; when the camera is triggered, the process variable data is matched from the circular cache according to the trigger time to achieve accurate alignment between the image and the process variable.

2. The hybrid storage system according to claim 1, characterized in that, The scientific data storage file is an HDF5 file, and the metadata of the image data includes at least the acquisition time, device identifier, index in the HDF5 file, and HDF5 file name.

3. The hybrid storage system according to claim 1, characterized in that, The storage of the process variable time series data adopts a batch writing mechanism: multiple process variable data are aggregated according to time windows and then written to the time series database in batches. The latest value is written to the circular buffer each time the data is updated. When the camera is triggered, the process variable data is matched from the circular buffer according to the trigger time to achieve accurate alignment between the image and the process variable.

4. The hybrid storage system according to claim 1, characterized in that, The data caching and retrieval layer also includes: A cache index list that records the metadata and storage location of cached scientific data storage files; The query module is used to first query the cached index list based on the user's request. If a match is found, the image data is directly extracted from the corresponding cache medium. Otherwise, the corresponding scientific data storage file is downloaded from the distributed object storage and the cache is updated. The eviction module, based on the LFU algorithm, performs cache eviction according to the access frequency and last access time of scientific data storage files; and The tiered storage module stores the most frequently accessed scientific data files in a memory cache, while the remaining cached files are stored on a solid-state drive.

5. The hybrid storage system according to claim 1, characterized in that, The data visualization layer transmits image data streams and process variable data in real time based on the WebSocket protocol, supports historical data query and download, and provides real-time push of alarm information.

6. A hybrid storage method based on the system according to any one of claims 1 to 5, characterized in that, Includes the following steps: Step 1: The camera data processing module acquires image data in real time, performs preprocessing, and pushes it to the front end via WebSocket. At the same time, the original image is losslessly compressed and temporarily stored in the hard disk cache directory. Step 2: When the number of accumulated images reaches the threshold, the temporarily stored batch of images are packaged into a scientific data storage file in chronological order, compressed and stored in distributed object storage, and the metadata of each image is stored in the time series database. Step 3: Collect process variable data in real time through the process variable data processing module, clean and standardize it, write it into the time series database in batches, and update the circular cache synchronously. Step 4: When the camera is triggered, match the process variable data from the circular buffer according to the trigger time and establish an association index with the current image; Step 5: When a user requests an image, query the cache index list. If the cache is hit, retrieve the image directly from the corresponding cache medium; otherwise, retrieve the corresponding scientific data storage file from the distributed object storage and update the cache. Step 6: Display images and process variable data in real time through a visual interface, and provide historical data retrieval and download.

7. The hybrid storage method according to claim 6, characterized in that, In step 4, matching process variable data from the circular cache based on the trigger time specifically involves: extracting process variable data within a preset time window before and after the trigger time, and associating it with the current image.

8. The hybrid storage method according to claim 6, characterized in that, The caching in step 5 adopts a two-level caching system of memory and solid-state drive. The cache index list records the metadata and storage location of the cached scientific data storage files. The cache eviction adopts the LFU algorithm, which prioritizes the eviction of scientific data storage files with the lowest access frequency and the earliest last access time. Furthermore, the scientific data storage files with the highest access frequency are stored in the memory cache, and the remaining cache files are stored on the solid-state drive.

9. The hybrid storage method according to claim 6, characterized in that, The HDF5 file described in step 2 organizes the image dataset in chronological order and records the dataset index for each image.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 6 to 9.

11. A laser accelerator control system, characterized in that, It includes the hybrid storage system according to any one of claims 1 to 5.

Images