Single event test monitoring system based on virtual cluster

Abstract

The invention discloses a single-particle test monitoring system based on a virtual cluster, solves the problem that a traditional monitoring system is difficult to meet the requirements of different experimental environments, and belongs to the field of particle physical experiments. The method comprises the following steps: a virtual cluster comprises a management node and a plurality of working nodes, and each node is a Docker container; each component is packaged in a Docker container to operate; the data acquisition engine assembly communicates with an accelerator through an EPICS system, obtains process variable data in a single event test of the accelerator in real time, and stores the process variable data in the time sequence database assembly. The data acquisition engine assembly is also used for connecting the time sequence database assembly and the visual tool assembly; the time sequence database component stores process variable data in the single event test of the accelerator in real time, and the data carries a timestamp; and the visualization tool assembly displays the process variable data in the form of a chart and an instrument board. The method is suitable for the single-particle test system of the proton heavy ion accelerator.

Description

Technical Field

[0001] The present invention relates to a single-particle experiment monitoring system based on a virtual cluster, belonging to the field of particle physics experiments. Background Art

[0002] The 300 MeV proton heavy ion accelerator adopts a combined scheme of an ECR ion source, a linear accelerator, and a synchrotron. It is currently a device that can simultaneously provide proton and heavy ion beams, enabling a large range of adjustment of the ion beam energy and the irradiation area. The platform has successfully debugged proton, He, O, C, Kr, Ta, Bi, and U beams. The accelerator is equipped with an ECR ion source for generating high-charge-state ion beams such as protons and heavy ions. The low-energy ion beam generated by the ion source is accelerated to an energy level suitable for entering the synchrotron ring through a low-energy injection line. The synchrotron ring is the core component of the accelerator, consisting of a radio frequency quadrupole acceleration cavity (RFQ) and an intermediate acceleration cavity (IH-DTL), etc., responsible for accelerating the ion beam to the designed energy of 300 MeV. The entire acceleration process ensures that the particle beam can reach a high energy level, thereby providing the required high-energy particle source for subsequent experiments.

[0003] The main function of the 300 MeV proton heavy ion accelerator is to simulate the irradiation effects of high-energy particles in space on spacecraft, materials, devices, and living organisms. Through the high-energy proton and heavy ion beams generated by the accelerator, it can simulate the common particle radiation in the space environment, providing an experimental platform for the design of spacecraft materials, the radiation resistance test of spacecraft components, and biological research. As an important device for particle physics experiments, the operating state of the particle accelerator is crucial for the accuracy of the experiment. Currently, the monitoring of single-particle experiments mostly relies on traditional hardware control systems and manual monitoring methods, which cannot effectively achieve real-time data acquisition, monitoring, and display, and there are problems such as data acquisition delay and cumbersome operation. In addition, traditional monitoring systems are often unable to be flexibly expanded and are difficult to meet the requirements of different experimental environments. Summary of the Invention

[0004] Aiming at the problem that the traditional monitoring system is difficult to meet the requirements of different experimental environments, the present invention provides a single-particle experiment monitoring system based on a virtual cluster.

[0005] A single-particle experiment monitoring system based on a virtual cluster of the present invention is applicable to the single-particle experiment system of a proton heavy ion accelerator and includes:

[0006] The virtual cluster includes 1 management node and multiple working nodes, and each node is a Docker container running on a physical server or a virtual machine;

[0007] The management node is used to coordinate the task allocation and scheduling within the cluster;

[0008] Each worker node receives tasks from the management node and runs the received tasks;

[0009] The monitoring system includes a time series database component, a data acquisition engine component, and a visualization tool component. Each component runs encapsulated in a Docker container;

[0010] The data acquisition engine component is used to communicate with the accelerator control system through the EPICS system, obtain process variable data in the accelerator single-particle experiment in real time, and store it in the time series database component; it is also used to connect the time series database component and the visualization tool component, and send the data obtained from the time series database component to the visualization tool component;

[0011] The time series database component is used to store process variable data in the accelerator single-particle experiment in real time, and this data has a timestamp;

[0012] The visualization tool component is used to display the obtained data in the form of charts and dashboards.

[0013] Preferably, the time series database component is implemented using the time series database InfluxDB.

[0014] Preferably, the visualization tool component is implemented using the visualization platform Grafana.

[0015] Preferably, the EPICS system in the data acquisition engine component includes multiple nodes, and each device on the node has one or more process variables in the accelerator single-particle experiment. The data acquisition engine component uses the PyEPICS library to interact with the EPICS system and obtain process variable data from each node in the EPICS system in real time.

[0016] Preferably, the data acquisition engine component adopts asynchronous programming and multithreaded processing.

[0017] Preferably, the data acquisition engine component also includes a two-level cache design, including a static dictionary PVINDEX and a dynamic list PV CACHE LIST;

[0018] The static dictionary PV INDEX is used to map readable channel names and unique integer identifiers;

[0019] The dynamic list PV CACHE LIST is a dynamic list, including dynamic dictionaries with the same number of channels as the channels. Each dynamic dictionary stores the current value of a channel.

[0020] Advantages of the present invention: By improving the efficiency of data acquisition, storage, and processing, the present invention reduces the operation and maintenance costs of the system, improves the accuracy and reliability of data, and has strong scalability, effectively supporting the high-frequency data acquisition and real-time monitoring requirements in single-particle experiments of accelerators. Specifically, it is manifested as follows:

[0021] Through Docker containerized deployment, the present invention can efficiently utilize computing resources and simplify the system deployment, maintenance, and expansion processes. Compared with traditional virtualized environments, Docker containers have lower resource occupancy and higher deployment efficiency, significantly reducing hardware and operation costs. Real-time data acquisition and automatic storage reduce manual operations, avoid data loss and errors, improve work efficiency, and save labor and time costs. The present invention has good scalability and can meet the requirements of different experimental environments. Through Docker cluster management technology, the system can be flexibly expanded to improve the stability and processing capacity of the system. As the data volume increases, it is convenient to enhance data processing and storage capabilities by expanding containers and cluster nodes without large-scale modification of the existing architecture.

[0022] The present invention adopts asynchronous programming and multi-threaded processing technologies, significantly improving the performance of data acquisition. In traditional single-threaded or synchronous data acquisition methods, bottlenecks may occur when processing a large amount of concurrent data. Through concurrent execution and multi-threaded task management, the present invention can maximize the utilization of system resources while real-time collecting and processing data. The engine can operate efficiently in a high-frequency data acquisition environment, minimizing latency and increasing data processing speed. Compared with traditional single-threaded or inefficient caching systems, the performance of the present invention is improved by approximately 50% (the specific value can be further adjusted according to actual tests), ensuring the stability of the system in the processing of a large amount of concurrent data.

[0023] The present invention uses InfluxDB as the data storage system, which is designed specifically for time-series data, can efficiently store the real-time data of particle accelerators, and supports fast query and analysis. Compared with traditional relational database systems, InfluxDB has more efficient support for time-series data, can handle several times the data volume of traditional systems, and has a shorter query response time. Especially when storing data for a long time period, the performance advantage is more obvious. Through a caching mechanism (including a multi-level caching system), the present invention significantly improves the data reading speed and storage efficiency, reducing cache contention and latency while ensuring high-concurrency access.

[0024] The present invention realizes efficient communication with the EPICS system through the PyEPICS library, enabling accurate data acquisition. Compared with the traditional manual data acquisition method, it reduces the errors caused by manual operations and can accurately monitor the changes in process variable (PV) data during real-time data acquisition. Through the automated data acquisition and cache synchronization mechanism, the present invention can respond to device status changes in real time, ensuring the integrity and accuracy of experimental data. During the experiment, the acquisition accuracy and real-time performance of the data have been greatly improved.

[0025] The data acquisition engine of the present invention is written in Python. By integrating the PyEPICS library and InfluxDB, it realizes a simple and easy-to-use interface and functions. Compared with the traditional manual monitoring and complex data acquisition systems, the operation is more convenient, and the complexity of configuration and deployment can be reduced. The present invention supports users to customize the number of threads and cache policies, and can be flexibly configured to meet the data acquisition requirements in different experimental environments, facilitating adjustment according to experimental needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the virtualization cluster architecture of the present invention;

[0027] Figure 2 It is the working node in the virtualization cluster of the present invention;

[0028] Figure 3 It is the system composition of the present invention;

[0029] Figure 4 Single-particle test monitoring system interface;

[0030] Figure 5 It is a multi-level cache structure.

[0031] Figure 6 It is the EPICS data acquisition engine component architecture. DETAILED DESCRIPTION OF THE INVENTION

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0034] Next, the present invention will be further described in conjunction with the accompanying drawings and specific embodiments, but it is not a limitation of the present invention.

[0035] This implementation mode is achieved by using Docker virtualization cluster technology. Docker virtualization cluster is a system architecture implemented through containerization technology, which can run and manage containers on multiple physical or virtual hosts. Each container contains all the dependent environments required to run a certain application, enabling the application to run consistently in any environment. Docker virtualization cluster automates the deployment, management, scaling, and monitoring of containers through container orchestration tools (such as Docker Swarm or Kubernetes). This clustered management mode enables the system to dynamically increase or decrease the number of containers according to demand, providing high availability, fault tolerance, and scalability. Its functions include container isolation, resource optimization, high availability, and flexibility, ensuring that each container application can run efficiently between different nodes and achieve load balancing.

[0036] A Docker cluster is composed of multiple nodes, which can be physical servers or virtual machines. Docker Engine runs on each node, responsible for creating and managing containers. The nodes of a Docker cluster are divided into two categories: management nodes and worker nodes.

[0037] Management nodes: Management nodes are responsible for the management and control of the cluster. Their main function is to coordinate the task allocation and scheduling within the cluster. Management nodes manage services and tasks in the cluster through the Docker Swarm mode, responsible for controlling and monitoring the status of worker nodes, as well as receiving operation requests from users. Management nodes will convert these operations into tasks and allocate them to suitable worker nodes for execution. Usually, the number of management nodes is odd to ensure that the election mechanism in the cluster can operate correctly and avoid the split-brain problem.

[0038] Worker nodes: Worker nodes are the nodes that execute actual tasks. Worker nodes receive tasks from management nodes and start containers locally to run these tasks. Each task usually represents a service container instance, and worker nodes ensure that containers run according to the predefined configuration. Worker nodes do not directly handle cluster management tasks but focus on executing the workload.

[0039] Tasks: Tasks are the smallest scheduling units in a Docker cluster. Each task is a container instance running on a worker node, executing the application defined by the service. Tasks are assigned by management nodes to worker nodes, and they are responsible for executing specific workloads. The running of each task is monitored and managed by the management nodes of the Docker cluster. Tasks usually contain the actual application programs or containers of the service and ensure that they run as expected. Services are composed of multiple tasks, representing the running status of multiple container instances of an application in the cluster. This hierarchical management and scheduling method enables the Docker cluster to run efficiently and reliably in a distributed environment.

[0040] The single-particle test monitoring system based on a virtual cluster in this embodiment is applicable to the single-particle test system of a proton heavy ion accelerator, and includes:

[0041] The virtual cluster includes 1 management node and multiple worker nodes, and each node is a Docker container running in a physical server or a virtual machine;

[0042] The management node is used to coordinate the task allocation and scheduling within the cluster;

[0043] Each worker node receives tasks from the management node and runs the received tasks;

[0044] The monitoring system includes a time series database component, a data acquisition engine component, and a visualization tool component, and each component is encapsulated and run in a Docker container;

[0045] The data acquisition engine component is used to communicate with the accelerator control system through the EPICS system, obtain the process variable data in the accelerator single-particle test in real time, and store it in the time series database component; it is also used to connect the time series database component and the visualization tool component, and send the data obtained from the time series database component to the visualization tool component;

[0046] The time series database component is used to store the process variable data in the accelerator single-particle test in real time, and this data has a timestamp;

[0047] The visualization tool component is used to display the obtained data in the form of charts and dashboards.

[0048] At the same time, the visualization tool component sends the data to the central monitoring system, and the accelerator control system sends the control data to the central monitoring system.

[0049] In the monitoring system of a particle accelerator, the application of a Docker virtualized cluster has significant advantages.

[0050] First of all, Docker container technology enables each component of the accelerator monitoring system to be independently deployed and effectively managed. Each component is encapsulated in a container and can be distributed and run on multiple hosts, which provides the system with efficient resource utilization and flexible scalability. When the accelerator monitoring needs to support more data traffic or more complex calculations, the Docker cluster can dynamically expand the number of containers to cope with load changes and ensure the efficient operation of the system.

[0051] The Docker virtualization cluster can optimize the storage and processing of real-time data. The monitoring system of the accelerator involves a large number of real-time data collection and storage tasks. Docker containers can run time series database components to process the operation data of the accelerator. These containerized databases can flexibly adjust resources as needed to ensure that the system can still efficiently store and query data when the data volume increases, while avoiding waste of hardware resources. The visualization tool components can also be containerized and integrated into the Docker cluster for management. Through containerized deployment, the visualization tool components can be connected to the time series database components in real time to display the visualization data panel of the accelerator operation status. Administrators and experimenters can intuitively monitor the working status of the accelerator through the graphical interface and make timely adjustments and decisions. Through the Docker cluster, the deployment and management of the visualization tool components become more efficient and flexible. The virtualization cluster also simplifies the operation and maintenance work of the monitoring system. Due to the independence and convenient deployment method of Docker containers, administrators can update and upgrade each component without interrupting the system operation. In addition, the Docker cluster provides a unified management and monitoring platform, making the operation and maintenance of the system more efficient and centralized. Through the container monitoring tools in the cluster, administrators can view the health status of the containers in real time to ensure that the monitoring system is always in the best operating state.

[0052] The application of the Docker virtualization cluster in the particle accelerator monitoring not only provides flexible and efficient system management and expansion capabilities, but also ensures the high availability, fault tolerance and data storage capabilities of the system. Through this technology, the monitoring system of the particle accelerator can operate more stably and efficiently, providing reliable support for experimenters.

[0053] The data collection engine and its dependencies (such as InfluxDB, Grafana, etc.) can all be deployed through Docker containerization, which is convenient for system expansion and management. Docker containers can be deployed on multiple physical servers or virtual machines, and container orchestration and load balancing can be achieved through Docker Swarm or Kubernetes. Docker containerized deployment enables the system to have good scalability and high availability, ensuring that data collection during the experiment is not interrupted. The containerization solution of this embodiment can be replaced by a virtual machine cluster or a physical server cluster, especially in the case of no Docker environment. However, the advantage of containerized deployment lies in its high efficiency, portability and easy expansion characteristics, which can better meet the high-concurrency data collection requirements in accelerator experiments.

[0054] In a single-particle experiment, the system can collect experimental data in real time and accurately, avoiding the errors and delays in manual data collection. The system architecture is designed flexibly and can expand the data collection scope and storage capacity according to requirements, ensuring the efficient storage and management of large-scale experimental data.

[0055] In the preferred embodiment, the time series database component is implemented using the time series database InfluxDB. The specific implementation methods include:

[0056] InfluxDB is an open-source time series database suitable for high-performance time series data, such as monitoring data, sensor data, event data, etc. InfluxDB can efficiently store and query data associated with timestamps. Its main features include:

[0057] Optimization of time series data: Specifically designed for the storage and query of time series data, providing efficient data compression and indexing mechanisms.

[0058] Efficient query and writing: InuxDB can efficiently write and query a large amount of data, making it suitable for storing continuously changing monitoring data.

[0059] Easy to expand: Supports horizontal expansion and can improve performance and reliability through methods such as sharding and replication.

[0060] Rich query functions: Supports the SQL-like query language InfluxQL, and users can filter and statistically analyze data through complex query statements.

[0061] Aggregation and continuous query: Supports data aggregation operations, such as summation, average value, maximum value, minimum value, etc., which is suitable for analyzing and processing experimental data.

[0062] In the single-particle experiment of the accelerator in this embodiment, a large amount of real-time data needs to be collected to monitor the operating status of the accelerator, and these data need to be stored, processed, and analyzed. At this time, the combination of InfluxDB and Docker virtual containers provides a powerful solution, and the specific application is as follows:

[0063] (1) Real-time data storage

[0064] In single-particle experiments of accelerators, the accuracy and reliability of the experiments highly depend on the real-time monitoring of the accelerator's operating status. Accelerators are highly complex devices involving a large number of physical quantities and technical parameters, and these data are usually continuously collected through various sensors and instruments. These data include key physical parameters such as temperature, pressure, current, and particle beam intensity. At the same time, they also carry timestamps, reflecting the time series characteristics of the data. Due to the dynamic nature of these data, an efficient system is needed to store, query, and analyze these data to help experimenters make timely adjustments and optimizations.

[0065] Traditional databases, such as relational databases, although suitable for storing static data, are inefficient and underperform when faced with large amounts of high-frequency, time series data. Especially in scenarios like accelerator experiments where real-time data requirements are extremely high, traditional databases are difficult to meet the needs. To solve this problem, InfluxDB, a database designed specifically for time series data, can efficiently process these data with timestamps and frequent updates.

[0066] InfluxDB's design concept is centered around time. Through technologies such as optimized timestamp indexing and efficient data compression algorithms, the data writing and querying speeds are extremely fast. At the same time, InfluxDB supports complex queries based on time ranges, enabling experimenters to quickly obtain data within a specified time period. This is particularly important for accelerator experiments because the amount of data in the experiments is extremely large and constantly changing. How to efficiently store these data and be able to quickly and accurately retrieve historical states is the key to the success of the experiments.

[0067] By storing the operating data of the accelerator, especially the process variables (PV), in InfluxDB, experimenters can real-time track and monitor every operating status of the accelerator. Since the working environment of the accelerator is extremely sensitive to parameters such as particle beam intensity, temperature, and pressure, any small change may affect the experimental results or even lead to system failures. Therefore, it is particularly important to monitor the status at each moment in real time. Through InfluxDB, experimenters can not only view the current operating status in real time but also conveniently retrieve historical data to understand the system performance at different experimental stages and under different conditions. This provides valuable data support for fault troubleshooting, performance optimization, and scientific experiments.

[0068] InfluxDB can also help experimenters deeply analyze the data of accelerator operation. For example, by querying the change trend of particle beam intensity, temperature fluctuations, etc., and combining with experimental conditions, potential laws or problems can be found, and then the operation parameters of the accelerator can be adjusted or the experimental methods can be optimized. This analysis based on historical data not only improves the accuracy and efficiency of the experiment, but also helps to ensure the long-term stable operation of the accelerator and reduce the probability of failures.

[0069] In summary, the application of InfluxDB in the single-particle experiment of the accelerator provides an efficient and reliable solution. It can efficiently store and manage the real-time data of the accelerator, support accurate time-based queries, help experimenters better grasp the operation status of the accelerator, discover potential problems, and improve the accuracy and efficiency of the experiment. By combining with other monitoring systems (such as EPICS system, Grafana visualization tool, etc.), InfluxDB further enhances the comprehensive monitoring ability of the system and provides strong technical support for accelerator experiments.

[0070] (2) Docker containerized deployment

[0071] In an accelerator monitoring system, multiple components are usually required to achieve data collection, data storage, analysis, and visualization. Docker containerized deployment of InfluxDB is a fast, flexible, and efficient way to manage time series data, especially in application scenarios such as accelerators or other applications that require high-frequency data collection and storage. Deploying InfluxDB using Docker containers can bring advantages such as easy management, fast deployment, strong isolation, and high scalability. The following are the detailed steps on how to deploy InfluxDB through Docker containerization:

[0072] 1) Install Docker;

[0073] 2) The official Docker image of InfluxDB can be obtained from Docker Hub. Run commands in the terminal to pull the latest InfluxDB image;

[0074] 3) After pulling the image, start the InfluxDB container so that the InfluxDB service can be accessed externally.

[0075] 4) Configure InfluxDB, and you can also adjust the storage location, modify the listening port, set authentication, etc.

[0076] 5) Use the running detection command to check if the InfluxDB container has been successfully started and is running. The running detection command will list all running containers. If the InfluxDB container is running, it should be visible in the list. Additionally, you can also verify if InfluxDB is working properly by accessing port 8086 on the host machine. If you see the welcome page of InfluxDB in the browser, it means the InfluxDB container has been successfully run.

[0077] 6) Connect to InfluxDB via the CLI: InfluxDB provides a command-line tool influx, through which interactive queries can be made. To enter the InfluxDB CLI, run the InfluxDB connection command. After entering, databases can be created, data inserted, or queries performed via the CLI. For example, create a database named testdb;

[0078] 7) Use InfluxDB to store and query data: Once the InfluxDB container is running, time series data can be written to and queried from the database via a client or API. In experimental applications, experimental data (such as temperature, particle beam intensity, etc.) can be written to InfluxDB in real time through tools like Python scripts and Grafana. For example, use the InfluxDB client library influxdb-client in Python to write data, and then write the data in the Python code, which will write the temperature data to the temperature table in the testdb database.

[0079] 8) Management and maintenance of Docker containers: After deploying InfluxDB in a containerized manner, management and maintenance become more convenient. Containers can be managed through common Docker commands, such as:

[0080] Stop the container: docker stop influxdb

[0081] Start the container: docker start influxdb

[0082] View the container logs: docker logs influxdb

[0083] Delete the container: docker rminfluxdb (make sure the container has been stopped)

[0084] 9) Scaling and high availability: For scenarios that need to process a large amount of data or require high availability, multiple InfluxDB instances can be formed into a cluster through tools like Docker Swarm or Kubernetes for scaling, providing high availability and data redundancy.

[0085] Deploying InfluxDB with Docker containerization can provide an efficient, flexible, and easy-to-manage data storage solution for applications such as single-event effects experiments on accelerators. Through Docker containerization, the installation, configuration, and maintenance of InfluxDB become simpler, and it can also be quickly scaled to meet the increasing data storage requirements.

[0086] In this embodiment, different storage solutions can be selected according to experimental requirements. For example, a distributed storage system (such as Apache Kafka or Elasticsearch) can be used to replace InfluxDB to cope with larger-scale data storage requirements. Also, according to the data write frequency and query requirements, the sharding and indexing functions of the database can be used to improve the performance and query efficiency of the storage system.

[0087] In the preferred embodiment, the visualization tool component of this embodiment is implemented using the visualization platform Grafana. Specifically, it includes:

[0088] (1) The principle and structure of Grafana

[0089] Grafana is an open-source visualization platform that can obtain data from different data sources and display it in visual ways such as charts and dashboards. These data sources can include time series databases such as InfluxDB, Prometheus, and even relational databases such as MySQL, PostgreSQL. The flexibility of Grafana enables it to adapt to different application scenarios. The architecture of Grafana and its core components are as follows:

[0090] The architecture of Grafana can be divided into several main core components, specifically including:

[0091] Front-end interface: Users access the Web user interface provided by Grafana through a browser for data visualization and dashboard management. The front-end interface is the main entry point for users to interact with Grafana, providing functions such as intuitive dashboard display, data query, and alarm configuration. The front-end of Grafana is built using modern Web technologies (such as React and Angular) to ensure a good user experience and response speed. Through the front-end, users can create and modify dashboards, charts, queries, alarm rules, etc.

[0092] API Layer: Grafana provides a RESTful API layer for the front-end to interact with the back-end and third-party applications. The API layer serves as a bridge between the front-end and back-end services, providing interfaces for data query, dashboard management, user permission management, etc. Through the API layer, users can automate some tasks, such as creating dashboards, managing users, obtaining data, etc., and can even integrate with external systems.

[0093] Graph Rendering Engine: The graph rendering engine is responsible for converting the data obtained from the data source into a graphical view, supporting various chart types, such as line charts, bar charts, pie charts, Gantt charts, etc. It draws charts that meet the requirements based on the user's requests and data, helping users intuitively understand the data. For example, when a user queries certain time-series data, the graph rendering engine will display this data in the form of a line graph or a bar graph.

[0094] Data Source: Grafana supports connecting to multiple data sources, such as Prometheus, InfluxDB, Elasticsearch, MySQL, PostgreSQL, CloudWatch, etc. Each data source is extended in the form of a plugin, and Grafana provides corresponding plugins according to different database types. Users can add, configure, and manage data sources as needed. By connecting to different data sources, Grafana can obtain data and perform visual display. Different data sources also provide different query languages. For example, InfluxDB uses InfluxQL or Flux, and Prometheus uses PromQL.

[0095] Back-end Service: The back-end service is responsible for handling the business logic and data storage of Grafana. It includes functions such as user authentication, authorization, snapshot management, configuration management, etc. The back-end service is also responsible for managing the permissions of users and dashboards, ensuring that users can only access the data and dashboards they are authorized to. At the same time, it also handles functions such as data caching and logging to ensure the stability and performance of the system.

[0096] Data Flow Process:

[0097] User Request: The user accesses the Grafana front-end interface through a browser and submits a data query request.

[0098] API Layer Processes the Request: The front-end sends the request to the API layer of Grafana through the RESTful API, and the API layer receives the request and parses it.

[0099] Data source plugin retrieves data: The API layer calls the corresponding data source plugin to send requests to the corresponding database or service to retrieve data. For example, sending a PromQL query request to Prometheus or an InfluxQL query request to InfluxDB.

[0100] Data return and processing: The data source plugin returns the query results, and the API layer processes the results, such as formatting the data, filtering the data, performing necessary aggregation operations, etc.

[0101] Front-end graphic rendering: The processed data is returned to the front end, and the front end uses a graphic rendering engine to present the data as the charts or visualization components specified by the user, such as line charts, tables, gauges, etc.

[0102] The architecture design of Grafana is known for its flexibility and scalability. It enables users to retrieve data from multiple data sources and visually display it in the form of charts and dashboards through the collaborative work of components such as the front end, API layer, graphic rendering engine, data source, and back-end service. Grafana not only supports multiple data sources but also has powerful query and visualization capabilities, making it a powerful tool for real-time data monitoring and analysis. Through Grafana, users can create custom dashboards, perform real-time data monitoring, set alarm rules, and quickly respond to problems in the system.

[0103] (2) Grafana virtualized cluster deployment

[0104] As an open-source monitoring and visualization tool, Grafana is usually used together with other monitoring tools (such as Prometheus, InfluxDB) to monitor and analyze the performance data of the system. Deploying Grafana in a Docker virtualized cluster can greatly enhance its management and scalability. Through Docker Swarm or Kubernetes, Grafana can run efficiently on multiple nodes, ensuring high availability, load balancing, and fault recovery. Grafana can be deployed through Docker containerization and managed in a Docker virtualized cluster.

[0105] 1) Docker environment preparation: First, ensure that Docker is installed and configured on each node of the cluster. In addition, if you plan to use Docker Swarm or Kubernetes cluster management tools to manage Grafana, you can install Docker Swarm or Kubernetes as needed. Here, Docker Swarm is used as an example for explanation, and the deployment method of Kubernetes is similar.

[0106] 2) Set up the Docker Swarm cluster: On the master node, execute the following command to initialize the Docker Swarm cluster: To add worker nodes, run the command on other worker nodes to join them to the Docker Swarm cluster: You can obtain the join token via docker swarm join-token worker.

[0107] 3) Grafana Docker deployment: On each node, use the following command to pull the official Grafana image; In the Docker Swarm cluster, Grafana will be managed as a service. You can deploy the Grafana service in Docker Swarm via the following command.

[0108] 4) Configure the Grafana data source: After deploying Grafana in a containerized manner, you need to configure a data source (such as Prometheus or InfluxDB) to obtain data.

[0109] 5) Grafana dashboard management: Create dashboards and add different types of panels (such as line charts, bar charts, gauges, etc.) to the dashboards to display monitoring data graphically.

[0110] 6) Grafana high-availability configuration: By creating multiple Grafana replicas in Docker Swarm, load balancing and high availability can be achieved. If a node or container fails, the Grafana containers on other nodes will continue to provide services. This is a core advantage of the Docker Swarm cluster. You can view the number of replicas and the running status of the Grafana service and update the service, such as increasing the number of replicas or modifying environment variables.

[0111] 7) Monitoring and log management of Grafana containers: Through Docker Swarm and Grafana, the entire cluster can be monitored. By viewing the service logs, problems can be discovered and solved in a timely manner; By deploying the Grafana cluster via Docker Swarm, high availability and fault tolerance can be achieved, ensuring the stable operation of the Grafana service on multiple nodes. The containerized deployment of Grafana simplifies the installation and maintenance process, and at the same time, data is persistently stored via Docker volumes to avoid data loss problems caused by container restart. Data sources can be flexibly added and configured, custom dashboards can be created, and real-time monitoring and alarm functions can be implemented to meet different business requirements.

[0112] (3) Real-time monitoring of Grafana single-particle experiment information

[0113] In single-particle experiments, the monitoring system is crucial as it helps experimenters monitor the operating status of the accelerator in real time and ensures the smooth progress of the experiment. Grafana, as a powerful data visualization and monitoring tool, is very suitable for displaying and analyzing various physical parameters during the operation of the accelerator. By integrating Grafana into the single-particle experiment monitoring system of the accelerator, experimenters can view the operating status of the accelerator in real time, obtain important parameters (such as particle beam intensity, current, temperature, etc.), and conduct effective analysis and troubleshooting.

[0114] In the single-particle experiment of the accelerator, the experiment mainly involves the control and testing of high-energy particle beams. Parameter monitoring during the experiment needs to be very precise and real-time. For example, particle beam intensity, the temperature of the accelerator cavity, vacuum degree, current, etc. all directly affect the success of the experiment. In order to be able to respond quickly to these changes, the monitoring system must have the following characteristics:

[0115] Real-time data acquisition: The operating status of the accelerator needs to be monitored in real time through sensors and instruments.

[0116] Data storage and visualization: The real-time data collected needs to be stored and visually displayed in the form of charts.

[0117] Alarm function: When certain physical parameters during the experiment exceed the set thresholds, it can trigger an alarm in a timely manner to notify the experimenters.

[0118] Data analysis and trend prediction: Through the analysis of real-time data, potential problems can be predicted and adjustments can be made to optimize the experimental process.

[0119] Grafana can be integrated with multiple data sources (such as Prometheus, InfluxDB, Elasticsearch, etc.) to visually display the data obtained from the accelerator monitoring system (such as EPICS). Through Grafana, experimenters can easily create custom dashboards, view key data in real time, and make decisions based on this data.

[0120] In the single-particle experiment of the accelerator, EPICS (Experimental Physics and Industrial Control System) may be used to obtain the process variable (PV) data of the accelerator. Grafana can integrate these data sources through plugins or APIs, connect the experimental data with Grafana, and display it in real time.

[0121] EPICS data source: Through a custom EPICS data scraping engine, process variables (such as particle beam intensity, current, temperature, etc.) are obtained from the accelerator in real time and stored in InfluxDB or other time series databases.

[0122] InfluxDB Data Source: InfluxDB is used to store time - series data collected from accelerator sensors. Grafana fetches data from InfluxDB and displays it to users using charts.

[0123] Prometheus Data Source: If Prometheus is used to monitor other aspects of the accelerator system (such as CPU usage and memory occupancy of computer hardware), Grafana can also be integrated with Prometheus to display real - time monitoring data.

[0124] Grafana allows users to customize dashboards and display different monitoring data in an intuitive graphical way. In single - particle experiments, you can create a dashboard with multiple panels to display the following:

[0125] Particle Beam Intensity Chart: Displays the change in the intensity of the particle beam over time, which can help experimenters determine whether the particle beam is stable and whether accelerator parameters need to be adjusted.

[0126] Temperature and Current Monitoring Panels: Temperature, vacuum, current, etc. are key indicators for the normal operation of the accelerator. Through Grafana, you can view the temperature and current data of accelerator devices in real - time to ensure that the devices operate within a safe range.

[0127] Real - Time Monitoring of Process Variables (PV): Displays the changes in different process variables. For example, monitor the temperature of each cavity of the accelerator, the power of the particle beam, the current of the accelerator, etc., to ensure that these key parameters meet the experimental requirements.

[0128] Alarm Panel: When certain parameters exceed the set thresholds, Grafana will trigger an alarm and display the alarm information on the dashboard to remind experimenters to make timely adjustments.

[0129] Through the aggregation function of Grafana, experimenters can analyze experimental data within different time ranges and discover potential trends and anomalies. For example, by summarizing and trend - analyzing the changes in particle beam intensity, current, and temperature, experimenters can predict the risk of equipment failure and make adjustments in advance.

[0130] The application of Grafana in the monitoring of single - particle experiments of accelerators provides a powerful data visualization platform for experimenters. It can monitor the operating status of the accelerator in real - time, analyze the change trends of key parameters, and trigger alarms in a timely manner when abnormal situations occur. By integrating with data sources (such as EPICS, InfluxDB, Prometheus), Grafana provides a flexible and efficient monitoring system to help experimenters make timely decisions and ensure the accuracy of the experimental process and the stable operation of the accelerator.

[0131] In a preferred embodiment, the EPICS system in the data acquisition engine component of this embodiment includes multiple nodes, and each device on the node has one or more process variables in the accelerator single-particle experiment. The data acquisition engine component uses the PyEPICS library to interact with the EPICS system and obtains process variable data from each node in the EPICS system in real time. The specific implementation process of the data acquisition engine component of this embodiment is as follows:

[0132] In a single-particle experiment, the data acquisition and monitoring system is crucial. The core of the system is a Python-based data acquisition engine that communicates with accelerator devices through the EPICS (Experimental Physics and Industrial Control System) protocol to obtain process variable (PV) data of the devices in real time. These data include physical parameters such as particle beam intensity, temperature, current, and vacuum degree of the accelerator, which are crucial for the smooth progress of the experiment.

[0133] 1) Workflow of the data acquisition engine: The data acquisition engine uses the PyEPICS library to interact with the EPICS ChannelAccess (CA) protocol. PyEPICS provides a simple and efficient way to help read and write data from the EPICS system. The engine establishes monitoring channels and subscribes to data changes in these channels. Once the data changes, it obtains the updated value through the callback mechanism and processes it. To improve efficiency, the engine adopts asynchronous programming and multithreading techniques, which can achieve efficient concurrent processing and avoid performance bottlenecks when processing a large amount of data.

[0134] The workflow of the data acquisition engine includes the following steps:

[0135] Data acquisition: Through the PyEPICS library, the data acquisition engine obtains process variable (PV) data from the EPICS system in real time. These data are usually the key operating parameters of the accelerator and have real-time and timing characteristics.

[0136] Data caching: When each acquired data value changes, the engine first stores the data in the local cache for quick access. To optimize performance, the engine uses a multi-layer cache structure to ensure the timely access and efficient processing of data.

[0137] Data storage: The cached data is periodically or on-demand written to a time-series database such as InfluxDB for long-term storage. InfluxDB is a database specifically designed to handle time-series data, which can efficiently store a large amount of experimental data and support fast query and analysis.

[0138] Data Visualization and Analysis: The data stored in InfluxDB can be monitored in real time and analyzed through visualization tools such as Grafana. Grafana provides rich chart and dashboard functions, which can intuitively display the operating status of the particle accelerator, helping scientific researchers to detect problems in a timely manner and adjust experimental conditions.

[0139] The engine of this embodiment adopts a two-level cache design, including a static dictionary PV INDEX and a dynamic list PVCACHE LIST; the static dictionary PV INDEX is a static and read-only dictionary used to map readable channel names and unique integer identifiers; the dynamic list PV CACHE LIST is a dynamic list, and each dictionary stores the current value of a specific channel. This two-level cache ensures data consistency through a synchronization mechanism, and at the same time optimizes the performance of concurrent processing through multithreading and asynchronous programming. By flexibly configuring the number of worker threads, users can adjust the processing power of the engine according to the load and requirements to cope with different workloads. This embodiment can use the asyncio module of Python to improve the ability of concurrent data acquisition.

[0140] The Python-based EPICS data acquisition engine not only effectively solves the problems of real-time and high efficiency of data acquisition, but also ensures the reliability and availability of accelerator experiment data through a flexible cache mechanism and a high-performance storage solution. This system greatly improves the management efficiency of experimental data and provides strong support for subsequent analysis and decision-making.

[0141] (1) Methods for creating a Python EPICS data engine:

[0142] 1) Install the required Python libraries

[0143] 2) Connect to the EPICS server: The EPICS system usually consists of multiple nodes, and each device on the node can have one or more process variables (PVs). Through pyepics, you can easily connect to the EPICS server in Python and obtain the values of process variables.

[0144] 3) Real-time monitoring and data acquisition: If you need to continuously obtain the data of PVs within a time window, it can be achieved by continuously polling the PVs;

[0145] 4) Store the obtained data in InfluxDB for analysis and visualization as time series data, which can be achieved through the influxdb-client library

[0146] 5) Once the data is stored in InfluxDB, it can be visualized through Grafana. Grafana can directly connect to the InfluxDB database, query the stored data, and display it on the dashboard. For example, you can create a Grafana dashboard containing charts and real-time data panels to help experimenters intuitively see the operating status of the accelerator.

[0147] By using the EPICS data engine written in Python, we can collect PV data from the accelerator in real time and store it in InfluxDB. Combined with the Grafana visualization tool, it is further possible to monitor and analyze the single-particle test data of the accelerator. This approach not only helps experimenters better understand the operating status of the accelerator but also triggers an alarm when an anomaly occurs to ensure the smooth progress of the experiment.

[0148] The data acquisition engine component of this embodiment is responsible for obtaining real-time data of process variables from the EPICS server. Using the interfaces provided by the PyEPICS library, this module implements the connection to the EPICS system and data acquisition. Different libraries can be selected to replace PyEPICS, such as python-epics or epics, which provide similar functions but have differences in implementation.

[0149] The caching system of this embodiment can choose different implementation methods according to needs. For example, using a distributed cache (such as Redis) instead of an in-memory cache can improve the efficiency of data access, especially in high-concurrency scenarios. To reduce cache contention, a lock mechanism (such as Python's threading.Lock) can also be used to ensure cache consistency. The cache size and synchronization strategy can be adjusted according to actual needs.

[0150] The EPICS data acquisition engine solution provided by the present invention meets the requirements for real-time data processing in particle accelerator experiments through efficient data acquisition, caching, storage, and visualization capabilities. Through flexible cache design and multiple storage schemes, the system structure and performance can be adjusted according to different experimental requirements. Through Docker containerized deployment, the scalability and high availability of the system are ensured. The alternative solutions and optimization methods for each part further enhance the flexibility and adaptability of the system, enabling it to better support the needs in different experimental scenarios.

[0151] While the present invention has been described herein with reference to particular embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. Accordingly, it should be understood that numerous modifications may be made to the exemplary embodiments, and other arrangements may be devised, without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that different dependent claims and the features described herein may be combined in ways different from those described in the original claims. It should also be understood that the features described in connection with individual embodiments may be used in other described embodiments.

Claims

1. A single-particle test monitoring system based on a virtual cluster, applicable to the single-particle test system of a proton and heavy ion accelerator, characterized in that, Including: The virtual cluster includes 1 management node and multiple worker nodes, and each node is a Docker container running in a physical server or a virtual machine; The management node is used to coordinate task allocation and scheduling within the cluster; Each worker node receives tasks from the management node and runs the received tasks; The monitoring system includes a time series database component, a data acquisition engine component, and a visualization tool component, and each component runs encapsulated in a Docker container; The data acquisition engine component is used to communicate with the accelerator control system through the EPICS system, obtain process variable data in the accelerator single-particle test in real time, and store it in the time series database component; it is also used to connect the time series database component and the visualization tool component, and send the data obtained from the time series database component to the visualization tool component; The time series database component is used to store the process variable data in the accelerator single-particle test in real time, and this data has a timestamp; The visualization tool component is used to display the obtained data in the form of charts and dashboards.

2. The single-particle test monitoring system based on a virtual cluster according to claim 1, characterized in that The time series database component is implemented using the time series database InfluxDB.

3. The single-particle test monitoring system based on a virtual cluster according to claim 1, wherein The visualization tool component is implemented using the visualization platform Grafana.

4. The single-particle test monitoring system based on a virtual cluster according to claim 1, characterized in that The EPICS system in the data acquisition engine component includes multiple nodes, and each device on the node has one or more process variables in the accelerator single-particle test. The data acquisition engine component uses the PyEPICS library to interact with the EPICS system and obtain process variable data from each node in the EPICS system in real time.

5. The single-particle test monitoring system based on a virtual cluster according to claim 1, characterized in that The data acquisition engine component adopts asynchronous programming and multithreaded processing.

6. The single-particle test monitoring system based on a virtual cluster according to claim 1, characterized in that The data acquisition engine component also includes a two-level cache design, including a static dictionary PV INDEX and a dynamic list PV CACHE LIST; The static dictionary PV INDEX is used to map readable channel names and unique integer identifiers; The dynamic list PV CACHE LIST is a dynamic list, including dynamic dictionaries with the same number of channels as the channels, and each dynamic dictionary stores the current value of a channel.

Images