Synchrotron radiation detector-oriented high-flux multi-mode real-time multi-stream data acquisition method

Abstract

The invention discloses a synchrotron radiation detector-oriented high-flux multi-mode real-time multi-stream data acquisition method, and belongs to the technical field of synchrotron radiation and computers. The method comprises the following steps: configuring a plurality of detector devices to generate data in an SWMR mode, and storing multi-dimensional data in real time by adopting an HDF5 format; distributing a write buffer area for each device, reading data in real time and monitoring the data volume of each buffer area; and when the data reaches a set threshold value, carrying out batch packaging and synchronous packaging on the multi-path data, and sending the multi-path data to a downstream processing system. According to the method, the problems of data loss and difficult synchronization caused by inconsistent data writing speeds of multiple detectors are effectively solved, the real-time performance, accuracy and system throughput of data acquisition are improved, and the method has good fault-tolerant capability and is suitable for a high-throughput multi-mode synchrotron radiation experiment scene.

Description

Technical Field

[0001] This invention belongs to the field of synchrotron radiation and computer technology, specifically relating to a high-throughput, multimodal, real-time, multi-stream data acquisition method for synchrotron radiation detectors. Background Technology

[0002] Large-scale scientific facilities such as high-energy synchrotron radiation sources, free-electron lasers, and spallation neutron sources have become indispensable research platforms in fields such as materials science, biomedicine, environmental science, physics, and chemistry. Synchrotron radiation sources, with their superior characteristics of high brightness, high energy, and high stability, can deeply analyze the microscopic structure and dynamic reaction processes of matter, greatly promoting the development of cutting-edge science. However, as the scale and complexity of these facilities continue to increase, the amount of experimental data is surging at a rate of hundreds to thousands of petabytes per year, growing by two to three orders of magnitude. Furthermore, the data modes are becoming increasingly diverse and the structures increasingly complex, posing unprecedented challenges to real-time data acquisition and processing technologies.

[0003] Traditional synchrotron radiation data acquisition systems are typically designed based on a single data stream, making it difficult to efficiently synchronize and integrate multi-dimensional data from different detectors. In actual operation, when multiple detectors operate in parallel, inconsistent data generation rates among the devices often lead to problems such as data acquisition delays, frame loss, and even data inconsistencies, severely restricting the feasibility of real-time data alignment and fusion. Furthermore, existing methods are difficult to deploy quickly to adapt to the needs of multi-stream data acquisition, requiring downstream data analysis and application stages to undertake a large amount of redundant preprocessing work. This not only increases system complexity and latency but also significantly reduces the overall processing efficiency and research effectiveness of high-throughput, multimodal experimental data.

[0004] Therefore, there is an urgent need for a new method that can acquire and synchronize multi-source detector data in real time, efficiently and reliably, in order to meet the severe challenges of the massive data scale, diverse modes and heterogeneous generation rates in current large scientific facilities. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a high-throughput, multimodal, real-time, multi-stream data acquisition method for synchrotron radiation detectors, thereby solving the problems of slow data acquisition speed and poor synchronization, and improving the efficiency and accuracy of synchrotron radiation source detectors in data acquisition and processing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for acquiring high-throughput, multimodal, real-time, multistream data for synchrotron radiation detectors, the method comprising:

[0008] Multiple detector devices can be connected as data stream sources;

[0009] Configure experimental parameters for the device and set the data generation mode to single write multiple read (SWMR) mode, and write multi-dimensional data to HDF5 file in real time during the data acquisition process;

[0010] Calculate the length of the data block to be acquired.

[0011] Monitor whether each HDF5 file has collected data blocks of length. If not, refresh the dataset and obtain new data (newdata) for files that have not been written to the buffer.

[0012] Write newdata to the corresponding buffer and update the actual length;

[0013] Once all file data has reached the length, update the read index and batch pack the data of length in each buffer.

[0014] The packaged data is sent to the downstream system.

[0015] Furthermore, the detector device includes a surface detector, a camera, a motor, and sensors.

[0016] Furthermore, the experimental parameters include frame rate and exposure time.

[0017] Furthermore, the HDF5 file supports storing data of any number of dimensions.

[0018] Furthermore, the calculation of the length of the data block to be acquired includes setting the default size of the data block to be acquired as length, and if it is insufficient, then length is set to the remaining size of the data.

[0019] Furthermore, the actual length reallen of the buffer is updated as follows: reallen = reallen + length.

[0020] Furthermore, before the data is packaged in batches, the data is aligned according to the timestamp or frame number recorded when each detector device collects data, and synchronous encapsulation and transmission are only performed when multiple data streams belong to the same time window or the same frame number.

[0021] Furthermore, the downstream system includes a real-time monitoring system, a data analysis unit, or a storage system.

[0022] In a second aspect, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for high-throughput multimodal real-time multistream data acquisition for synchrotron radiation detectors.

[0023] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned high-throughput multimodal real-time multistream data acquisition method for synchrotron radiation detectors.

[0024] The beneficial effects of this invention are as follows:

[0025] This invention effectively solves problems such as frame loss, data misalignment, and synchronization difficulties caused by inconsistent data write rates among multiple source devices in high-energy synchrotron radiation detection by combining the SWMR mechanism with HDF5 multi-dimensional real-time data storage. Utilizing independent buffer management and real-time monitoring of multiple data streams, automatic alignment and encapsulation of data with the same frame number are achieved, significantly improving the integration efficiency and accuracy of multimodal data streams. Through a data batch packaging and transmission mechanism, not only is system throughput and data transmission efficiency greatly improved, but the system's fault tolerance to network anomalies and other faults is also enhanced, avoiding local data loss. This method significantly improves the real-time performance of data acquisition, processing efficiency, and downstream analysis reliability in high-throughput multimodal experimental scenarios, providing efficient, stable, and scalable data support for large-scale synchrotron radiation experiments. Attached Figure Description

[0026] Figure 1 This is a flowchart of a high-throughput, multimodal, real-time, multistream data acquisition method for synchrotron radiation detectors according to the present invention.

[0027] Figure 2 This is a structural diagram of a high-throughput multimodal real-time multistream data acquisition method for synchrotron radiation detectors according to the present invention.

[0028] Figure 3 This is a schematic diagram of a high-throughput, multimodal, real-time, multi-stream data acquisition method for synchrotron radiation detectors according to the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] like Figure 1 As shown in the figure, this invention discloses a method for acquiring high-throughput, multi-modal, real-time, multi-stream data for high-energy synchrotron radiation detectors. This method includes the following steps:

[0031] S1 connects multiple devices as data stream inputs. These devices include various types of sensors or detectors, such as sensors, cameras, or other data acquisition devices, responsible for capturing experimental data in real time. Their data flow can be dynamically adjusted according to experimental needs, and relevant experimental parameters can be configured.

[0032] S2 configures experimental parameters for the connected devices. This includes setting specific conditions and parameters for data acquisition, such as frame rate and exposure time; setting the data generation mode to SWMR for easier real-time data acquisition; and storing data in the HDF5 file in real time during data acquisition. The method proposed in this invention allows for data of arbitrary multi-dimensionality, ensuring real-time acquisition of multimodal and multi-dimensional data.

[0033] S3 calculates the length of the data block to be acquired. The default value is the length; if it is insufficient, the length is set to the remaining size of the data.

[0034] S4 monitors multiple h5 files and determines whether all h5 files have collected data blocks of length `length`. If not all h5 files have collected data blocks of length `length`, then process each h5 file. For each h5 file, if newly generated data has not yet been written to the corresponding buffer, execute S5.

[0035] S5 refreshes the dataset corresponding to the current h5 page.

[0036] S6, retrieve the newly generated data newdata from h5, and return newdata;

[0037] S7. Place the newly generated newdata block into the buffer corresponding to h5. Then, increment the actual length of the buffer, reallen, by the size of length, where reallen = reallen + length.

[0038] S8, continue, repeat step S4 until multiple h5s have collected data of length. If the condition is met, then execute S9.

[0039] S9, the index position of the next data to be read is increased by the length.

[0040] S10 returns the data of length corresponding to each h5. The encapsulated data is then batch-packaged. This step aims to reduce the frequency of data transmission and improve efficiency.

[0041] S11 sends the packaged data to downstream systems that require it. These devices may include real-time monitoring systems, data analysis units, or storage systems.

[0042] S12, repeat steps S3 to S11 to continuously collect and process data in real time;

[0043] S13, determine if there is still data to be acquired. If so, continue with the above steps. If not, end the method flow until all data has been acquired.

[0044] like Figure 2 The diagram shows the structure of the execution device involved in the method of this invention. This invention relates to access devices, data processing, and downstream devices. Regarding access devices, this invention supports configuring experimental parameters and achieves real-time data acquisition through a SWMR mechanism. This allows multiple devices to work efficiently during independent write and read processes. Regarding data processing, this invention ensures data consistency by confirming and synchronizing data generated by different devices and recording the frame number of each device's data transmission and the unsynchronized frame number. Data with the same frame number is encapsulated to solve the data misalignment problem caused by inconsistent data acquisition rates among multiple devices. The encapsulated data is cached for a period of time for batch processing. This batch-packaged data is then distributed to downstream devices for subsequent processing, thereby improving the overall system throughput and data processing efficiency.

[0045] like Figure 3 As shown, this is a schematic diagram of the principle of the method of the present invention. Several detectors use the SWMR mechanism to collect h5 data in real time and store it in a buffer until multiple detectors have collected data of the same length. The data is then aligned and packaged for distribution to downstream devices.

[0046] Through the above steps, this embodiment can realize real-time data acquisition, block management, and batch transmission of multi-source devices, thereby improving the efficiency of data transmission and processing while ensuring data integrity.

[0047] In a second aspect, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for high-throughput multimodal real-time multistream data acquisition for synchrotron radiation detectors.

[0048] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned high-throughput multimodal real-time multistream data acquisition method for synchrotron radiation detectors.

[0049] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector, characterized in that, The method comprises: accessing multiple detector devices as data stream sources; configuring experimental parameters for the devices and setting a data generation mode to a single-write-multiple-read (SWMR) mode, and writing multi-dimensional data into HDF5 files in real time during data acquisition; calculating the length of a data block to be acquired; monitoring whether each HDF5 file has acquired a data block with the length, and if not, flushing a dataset and acquiring new data for a file not written into a buffer; writing the new data into the corresponding buffer and updating the actual length; after all files have acquired data with the length, updating a reading index, and batch-packaging data with the length in each buffer; sending the packaged data to a downstream system.

2. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, characterized in that, The detector devices comprise a face detector, a camera, a motor, and a sensor.

3. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, The experimental parameters comprise a frame rate and an exposure time.

4. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, The HDF5 file supports storage of any multi-dimensional data.

5. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, The calculation of the length of the data block to be acquired comprises, by default, setting the size of the data block to be acquired to the length, and if not, setting the length to the remaining size of the data.

6. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, The actual length of the buffer is updated by increasing the actual length of the buffer to the length.

7. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, Before batch-packaging the data, the data is aligned according to timestamps or frame numbers recorded when the detector devices acquire the data, and the data is synchronously encapsulated and sent only when the multiple data streams belong to the same time window or the same frame number.

8. The high-throughput multi-modal real-time multi-stream data acquisition method for a synchrotron radiation detector according to claim 1, wherein, The downstream system comprises a real-time monitoring system, a data analysis unit, or a storage system.

9. An electronic device, comprising: comprise: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the high-throughput multi-modal real-time multi-stream data acquisition method for synchrotron radiation detectors according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, A computer readable storage medium having stored thereon executable instructions that, when executed by a processor, enable the processor to implement the high-throughput multi-modal real-time multi-stream data acquisition method for synchrotron radiation detectors according to any one of claims 1-8.

Images