Synchrotron radiation experiment-oriented HDF5 data online visualization method and system

Abstract

The invention discloses a synchrotron radiation experiment-oriented HDF5 data online visualization method and a synchrotron radiation experiment-oriented HDF5 data online visualization system. According to the invention, a webpage system based on a front-end React, a rear-end Django and a database MongoDB is constructed, and the webpage system is deployed on a server through a Docker. The system firstly provides user login and authority management functions, and hierarchical access control can be realized according to the requirements of different experiment line stations; then, a user is supported to browse an HDF5 data file in a directory tree form at a webpage end, and structure information such as group, dataset and attrites in the file can be analyzed; according to the method, a user can check multi-dimensional data in a file at a webpage end, and can directly perform visual operation on image data, including interaction of zooming, overturning, display mode switching and the like, so that online rapid data understanding and analysis are realized, and the data processing and utilization efficiency of a synchrotron radiation experiment is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the fields of synchrotron radiation and computers, and specifically relates to an online visualization method and system for HDF5 data in synchrotron radiation experiments. Background Technology

[0002] Synchrotron radiation sources are scientific research platforms that generate high-brightness X-rays using accelerator devices. They possess advantages such as high energy resolution, high stability, and tunable wavelengths, and are widely used in fields such as physics, chemistry, materials science, biomedicine, environmental science, and geology. Through synchrotron radiation experiments, researchers can reveal the structure and dynamic processes of matter at the atomic and molecular levels, making it an indispensable tool in modern scientific research.

[0003] With the rapid development of large-scale scientific facilities such as next-generation synchrotron radiation sources, free-electron lasers, and spallation neutron sources, experimental conditions are becoming increasingly complex, acquisition rates are significantly increasing, and the resulting data volume is growing exponentially, reaching hundreds or even thousands of petabytes per year. This data is not only massive in quantity but also encompasses multi-dimensional and multi-modal information, placing higher demands on storage, management, and analysis. How to ensure data integrity while achieving rapid access and visualization has become a significant challenge in current scientific data management.

[0004] Currently, synchrotron radiation experimental data are commonly stored in the HDF5 file format. HDF5, as a hierarchical file format supporting large-scale scientific data, can conveniently store multi-dimensional and multi-type data content. However, in practical scientific applications, while HDF5 files are beneficial for data storage and sharing, they also present a high barrier to entry: researchers unfamiliar with the HDF5 format often find it difficult to quickly read and effectively utilize the data. Furthermore, most existing third-party software or systems can only read numerical data from HDF5 files. If researchers need to view and process image data, they must use additional software, making the experimental data viewing process cumbersome and inefficient, failing to meet the demands of high-throughput experiments for rapid data analysis and visualization. Summary of the Invention

[0005] The purpose of this invention is to propose an online visualization method and system for HDF5 data in synchrotron radiation experiments, addressing the problems of inconsistent management and difficulty in online viewing of HDF5 file data in existing technologies. Through this invention, researchers can browse and visualize HDF5 files on a web page without installing multiple software programs locally, thereby improving the efficiency of experimental data utilization and the convenience of the research workflow.

[0006] This invention constructs a web-based system using React as the front-end, Django as the back-end, and MongoDB as the database, deployed on a server via Docker. The system first provides user login and permission management functions, enabling hierarchical access control based on the needs of different experimental web sites. Subsequently, the system allows users to browse HDF5 data files in a directory tree format on the web interface, and can parse structural information such as groups, datasets, and attributes within the files. Users can not only view multi-dimensional data in the files on the web interface, but also directly perform visualization operations on image-based data, including zooming, flipping, and switching display modes, thereby achieving rapid online data understanding and analysis.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An online visualization method for HDF5 data in synchrotron radiation experiments, comprising the following steps:

[0009] 1) Deploy a web system on the server and accept user registration information. The web system includes a front-end React, a back-end Django, and a database MongoDB. The front-end React builds a directory tree for each web site based on the HDF5 file storage location of the web site and sets a button for each web site. The database MongoDB is used to store the scan records and HDF5 files of the web sites.

[0010] 2) The front-end React receives the user's login request and sends it to the back-end Django;

[0011] 3) The backend Django verifies the user based on the information in the login request. If the verification is successful, the user is allowed to access the website services in the web system.

[0012] 4) When a user needs to view the scanning records of a front-end station, they can click the button corresponding to that station through the web interface of the web system. The front-end React generates a request body and sends it to the back-end Django. The back-end Django retrieves the scanning records of that station from the database MongoDB based on the information in the request body and returns them to the front-end React for display.

[0013] 5) When a user selects an HDF5 file to view through the directory tree of a web application, the front-end React sends the directory path of the HDF5 file to the back-end Django. The back-end Django searches the MongoDB database to see if there is already a cached record of the directory path. If not, it performs a directory scan on the web application where the HDF5 file is located, obtains the HDF5 file, and writes it to the MongoDB database. Then, it returns the directory information of the HDF5 file on the web application to the front-end React, which updates the directory tree of the web application based on the directory information. When the user selects an HDF5 file in the directory tree to view, the front-end React sends the path in the node information of the selected HDF5 file to the back-end Django for verification, ensuring that the selected HDF5 file exists and the selected HDF5 file path is valid. If the verification is successful, the back-end Django opens the selected HDF5 file and returns it to the front-end React. The front-end React parses and renders the received data, displays the internal directory of the required HDF5 file, and loads and displays data for any selected dimension based on the user's selection in the internal directory.

[0014] Preferably, in retrieval mode, the information in the request body includes station name (station_name), data name (data_name), scan number (scan_ID), username (user_name), email address, start date (start_time), end date (end_time), and beamtime number (beamtime_ID); in initialization mode, the information in the request body includes station name (station_name) and table name (table_name).

[0015] Preferably, the request header of the request body includes Content-Type: application / json, X-CSRFToken (cross-site request forgery token), and Authorization information.

[0016] Preferably, the directory information includes the HDF5 file and its directory path, name, size, and type.

[0017] Preferably, the method for verifying that the selected HDF5 file is opened by the backend Django and returned to the frontend React is as follows: If the selected HDF5 file is a dataset, the multidimensional data is sliced ​​according to the selection condition requested by the frontend React, and the data is converted according to the user-specified data type dtype to obtain an array; then the array data is converted into binary or Base64 format and returned to the frontend React; if the selected HDF5 file is a group or a link, the backend Django returns its metadata information, including a list of child nodes and type information, encoded in JSON format so that the frontend React can display the tree structure; if the frontend React requests the attributes of the dataset / group of the selected HDF5 file, the backend Django processes the raw attribute values ​​into a serializable form and packages them into a JSON dictionary and returns it to the frontend React; if the frontend React requests the statistical information of the selected HDF5 file, the backend Django calculates the statistics of the dataset and encodes the statistical results into JSON and returns them to the frontend React.

[0018] Preferably, the backend Django uses h5py.File to open the selected HDF5 file.

[0019] Preferably, the backend Django returns the retrieved requested data to the frontend React in JSON format for display.

[0020] Preferably, the front-end React sends the request body to the back-end Django using the POST method.

[0021] An online visualization system for HDF5 data in synchrotron radiation experiments is characterized by comprising a front-end React, a back-end Django, and a database MongoDB. The front-end React constructs a directory tree for each beamline based on the HDF5 file storage location and provides a button for each beamline. The MongoDB database stores the scan records and HDF5 file paths of the beamlines.

[0022] The front-end React is used to receive the user's login request and send it to the back-end Django;

[0023] The backend Django is used to verify the user based on the information in the login request. If the verification is successful, the user is allowed to access the website services in the web system.

[0024] When a user needs to view the scanning records of a front-end station, they can click the button corresponding to that station. The front-end React generates a request body and sends it to the back-end Django. The back-end Django retrieves the scanning records of that station from the database MongoDB based on the information in the request body and returns them to the front-end React for display.

[0025] When a user selects an HDF5 file to view through the directory tree of a web application, the front-end React sends the directory path of the HDF5 file to the back-end Django. The back-end Django searches the MongoDB database to see if there is a cached record of this directory path. If not, it scans the web application containing the HDF5 file, retrieves the HDF5 file, and writes it to the MongoDB database. Then, it returns the directory information of the HDF5 file to the front-end React, which updates the directory tree of the web application based on this information. When the user selects an HDF5 file from the directory tree to view, the front-end React sends the path in the node information of the selected HDF5 file to the back-end Django for verification, ensuring that the selected HDF5 file exists and the selected HDF5 file path is valid. If the verification is successful, the back-end Django opens the selected HDF5 file and returns it to the front-end React. The front-end React parses and renders the received data, displays the internal directory of the required HDF5 file, and loads and displays data for any selected dimension based on the user's selection in the internal directory.

[0026] The effects of this invention are as follows:

[0027] Compared with existing HDF5 data processing methods that rely on multiple local software tools, this invention enables unified management and visualization on a web page, significantly lowering the barrier to data viewing for researchers. The system supports multi-user hierarchical access, ensuring that different researchers can quickly obtain data that matches their permissions. At the same time, the online image visualization function provided by this invention can replace the cumbersome operation of relying on multiple software programs, enabling researchers to efficiently understand and analyze experimental data, and significantly improving the data processing and utilization efficiency of synchrotron radiation experiments. Attached Figure Description

[0028] Figure 1 This is a flowchart of the online visualization method for HDF5 data in synchrotron radiation experiments according to the present invention.

[0029] Figure 2 This is a structural diagram of the HDF5 data online visualization system for synchrotron radiation experiments according to the present invention.

[0030] Figure 3This is a data flow diagram of HDF5 in this invention. Detailed Implementation

[0031] like Figure 1 As shown in the figure, this invention discloses an online visualization method and system for HDF5 data in synchrotron radiation experiments, comprising the following steps:

[0032] S1. Users access the system website via PC terminal and log in using their email address and password; users without an account need to fill in the information to complete the registration first.

[0033] In S2, the React frontend sends a login request to the Django backend. The backend generates a JWT (JSON Web Token) using a specified encryption method with a secret key, expiration time, and the correct username and password. JWT is an open standard based on JSON used for secure information transmission in network communication, commonly used for authentication and authorization. The JWT is returned to the frontend along with a successful verification message from the backend. The frontend stores the JWT in the browser, and subsequent business operations will include the JWT as a token in the network request header for quick authentication in all subsequent network requests, verifying the user's authenticity. This token has an expiration time; after the verification period expires, a new login is required.

[0034] In step S3, the backend Django extracts the email and password from the login request and validates them in the database. If the validation passes, the user successfully logs in and can access the website. If the validation fails, a prompt message is returned. Steps S1 to S3 can be repeated.

[0035] S41. When a user needs to view the scanning records of a line station, they can click the corresponding line station button on the web page.

[0036] S42, the frontend React sends a queryConditions request body to the backend Django via POST. In retrieval mode, the request body includes the station name (station_name), data name (data_name), scan ID (scan_ID), username (user_name), email address, start date (start_time), end date (end_time), and beamtime ID (a beamtime ID may have multiple scan_IDs). In initialization mode, it only includes the station name and table name (table_name). The request header includes the content type (Content-Type: application / json), a cross-site request forgery token (X-CSRFToken), and user authentication information (Authorization). The content type specifies that subsequent frontend and backend data transmission will use JSON format. The X-CSRFToken ensures that each network request originates from a legitimate user and has not been tampered with. The user authentication information stores the encrypted JWT token for the user's current session. Based on these conditions, the backend retrieves data from MongoDB and returns it to the frontend in JSON format for display.

[0037] In step S51, when a user needs to view an HDF5 file on a web site, the front-end React sends the current directory path of the selected HDF5 file to the back-end. The back-end searches MongoDB for a cached record of this directory. If not, it rescans the directory of the web site containing the HDF5 file, writes the corresponding directory structure to the database, and updates the directory tree to obtain the correct HDF5 file path. If a record exists, it first scans the current directory path and compares it with the cached directory in the database to determine if the directory has changed and updates it if necessary. The back-end returns the processed directory information in a standardized JSON format, including attributes such as the file or directory path, name, size, and type. The front-end receives this information and calls the directory update function to insert the results into the corresponding nodes, thus generating an interactive directory tree level by level. This method avoids transmitting all directory information at once when dealing with large amounts of HDF5 data, reducing resource consumption and ensuring that users always receive the latest directory structure. The directory tree shown by the front-end filters out files with other file extensions, and users can click on the selected HDF5 file in a subdirectory to view it.

[0038] In S52, after the React frontend receives a user click, it sends the path (i.e., the complete path of the selected HDF5 file) from the node information of the selected HDF5 file to the Django backend. Upon receiving the HDF5 file path and node information from the frontend, the backend first verifies the file path's validity, ensuring the file exists and the user-requested HDF5 internal path is valid. Then, the backend opens the HDF5 file using `h5py.File` and creates a content object for the HDF5 file path using `create_content(f,path,resolve_links=...)`. Based on the passed HDF5 file path, it parses the nodes (which could be datasets, groups, or links) under that HDF5 file path and generates a data description structure usable by the frontend. The backend then processes the nodes according to their type:

[0039] If the node is a dataset, the multidimensional data is sliced ​​according to the selection criteria requested by the frontend. Optional operations include flattening (converting a multidimensional array to a one-dimensional array) and converting the data type (dtype) as specified by the user to obtain the final array. Subsequently, the array data is converted to binary or Base64 format using the specified encoding type (encode(...,encoding=...)) to reduce network transmission. If the node is a group or link, the backend returns its metadata information, including a list of child nodes and their types, encoded in JSON format for the frontend to display the tree structure. If the frontend requests attributes from the selected HDF5 file's dataset / group, the backend processes the native attribute values ​​into serializable form: converting NumPy scalars to Python native types, decoding bytes data into UTF-8 strings, and then packaging them into a JSON dictionary for return. If the frontend requests statistics from the selected HDF5 file, the backend calculates basic statistics such as the dataset's minimum (min), maximum (max), mean (mean), and standard deviation (std), and encodes the results into JSON for return.

[0040] The entire processing flow ensures efficient data transmission of the dataset. Metadata, attributes, and statuses can be directly displayed or plotted on the front end. The unified encoding method makes the interface scalable and easy for the front end to parse.

[0041] S53: After receiving the data returned by the backend, the frontend parses and renders it, displays the internal directory structure of the HDF5 file, and loads and displays data for any selected dimension according to the user's selection in the internal directory.

[0042] like Figure 2 As shown, this invention achieves unified management and online visualization of HDF5 data from synchrotron radiation experiments by constructing a web-based system with a React frontend, Django backend, and MongoDB database. This system not only provides a secure and efficient user login and access control mechanism, but also intuitively displays HDF5 files through a directory tree structure and supports online parsing and interactive image viewing, including zooming, flipping, and display mode switching. Compared to traditional methods relying on multiple local software programs, this invention significantly lowers the barrier to data access and use, improves researchers' efficiency in acquiring and analyzing experimental data, and has good practicality and promotional value.

[0043] Example

[0044] This embodiment discloses an online visualization method and system for HDF5 data in synchrotron radiation experiments. The system is built on a front-end React, a back-end Django, and a MongoDB database, and deployed on a server using Docker. In practical applications, such as at a synchrotron radiation source experimental station, researcher A accesses the system website via a PC terminal and follows these steps:

[0045] S1, as Figure 3 As shown, User A enters their email address abc@ihep.ac.cn and password on the web login page. If it is the first time using the service, they need to fill in basic information to complete the registration.

[0046] In S2, after the front-end React receives user input, it sends the login request to the back-end Django. The back-end verifies the request and generates the corresponding JWT, which is used for fast identity verification in subsequent network requests.

[0047] In S3, Django extracts the account information from the request and compares it with the registered user data in MongoDB. If the information is correct, it returns a successful login status and allows access to system functions; if the verification fails, it prompts the user to re-enter the information.

[0048] S41. After successfully logging in, user A selects an experimental beamline on the webpage and clicks the Task Logs button for Beamline B1.

[0049] In step S42, React sends a query request to Django. The backend parses the `station_name` parameter in the request, retrieves the experimental scan records from the corresponding table in MongoDB, and serializes the results into JSON format before returning them to the frontend. The frontend then parses the data and displays it in a table format, allowing users to quickly browse the experimental records.

[0050] S51, User A further selects the H5 Data File for Beamline B1 on the webpage, and the front-end displays the corresponding directory tree for that beamline site. The directory tree expands dynamically using lazy loading, enabling real-time display of the data directory and reducing front-end resource consumption. The user clicks on one of the HDF5 files with the extension ".h5" or ".nxs".

[0051] In S52, React sends the HDF5 file path to Django. The backend parses the path and verifies the validity and existence of the file. Then, based on the user's request, it parses the dataset, group, attributes, stats, and other information inside the file and returns it to the frontend in a standardized data format.

[0052] S53: After receiving the parsed results, the front-end displays the internal directory structure of the HDF5 file on a web page. Users can expand the directory layer by layer in the interface and select a specific dataset to view the data. If the dataset is image data, the web page directly renders it as a visual image, supporting operations such as zooming in, zooming out, flipping, and switching display modes. Users can quickly browse and analyze the data without downloading files or relying on additional software.

[0053] This embodiment enables users to log in and access experimental data, query scan records, parse HDF5 files, and perform online image visualization operations via a web browser. This method not only reduces the burden on researchers regarding local environment configuration and software dependencies but also improves the efficiency of data access and processing, significantly enhancing the user experience of synchrotron radiation experimental data.

[0054] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An online visualization method for HDF5 data in synchrotron radiation experiments, comprising the following steps: 1) Deploy a web system on the server and accept user registration information. The web system includes a front-end React, a back-end Django, and a database MongoDB. The front-end React builds a directory tree for each web site based on the HDF5 file storage location of the web site and sets a button for each web site. The database MongoDB is used to store the scan records and HDF5 files of the web sites. 2) The front-end React receives the user's login request and sends it to the back-end Django; 3) The backend Django verifies the user based on the information in the login request. If the verification is successful, the user is allowed to access the website services in the web system. 4) When a user needs to view the scanning records of a front-end station, they can click the button corresponding to that station through the web interface of the web system. The front-end React generates a request body and sends it to the back-end Django. The back-end Django retrieves the scanning records of that station from the database MongoDB based on the information in the request body and returns them to the front-end React for display. 5) When a user selects an HDF5 file to view through the directory tree of a web application, the front-end React sends the directory path of the HDF5 file to the back-end Django. The back-end Django searches the MongoDB database to see if there is already a cached record of the directory path. If not, it performs a directory scan on the web application where the HDF5 file is located, obtains the HDF5 file, and writes it to the MongoDB database. Then, it returns the directory information of the HDF5 file on the web application to the front-end React, which updates the directory tree of the web application based on the directory information. When the user selects an HDF5 file in the directory tree to view, the front-end React sends the path in the node information of the selected HDF5 file to the back-end Django for verification, ensuring that the selected HDF5 file exists and the selected HDF5 file path is valid. If the verification is successful, the back-end Django opens the selected HDF5 file and returns it to the front-end React. The front-end React parses and renders the received data, displays the internal directory of the required HDF5 file, and loads and displays data for any selected dimension based on the user's selection in the internal directory.

2. The method according to claim 1, characterized in that, In retrieval mode, the information in the request body includes station name (station_name), data name (data_name), scan number (scan_ID), username (user_name), email address, start date (start_time), end date (end_time), and beamtime number (beamtime_ID); in initialization mode, the information in the request body includes station name (station_name) and table name (table_name).

3. The method according to claim 2, characterized in that, The request header of the request body includes the content type Content-Type: application / json, the cross-site request forgery token X-CSRFToken, and the user authentication information Authorization.

4. The method according to claim 1, 2, or 3, characterized in that, The directory information includes the HDF5 file and its directory path, name, size, and type.

5. The method according to claim 1, 2, or 3, characterized in that, The method for verifying that the selected HDF5 file is opened by the backend Django and returned to the frontend React is as follows: If the selected HDF5 file is a dataset, the multidimensional data is sliced ​​according to the selection criteria requested by the frontend React, and the data is converted according to the user-specified data type (dtype) to obtain an array; then the array data is converted to binary or Base64 format and returned to the frontend React; if the selected HDF5 file is a group or a link, the backend Django returns its metadata information, including a list of child nodes and type information, encoded in JSON format so that the frontend React can display the tree structure; if the frontend React requests the attributes of the dataset / group in the selected HDF5 file, the backend Django processes the raw attribute values ​​into a serializable form and packages them into a JSON dictionary and returns it to the frontend React; if the frontend React requests the statistical information of the selected HDF5 file, the backend Django calculates the statistics of the dataset and encodes the statistical results into JSON and returns them to the frontend React.

6. The method according to claim 1, characterized in that, The backend Django uses h5py.File to open the selected HDF5 file.

7. The method according to claim 1, characterized in that, The backend Django will return the retrieved requested data to the frontend React in JSON format for display.

8. The method according to claim 1, characterized in that, The front-end React sends the request body to the back-end Django via POST.

9. An online visualization system for HDF5 data in synchrotron radiation experiments, characterized in that, This includes a front-end React, a back-end Django, and a database MongoDB. The front-end React builds a directory tree for each web site based on the HDF5 file storage location and provides a button for each site. The database MongoDB stores the scan records and HDF5 file paths of the web sites. The front-end React is used to receive the user's login request and send it to the back-end Django; The backend Django is used to verify the user based on the information in the login request. If the verification is successful, the user is allowed to access the website services in the web system. When a user needs to view the scanning records of a front-end station, they can click the button corresponding to that station. The front-end React generates a request body and sends it to the back-end Django. The back-end Django retrieves the scanning records of that station from the database MongoDB based on the information in the request body and returns them to the front-end React for display. When a user selects an HDF5 file to view through the directory tree of a web application, the front-end React sends the directory path of the HDF5 file to the back-end Django. The back-end Django searches the MongoDB database to see if there is a cached record of this directory path. If not, it scans the web application containing the HDF5 file, retrieves the HDF5 file, and writes it to the MongoDB database. Then, it returns the directory information of the HDF5 file to the front-end React, which updates the directory tree of the web application based on this information. When the user selects an HDF5 file from the directory tree to view, the front-end React sends the path in the node information of the selected HDF5 file to the back-end Django for verification, ensuring that the selected HDF5 file exists and the selected HDF5 file path is valid. If the verification is successful, the back-end Django opens the selected HDF5 file and returns it to the front-end React. The front-end React parses and renders the received data, displays the internal directory of the required HDF5 file, and loads and displays data for any selected dimension based on the user's selection in the internal directory.

Images