Method and system for automatic reading of advanced data information of solar observations into a database
By generating advanced data through the observer and using the database table fits_dir for automated comparison and updating, the real-time and integrity issues of solar observation data reading are resolved, and efficient data management and access are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZIJINSHAN ASTRONOMICAL OBSERVATORY CHINESE ACAD OF SCI
- Filing Date
- 2026-01-29
- Publication Date
- 2026-06-19
Smart Images
Figure CN122240629A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of solar physics technology, specifically relating to a method and system for automatically reading advanced solar observation data into a database. Background Technology
[0002] Advanced solar observation data, derived from processed raw observational data, is crucial and widely used in key areas such as solar physics research. This data is typically stored in hierarchical directories, archived in compressed formats, and requires continuous updates; therefore, it demands high real-time performance and completeness from data retrieval.
[0003] Existing systems or methods typically read information directly into databases, failing to achieve automatic, real-time, and complete reading of advanced solar observation data. They also have shortcomings in areas such as compression and parsing, tool adaptation, and batch processing, severely limiting data utilization efficiency. Therefore, there is an urgent need for a technical solution that can address these issues, enabling efficient and accurate reading of advanced solar observation data and fully realizing its application value. Summary of the Invention
[0004] This invention addresses the shortcomings of existing technologies by providing a method and system for automatically reading advanced solar observation data into a database.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] In a first aspect, the present invention provides a method for automatically reading advanced data information from solar observations into a database, comprising the following steps: The sun is observed using an instrument, and raw data is collected. The raw data is processed to generate advanced data for users to download and analyze. The advanced data storage directory is categorized by time, and different scanning frequencies are set for different categories; Starting from the current day's directory, scan each day according to the set scanning frequency; compare the scan results with the records in the database table fits_dir, which is used to store directory and file information for advanced data; if there are differences, update the records in the database table fits_dir and re-enter the file information in the directory; if the database table fits_dir does not contain the directory information, create the directory and enter the file information in the directory.
[0007] Optionally, the observers include four observers: hxi, fmg, sdi, and wst.
[0008] Optionally, the advanced data is further subdivided into multiple subcategories based on detector type and processing level.
[0009] Optionally, the advanced data storage directory is divided into two categories based on time: directories within three days and directories older than three days.
[0010] Optionally, setting different scanning frequencies for different categories specifically involves: For directories within the last three days, set the scanning frequency to once every half hour; For directories older than three days, set the scan frequency to once a day, starting at 0:00.
[0011] Optionally, comparing the scan results with records in the database table fits_dir specifically involves: Based on the directory's update time, size, and number of files, compare it with the records in the `fits_dir` database table. If there are differences, update the records in the `fits_dir` database table, set `need_handle_flag=y`, and re-enter the file information in the directory. If the `fits_dir` database table does not contain the directory information, create the directory, set `need_handle_flag=y`, and enter the file information in the directory.
[0012] Optionally, the process for storing and updating the file information is as follows: A thread is started at set intervals to retrieve records with need_handle_flag=y from the database table fits_dir; Once a record matching the criteria is found, the record is locked, and need_handle_flag is set to being_inserted; Update the directory's update time, size, and number of files; Create an uncompressed subdirectory under the deCompressDir directory, uncompress the .fits.gz file in the directory of the advanced data currently being imported into the database into a .fits file, and read the file header to parse out the parameter values; Write or update the file information and the parsed parameter values to the corresponding fits file table.
[0013] Optionally, during the process of retrieving records with need_handle_flag=y from the database table fits_dir, records with only directory information but not yet file information are retrieved first, and sorted in ascending order by directory date; if no such records are found, records with file information already in the directory but needing to be updated are retrieved, and sorted in ascending order by directory date.
[0014] Optionally, after all file information has been processed, delete the decompression subdirectory under deCompressDir and set need_handle_flag=n for the corresponding record in the fits_dir database table.
[0015] Secondly, the present invention provides a system for automatically reading advanced solar observation data into a database, comprising: The data acquisition unit is used to observe the sun using an observer and collect raw data. The data processing unit is used to process raw data and generate advanced data for users to download and analyze. The directory scanning and updating unit is used to classify the storage directory of advanced data according to time and set different scanning frequencies for different categories; starting from the current day's directory, it scans day by day according to the set scanning frequency; it compares the scan results with the records in the database table fits_dir, which is used to store the directory and file information of advanced data; if there are differences, it updates the records in the database table fits_dir and re-enters the file information in the directory; if the database table fits_dir does not contain the directory information, it adds the directory and enters the file information in the directory.
[0016] The beneficial effects of this invention are: This invention can automatically identify the directories that need to be updated after processing advanced solar observation information, while ensuring the real-time storage of advanced data. For all advanced data obtained after processing solar observation information, it automatically reads the directories and completes the process quickly, without missing any updated directories. It parses each compressed advanced data file and then stores it in the database, realizing efficient management, timed scanning, automatic updates, and user-friendly access to advanced solar observation data, ensuring the real-time performance and accuracy of the data. Attached Figure Description
[0017] Figure 1 This is a flowchart of the present invention that scans the entire directory to find the directories that need to be written to or updated in the database.
[0018] Figure 2 This invention updates directory and file information that is to be added to or updated in the database. Detailed Implementation
[0019] The invention will now be described in further detail with reference to the accompanying drawings.
[0020] This invention deploys data reading software for advanced data obtained after processing solar observation information on server A. Server A is used to access large storage B, which is used to store all the advanced data. The database is stored on server C.
[0021] Solar observation information is obtained from observations of the Sun by several detectors: HXI, FMG, SDI, and WST. The data obtained by these detectors is Level 0 data, which cannot be directly downloaded and analyzed by users. It needs to be processed to obtain Level 1 high-level data available for download and analysis. The data from each detector is further divided into the following levels: FMG_lev20, HXI_lev05, HXI_lev10, SDI_lev10, SDI_lev25, and WST_lev10 and WST_lev25.
[0022] Once Level 1 advanced data is generated, the software cannot know this in advance and must periodically access the directories of all data levels. The directory name for saving advanced data includes date information. The software divides the directories into two types: those within three days (including today) and those older than three days. The directories within three days are scanned more frequently, every half hour, because users are more concerned with data from recent days, requiring real-time data. The directories older than three days are scanned once a day, starting at midnight, ensuring that no directory is missed.
[0023] The scanning process is as follows: The directory is scanned day by day from the current date back. Factors such as the directory's update time, size, and number of files within it are considered. If these parameters differ from the data recorded in the `fits_dir` database table, it indicates that the files in that directory have changed. The `fits_dir` table needs to be updated with the directory's information, `need_handle_flag=y`, and the information for all files in that directory needs to be re-entered into the corresponding file tables. If the `fits_dir` table does not contain information for that directory, then the directory's information is written to the `fits_dir` table, `need_handle_flag=y`, and the information for all files in that directory needs to be re-entered into the corresponding file tables.
[0024] Every half hour, every minute, a thread retrieves all records with `need_handle_flag=y` from the `fits_dir` database table, processing the directories marked as needing updates. First, it searches for directory records that only contain directory information but no file information yet. If the search result is 0, it then searches for directory records in the database where file information has been inserted but needs updating. These records are sorted in ascending order by directory date, with the oldest record first. If no matching records are found, the thread does nothing.
[0025] If a record matching the criteria is found, the thread locks the first record by setting `need_handle_flag=being_inserted`. In the locked directory record, the directory update time, directory size, and number of files within the directory are updated. Then, all file information in the corresponding directory is read. Finally, a decompression subdirectory is created under the `deCompressDir` directory.
[0026] Every file in this directory has the .fits.gz extension. Therefore, it is first decompressed to the decompressed subdirectory to generate a .fits file with the same name. Then, the header of this file is read, and its contents are parsed to extract various parameter values. Finally, this file information and the parsed parameter values are written to or updated in the corresponding fits file table.
[0027] After all file information in the directory has been written to or updated in the `fits` file table, delete the decompression subdirectory under the `deCompressDir` directory, and set `need_handle_flag=n` for the record of that directory in the `fits_dir` database table. This completes the update of all records for that directory.
[0028] In one embodiment, the present invention proposes a method for automatically reading advanced solar observation data into a database, specifically including the following steps: (1) System deployment (a) Software Deployment: Deploy software specifically designed for reading and processing advanced solar observation data on server A. This software includes functions such as scheduled scanning, data comparison, information entry, and file processing.
[0029] (b) Storage configuration: Large Storage B: Establishes an access connection with Server A to permanently store all processed Level 1 high-level data.
[0030] Database server C: Installs and configures the database system to store directory information (stored in the fits_dir table) and file details (stored in the corresponding file table) for high-level data.
[0031] (2) Data sources and processing (a) Data Acquisition: The Sun was observed using four instruments (HXI, FMG, SDI, WST), and raw Level 0 data was collected. This data is unprocessed and cannot be directly downloaded and analyzed by users.
[0032] (b) Data Processing: Level 0 data is processed to generate Level 1 advanced data for users to download and analyze. This advanced data is further subdivided into several subcategories based on detector type and processing level, such as fmg_lev20, hxi_lev05, hxi_lev10, sdi_lev10, sdi_lev25, wst_lev10, and wst_lev25.
[0033] (3) Catalog scanning and updating (a) Directory Classification: The software divides the directory where advanced data is stored into two categories based on time: directories within three days (including today) and directories older than three days.
[0034] (b) Scan frequency setting: Catalog within three days: Due to high user attention and strong requirements for real-time data, the scanning frequency is set to once every half hour.
[0035] The catalog from three days ago: The scanning frequency was once a day, starting at 0:00.
[0036] (c) Scanning process (e.g.) Figure 1 (as shown) Sequential scanning: Starting from the current day's catalog, scan each day backwards.
[0037] Change detection: Based on factors such as directory update time, size, and number of files, compare the changes with the records in the `fits_dir` table in the database. If differences are found, update the database record, set `need_handle_flag=y`, and re-enter the information for all files in that directory into the database.
[0038] New directory handling: If the directory information does not exist in the database, a new record is created, need_handle_flag=y is set, and the file information is entered into the database.
[0039] (4) Document information entry and update process (e.g.) Figure 2 (As shown) (a) Thread scheduling: Every half hour, a thread is started every minute to retrieve the record with need_handle_flag=y from the database table fits_dir.
[0040] (b) Search order: Prioritize retrieving records that only have directory information but not yet file information, and sort them in ascending order by directory date.
[0041] If no such record exists, then retrieve records in the directory whose file information has been entered into the database but needs to be updated, and sort them in ascending order by directory date.
[0042] (c) Record locking and processing: Lock records: After retrieving records that meet the criteria, lock the first record and set need_handle_flag=being_inserted.
[0043] Update directory information: Update the update time, size, and number of files in this directory.
[0044] File decompression and parsing: Create a decompression subdirectory under the deCompressDir directory, copy all .fits.gz files from the directory of the high-level data currently being imported into the database to this decompression subdirectory, decompress them to obtain the .fits file, and read the file header to parse out the parameter values.
[0045] Information Input: The file information and parsed parameter values are written to or updated in the corresponding `fits` file table. The `fits` file table is a database table that stores the high-level data information input into the database; one record corresponds to one high-level data `fits` file. This record stores the information of that `fits` file and the parsed parameter values.
[0046] (d) Cleanup and status update: Clean up the decompression directory: After all file information has been processed, delete the decompression subdirectory under deCompressDir.
[0047] Update status flag: Set need_handle_flag=n in the directory record of the fits_dir database table, which indicates that the update process of the directory record is complete.
[0048] In another embodiment, the present invention proposes a system for automatically reading advanced solar observation data into a database, which implements the method for automatically reading advanced solar observation data into a database as described in the foregoing embodiments, comprising: The data acquisition unit is used to observe the sun using an observer and collect raw data. The data processing unit is used to process raw data and generate advanced data for users to download and analyze. The directory scanning and updating unit is used to classify the directories where advanced data is stored by time and set different scanning frequencies for different categories. Starting from the current day's directory, it scans the directory day by day according to the set scanning frequency. The scan results are compared with the records in the database table fits_dir, which stores directory and file information for advanced data. If there are differences, the records in the database table fits_dir are updated, and the file information in the directory is re-entered into the database. If the directory information is not in the database table fits_dir, the directory is created, and the file information in the directory is entered into the database.
[0049] In the system, the functions and execution processes of each unit are the same as the steps of the method in the aforementioned embodiments, so they will not be repeated here.
[0050] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for automatically reading advanced solar observation data into a database, characterized in that, Includes the following steps: The sun is observed using an instrument, and raw data is collected. The raw data is processed to generate advanced data for users to download and analyze. The advanced data storage directory is categorized by time, and different scanning frequencies are set for different categories; Starting from the current day's directory, scan each day according to the set scanning frequency; compare the scan results with the records in the database table fits_dir, which is used to store directory and file information for advanced data; if there are differences, update the records in the database table fits_dir and re-enter the file information in the directory; if the database table fits_dir does not contain the directory information, create the directory and enter the file information in the directory.
2. The method for automatically reading advanced solar observation data into a database as described in claim 1, characterized in that: The observers include four observers: hxi, fmg, sdi, and wst.
3. The method for automatically reading advanced solar observation data into a database as described in claim 1, characterized in that: The advanced data is further subdivided into several subcategories based on detector type and processing level.
4. The method for automatically reading advanced solar observation data into a database as described in claim 1, characterized in that: The advanced data storage directory is divided into two categories based on time: directories within the last three days and directories older than three days.
5. The method for automatically reading advanced solar observation data into a database as described in claim 4, characterized in that: The specific steps for setting different scanning frequencies for different categories are as follows: For directories within the last three days, set the scanning frequency to once every half hour; For directories older than three days, set the scan frequency to once a day, starting at 0:
00.
6. The method for automatically reading advanced solar observation data into a database as described in claim 1, characterized in that: The step of comparing the scan results with the records in the database table fits_dir is as follows: Based on the directory's update time, size, and number of files, compare it with the records in the `fits_dir` database table. If there are differences, update the records in the `fits_dir` database table, set `need_handle_flag=y`, and re-enter the file information in the directory. If the `fits_dir` database table does not contain the directory information, create the directory, set `need_handle_flag=y`, and enter the file information in the directory.
7. The method for automatically reading advanced solar observation data into a database as described in claim 6, characterized in that: The process for storing and updating the file information is as follows: A thread is started at set intervals to retrieve records with need_handle_flag=y from the database table fits_dir; Once a record matching the criteria is found, the record is locked, and need_handle_flag is set to being_inserted; Update the directory's update time, size, and number of files; Create an uncompressed subdirectory under the deCompressDir directory, uncompress the .fits.gz file in the directory of the advanced data currently being imported into the database into a .fits file, and read the file header to parse out the parameter values; Write or update the file information and the parsed parameter values to the corresponding fits file table.
8. The method for automatically reading advanced solar observation data into a database as described in claim 7, characterized in that: During the process of retrieving records with need_handle_flag=y from the database table fits_dir, records with only directory information but not yet file information are retrieved first, and sorted in ascending order by directory date; if no such records are found, records with file information already in the directory but needing to be updated are retrieved, and sorted in ascending order by directory date.
9. The method for automatically reading advanced solar observation data into a database as described in claim 7, characterized in that: After all file information has been processed, delete the decompression subdirectory under deCompressDir and set need_handle_flag=n for the corresponding record in the fits_dir database table.
10. A system for automatically reading advanced solar observation data into a database, characterized in that, include: The data acquisition unit is used to observe the sun using an observer and collect raw data. The data processing unit is used to process raw data and generate advanced data for users to download and analyze. The directory scanning and updating unit is used to classify the storage directory of advanced data according to time and set different scanning frequencies for different categories; starting from the current day's directory, it scans day by day according to the set scanning frequency; it compares the scan results with the records in the database table fits_dir, which is used to store the directory and file information of advanced data; if there are differences, it updates the records in the database table fits_dir and re-enters the file information in the directory; if the database table fits_dir does not contain the directory information, it adds the directory and enters the file information in the directory.