Sun observation fit file cross-server downloading management system and method

Abstract

The invention provides a cross-server downloading management system and a cross-server downloading management method for fitness files for solar observation, and belongs to the field of downloading management of solar observation files. Comprising a website server which is used for recording to-be-processed file downloading requests, including batch downloading requests and single file downloading requests, regularly checking the processing conditions of the file downloading requests, and providing a file downloading service for a user; the file storage server is used for storing a fit file for sun observation; and the intermediate server is used for regularly checking the to-be-processed file downloading request and copying the file from the file storage server to the website server. According to the invention, the problem of file downloading when the network of the website server and the file storage server is isolated is solved, and file downloading can still be provided under the condition that the website server and the file server are not communicated; and a breakpoint resume solution is provided for the accident of file transmission interruption possibly existing in the whole process.

Description

Technical Field

[0001] The present invention belongs to the field of solar observation file download management, and particularly relates to a download management system and method for FITS files of solar observation across servers. Background Art

[0002] Generally, when a website provides a file download service, it is necessary to mount the file storage server to the website server or save the file on the website server. When the website server and the file storage server are network-isolated, file download becomes a problem. The existing technology lacks a reliable download solution for FITS files of solar observation in a cross-network environment, and also cannot provide an effective solution for resume breakpoint transmission in case of unexpected file transmission interruption during the whole process. Summary of the Invention

[0003] Aiming at the problem of file download when the website server and the file storage server are network-isolated, as well as the technical defects of lacking resume breakpoint transmission and status monitoring, the present invention provides a download management system and method for FITS files of solar observation across servers.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] In the first aspect, the present invention provides a download management system for FITS files of solar observation across servers, including:

[0006] A website server, which is used to record file download requests to be processed, including batch download requests and single-file download requests, regularly check the processing status of file download requests, and provide download services for FITS files to users;

[0007] A file storage server, which is used to store FITS files of solar observation;

[0008] An intermediate server, which is used to regularly check file download requests to be processed, and copy files from the file storage server to the website server.

[0009] Optionally, when the file download request is a batch download request, the website server writes a download request record into the database packaging request table, marks the request processing flag bit and the email sending flag bit as to be processed, and writes all FITS file information into the database packaged FITS file table, and marks the processing flag bit of the FITS file as to be processed; the intermediate server regularly reads the database packaging request table, finds the requests with the processing flag bit to be processed, copies the corresponding FITS files in the database packaged FITS file table from the file storage server to the website server, marks the processing flag bit of the corresponding FITS file in the database packaged FITS file table as completed, packs the copied FITS files, and marks the processing flag bit of the corresponding request in the database packaging request table as completed; the website server regularly checks the database packaging request table and sends an email containing a download link for requests with the processing flag bit completed and the email sending flag bit to be processed.

[0010] Optionally, when the file download request is a single file download request, the website server writes the FITS file information to be downloaded into the database single FITS file table and marks the processing flag bit as to be processed; the intermediate server regularly reads the database single FITS file table, finds the FITS file to be processed, copies the FITS file from the file storage server to the website server, and marks the processing flag bit as completed; the website server regularly retrieves the database single FITS file table and sends the file path of the FITS file with the processing flag bit completed to the website page.

[0011] In a second aspect, the present invention provides a method for downloading and managing FITS files for solar observation across servers, including the following steps:

[0012] S11: According to the batch download request, the website server writes a download request record into the database packaging request table, assigns the request processing flag bit need_handle_flag and the email sending flag bit need_email_flag as y, where y indicates to be processed; writes all FITS file information into the database packaged FITS file table, and assigns the need_handle_flag of the FITS file as y;

[0013] S12: The intermediate server periodically reads the database packaging request table to find requests with need_handle_flag = y; copies the corresponding FITS files in the database packaging FITS file table from the file storage server to the website server; assigns the need_handle_flag of the corresponding FITS file in the database packaging FITS file table to n, where n indicates completion; packs the copied FITS files; assigns the need_handle_flag of the corresponding request in the database packaging request table to n;

[0014] S13: The website server periodically checks the database packaging request table and sends an email containing a download link for requests with need_handle_flag = n and need_email_flag = y.

[0015] Optionally, in step S1, the download request record includes the request number request_id, the user email email, the insertion time insert_time, the preset download address download_url, and the save address file_path; the FITS file information includes request_id, email, insert_time, file name file_name, the save path path_in_storage of the file on the file storage server, and the preset save directory target_dir of the file on the website server.

[0016] Optionally, in step S12, the intermediate server periodically reads the database packaging request table, searches for download request records with need_handle_flag = y and processes them sequentially; for each download request record to be processed, finds request_id, first assigns the current time to the start_copy_time of the download request record, then searches the database packaging request table for all FITS file records corresponding to request_id and assigns the current time to the start_copy_time of these FITS file records; reads the FITS files corresponding to request_id in the database packaging FITS file table into memory, and copies the FITS files from the file storage server to target_dir according to the FITS file information in memory; for the copied FITS files, assigns the need_handle_flag of the FITS file in the database packaging FITS file table to n, and at the same time continuously updates the uncopied_file_num of the corresponding download request record in the database packaging request table; after all FITS files are copied, assigns the current time to the end_copy_time of the download request record.

[0017] Optionally, in step S12, the target_dir on the website server is packaged into a file package with the suffix.tar, the end_pack_time of the download request record indicating the end of packaging is assigned the current time, and need_handle_flag is assigned n.

[0018] Thirdly, the present invention provides a method for downloading and managing FITS files for solar observation across servers, including the following steps:

[0019] S21: The website server writes the FITS file information to be downloaded into the single FITS file table in the database according to a single file download request, and assigns the processing flag bit need_handle_flag as y, where y indicates to be processed;

[0020] S22: The intermediate server periodically reads the single FITS file table in the database, finds the FITS file with need_handle_flag = y, copies the FITS file from the file storage server to the website server, and assigns need_handle_flag as n, where n indicates completed;

[0021] S23: The website server periodically retrieves the single FITS file table in the database and sends the file path of the FITS file with need_handle_flag = n to the website page.

[0022] Optionally, in step S21, the FITS file information includes the request number request_id, the user email email, the insertion time insert_time, the file name file_name, the save path path_in_storage of the file on the file storage server, and the preset save directory target_dir of the file on the website server.

[0023] Optionally, in step S22, the FITS file information is read into the memory, and according to the FITS file information in the memory, the FITS file is copied from the file storage server to target_dir; for the copied FITS file, in the single FITS file table in the database, need_handle_flag is assigned n, and the end_copy_time indicating the end of copying is assigned the current time.

[0024] The beneficial effects of the present invention are as follows: The present invention solves the problem of file downloading when the website server and the file storage server are network-isolated. Even when the website server and the file server are not connected, it can still provide FITS file downloading. And for the unexpected situation of file transfer interruption that may occur during the whole process, it provides a solution for resume downloading, which can effectively deal with file copy interruption caused by program stop, database stop or network failure, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 FIG. is a schematic diagram of the operation for a user to download a packaged FITS file.

[0026] Figure 2 FIG. is a schematic diagram of the operation for a user to download a single FITS file.

[0027] Figure 3 FIG. is a processing flow chart of the system for a user to download a packaged FITS file.

[0028] Figure 4 FIG. is a processing flow chart of the system for a user to download a single FITS file. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Now, the present invention will be further described in detail with reference to the accompanying drawings.

[0030] Embodiment 1

[0031] This embodiment provides a cross-server download management system for FITS files of solar observations, including: a website server for recording file download requests to be processed, including batch download requests and single file download requests, regularly checking the processing status of file download requests, and providing FITS file download services to users; a file storage server for storing FITS files of solar observations; and an intermediate server for regularly checking file download requests to be processed and copying files from the file storage server to the website server.

[0032] In use, the user selects the detector type, data level and mode on the page of the website server, enters the email address, and then clicks search. The page displays a list of FITS files that meet the conditions, as well as their total quantity and total size. The user can choose to download single or in batches.

[0033] As shown in Figure 1 , if the user selects batch download and clicks the "Packaged Download" button, the website server records the request number, user email address, insertion time, preset download link (address) and save address in the database, and marks the processing flag bit and the email sending flag bit as the to-be-processed status. At the same time, it records the names, save paths and save directories of all selected files in another database table, and also marks them as to-be-processed.

[0034] The intermediate server checks the batch download requests to be processed every 10 seconds and processes them in order: First, record the start time, create the target folder, and copy the files from the storage server to the website server. During the copying process, the number of unfinished files is updated in real time to support resume from breakpoint. After completion, the files are packaged and the request is marked as completed.

[0035] The program on the website server checks the completed and unnotified requests every 10 seconds and sends an email containing the download link to the user's email. The user can directly download the packaged file through the link.

[0036] As Figure 2 shown, if the user selects to download a single file, after clicking the download button of the corresponding file, the website server records the request number, email, time, file name, and storage path to the database and marks it as pending processing.

[0037] The intermediate server checks the single-file download requests to be processed every 5 seconds, copies the files from the storage server to the website server, and then marks the record as completed.

[0038] The program on the website server checks the single-file download records every 3 seconds. After finding that the file is ready, it immediately feedbacks the path to the page and triggers a download prompt box for the user to download.

[0039] Embodiment 2

[0040] Based on the download management system proposed in Embodiment 1, this embodiment further proposes a method for cross-server download management of FITS files for solar observations.

[0041] The website server A provides a download service for advanced FITS files for solar observations. On the

dataarchive

search

[0042] As Figure 3As shown in the figure, if the user wants to download all FITS files in a package, then click the

Tar and DownloadData

LOG_DL_PACK_REQUEST

LOG_DL_PACK_FITS

[0043] The program deployed on the intermediate server C reads the database table

LOG_DL_PACK_REQUEST

LOG_DL_PACK_REQUEST

LOG_DL_PACK_FITS

LOG_DL_PACK_FITS

LOG_DL_PACK_FITS

LOG_DL_PACK_REQUEST

LOG_DL_PACK_REQUEST

LOG_DL_PACK_REQUEST

[0044] The program on the web server A retrieves the database table

LOG_DL_PACK_REQUEST

[0045] This completes the process of packaging and downloading the fits file request.

[0046] like Figure 4 As shown, if the user wants to download a fits file separately, then click on the same line of the fits page.

[0047] [download] button, and then the program on website server A will write the record of this single file download into the database table [LOG_DL_ONE_FITS], including the download request request_id, user email, insertion time insert_time, file name file_name, the file's save path path_in_storage on file storage server B, the file's pre-set save directory target_dir on website server A, and assign the value y to need_handle_flag.

[0048] The program deployed on intermediate server C reads the database table [LOG_DL_ONE_FITS] every 5 seconds, searches for records with a need_handle_flag set to y, and processes these records sequentially. For each pending download request record, the program first reads the FIT file information associated with that record into memory. A pre-defined file copy directory, target_dir, is created on website server A. Based on the FIT file information stored in memory, the program copies the FIT file from the storage directory on file storage server B to target_dir on website server A. For each copied FIT file, the need_handle_flag of the FIT file record in the database table [LOG_DL_ONE_FITS] is set to n, and the end_copy_time is assigned to the current time. This completes the file copying for that FIT record.

[0049] The program on server A checks the value of need_handle_flag of the single file download record every 3 seconds. When need_handle_flag = n, it knows that the fits file has been copied to server A. Then it immediately sends the fits file path on server A to the front page, which automatically pops up a download dialog box for the fits file and the user clicks download.

[0050] At this point, the complete process of downloading a single fits file is completed.

[0051] The above are only the preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should be regarded as within the protection scope of the present invention.

Claims

1. A FITS file cross-server download management system for solar observations, characterized in that Including: A website server for recording file download requests to be processed, including batch download requests and single file download requests, regularly checking the processing status of file download requests, and providing users with the download service of FITS files; A file storage server for storing FITS files of solar observations; An intermediate server for regularly checking file download requests to be processed and copying files from the file storage server to the website server.

2. The download management system for FITS files of solar observations across servers according to claim 1, characterized in that: When the file download request is a batch download request, the website server writes a download request record into the database package request table, marks the processing flag bit and the email sending flag bit of the request as to be processed, writes all FITS file information into the database package FITS file table, and marks the processing flag bit of the FITS file as to be processed; The intermediate server regularly reads the database package request table, finds requests with the processing flag bit to be processed, copies the corresponding FITS files in the database package FITS file table from the file storage server to the website server, marks the processing flag bit of the corresponding FITS file in the database package FITS file table as completed, packs the copied FITS files, and marks the processing flag bit of the corresponding request in the database package request table as completed; The website server regularly checks the database package request table and sends an email containing a download link for requests with the processing flag bit completed and the email sending flag bit to be processed.

3. The FITS file cross-server download management system for solar observation according to claim 1, wherein: When the file download request is a single file download request, the website server writes the information of the FITS file to be downloaded into the database single FITS file table and marks the processing flag bit as to be processed; The intermediate server regularly reads the database single FITS file table, finds the FITS file to be processed, copies the FITS file from the file storage server to the website server, and marks the processing flag bit as completed; The website server regularly retrieves the database single FITS file table and sends the file path of the FITS file with the processing flag bit completed to the website page.

4. A method for downloading and managing FITS files for solar observations across servers, characterized in that, Including the following steps: S11: According to the batch download request, the website server writes a download request record into the database package request table, assigns the processing flag bit need_handle_flag and the email sending flag bit need_email_flag of the request as y, where y represents to be processed; writes all FITS file information into the database package FITS file table, and assigns the need_handle_flag of the FITS file as y; S12: The intermediate server periodically reads the database packaging request table to find requests with need_handle_flag = y; copies the corresponding FITS files in the database packaging FITS file table from the file storage server to the website server; assigns n to need_handle_flag of the corresponding FITS files in the database packaging FITS file table, where n indicates completion; packs the copied FITS files; assigns n to need_handle_flag of the corresponding request in the database packaging request table. S13: The website server periodically checks the database packaging request table and sends an email with a download link for requests with need_handle_flag = n and need_email_flag = y.

5. The download management method for FITS files of solar observations across servers according to claim 4, characterized in that: In step S1, the download request record includes the request number request_id, the user email email, the insertion time insert_time, the preset download address download_url, and the save address file_path; the FITS file information includes request_id, email, insert_time, file name file_name, the save path path_in_storage of the file on the file storage server, and the preset save directory target_dir of the file on the website server.

6. A method for downloading and managing FITS files for solar observation across servers as described in claim 5, characterized in that: In step S12, the intermediate server periodically reads the database packaging request table, searches for download request records with need_handle_flag = y and processes them sequentially; for each download request record to be processed, finds request_id, first assigns the current time to the start copy time start_copy_time of the download request record, then searches the database packaging request table for all FITS file records corresponding to request_id and assigns the current time to the start_copy_time of these FITS file records; reads the FITS files corresponding to request_id in the database packaging FITS file table into memory, and copies the FITS files from the file storage server to target_dir according to the FITS file information in memory; for the copied FITS files, assigns n to need_handle_flag of the FITS file in the database packaging FITS file table, and at the same time continuously updates the number of uncopied files uncopied_file_num of the corresponding download request record in the database packaging request table; after all FITS files are copied, assigns the current time to the end copy time end_copy_time of the download request record.

7. The method for downloading and managing FITS files for solar observation across servers according to claim 6, wherein: In step S12, the target_dir on the website server is packaged into a file package with the suffix.tar, the end_pack_time of the download request record is assigned the current time, and need_handle_flag is assigned n.

8. A method for downloading and managing FITS files for solar observations across servers, characterized in that, It includes the following steps: S21: The website server writes the FITS file information to be downloaded into the single FITS file table in the database according to a single file download request, and assigns the processing flag bit need_handle_flag as y, where y indicates to be processed; S22: The intermediate server regularly reads the single FITS file table in the database, finds the FITS file with need_handle_flag = y, copies the FITS file from the file storage server to the website server, and assigns need_handle_flag as n, where n indicates completed; S23: The website server regularly retrieves the single FITS file table in the database and sends the file path of the FITS file with need_handle_flag = n to the website page.

9. A method for downloading and managing FITS files for solar observation across servers as claimed in claim 8, characterized in that: In step S21, the FITS file information includes the request number request_id, the user email email, the insertion time insert_time, the file name file_name, the save path path_in_storage of the file on the file storage server, and the preset save directory target_dir on the website server.

10. A method for downloading and managing FITS files for solar observations across servers as described in claim 9, characterized in that: In step S22, the FITS file information is read into the memory, and according to the FITS file information in the memory, the FITS file is copied from the file storage server to target_dir; for the copied FITS file, in the single FITS file table in the database, need_handle_flag is assigned n, and the end_copy_time of the copy completion is assigned the current time.

Images