Marine data automatic downloading method and system

Abstract

The invention relates to an automatic marine data downloading method and system, and belongs to the technical field of data transmission, and the method comprises the steps that downloading parameters and task parallel requirements are acquired, and the downloading parameters comprise remote server information and the number of downloaded files; a downloading thread is created through a main thread according to the number of downloaded files and task parallel requirements, and the downloading thread independently sends a downloading request to a remote server; the remote server sends a downloading file to a downloading thread according to the downloading request, the downloading thread processes data of the downloading file in parallel according to a preset semaphore, the downloading file is stored in a database, and a downloading success notification is sent to the main thread; transmitting the successfully downloaded download file to a remote server through an asynchronous mechanism; wherein the asynchronous transmission step and the multi-thread downloading step are carried out in parallel; through multi-thread parallel downloading and asynchronous transmission, quick and reliable downloading and distribution of files are achieved, the number of downloading threads is controlled through semaphores, and stability and high efficiency of the downloading process are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a method and system for automatically downloading marine data. Background Art

[0002] At present, the acquisition of ocean data mainly relies on the data download services provided by various data centers and online databases. These services usually provide data download interfaces through protocols such as FTP and HTTP. Users need to manually select and download the required data files, and most of them use single-threaded download tools, such as traditional wget and curl tools to achieve batch download of data, but the speed is slow and the efficiency is low. It can be seen that traditional single-threaded download tools and scripts cannot fully utilize bandwidth resources, resulting in slow download speeds.

[0003] In the process of downloading other data, in order to solve the problem that the single download thread makes the download of the target file slow, multi-threaded downloading is proposed. For example, CN118018544A discloses a method, device, electronic device and storage medium for downloading a target file, which discloses downloading a plurality of sub-files to be downloaded divided by the target file in parallel based on multiple threads to obtain the number of successfully downloaded sub-files; compressing the successfully downloaded sub-files, merging all the compressed sub-files, and decompressing the merged compressed files to obtain the target file. However, the above patent still does not limit the number of download requests and the number of multi-threads, which may easily cause obvious abnormalities in the server response time when the number of multiple download threads exceeds the range of the number of parallel requests. Summary of the invention

[0004] In view of the deficiencies existing in the related art, the purpose of the present invention is to provide a method and system for automatic downloading of marine data, so as to solve the technical problem that the prior art does not limit the number of download requests and the number of multiple threads, which easily causes obvious abnormalities in server response time when the number of multiple download threads exceeds the range of the number of parallel requests.

[0005] The present invention provides a method for automatically downloading ocean data, comprising the following steps:

[0006] Data acquisition step: obtaining download parameters and task parallel requirements, wherein the download parameters include remote server information and the number of downloaded files;

[0007] Thread creation step: creating a download thread through the main thread according to the number of downloaded files and the task parallel requirements, and the download thread independently sends a download request to the remote server;

[0008] Multi-thread downloading step: the remote server sends the download file to the download thread according to the received download request, the download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread;

[0009] Asynchronous transmission step: transmitting the successfully downloaded file to the remote server through an asynchronous mechanism;

[0010] Wherein, the asynchronous transmission step and the multi-thread downloading step are performed in parallel.

[0011] The embodiment of the present invention realizes fast and reliable downloading and distribution of files through multi-threaded parallel downloading and asynchronous transmission, and controls the number of downloading threads through semaphores to ensure the stability and efficiency of the downloading process.

[0012] In some embodiments of the present invention, the thread creation step also includes: dynamically adjusting the maximum number of download requests by sending a plurality of preset numbers of download requests to the remote server and detecting the response of each download request, and dynamically adjusting the value of the semaphore according to the response speed of the remote server within the range of the maximum number of download requests.

[0013] The embodiment of the present invention determines the maximum number of download requests and adjusts the download thread through a semaphore, so that the maximum speed download can be performed by using multiple threads without being restricted.

[0014] In some embodiments of the present invention, the asynchronous transmission step is specifically:

[0015] The copy thread is triggered by an asynchronous mechanism, the copy thread reads the download file from the database and asynchronously transmits it to the remote server, the remote server returns a transmission completion notification to the copy thread after receiving the download file, and the copy thread sends the transmission completion notification to the download thread.

[0016] The embodiment of the present invention combines asynchronous transmission with multi-threading technology so that file downloading and transmission can be performed in parallel, thereby improving the system's concurrency capability and resource utilization efficiency. Threads communicate and synchronize through queues or events to ensure the orderliness of operations, and through exception handling mechanisms and data integrity confirmation, ensure the stability of the system and the reliability of file transmission.

[0017] In some embodiments of the present invention, after the download thread completes parallel processing of the data of the downloaded file, it creates an identification file, and determines whether all the downloaded files are downloaded by judging whether the number of the identification files is the same as the number of downloaded files. If the number of the identification files is not the same as the number of downloaded files, the downloaded files are not fully downloaded, and the download of the downloaded files continues using the breakpoint resume technology.

[0018] The embodiment of the present invention provides an integrity check operation, which can complete data verification and error handling mechanisms according to the identification file, thereby ensuring the integrity and correctness of the downloaded file.

[0019] In some embodiments of the present invention, range check, mutation check, consistency check, spatiotemporal consistency check and drift check are used to monitor and verify in real time whether the data quality of the downloaded file meets the requirements of ocean forecasting and ocean assimilation.

[0020] The data quality control and data verification mechanisms integrated in the embodiment of the present invention ensure that the downloaded data meets the requirements of ocean forecasting and ocean assimilation, thereby improving the forecast accuracy.

[0021] Some embodiments of the present invention further provide a marine data automatic downloading system, comprising:

[0022] Data acquisition module: obtains download parameters and task parallel requirements, wherein the download parameters include remote server information and the number of downloaded files;

[0023] Thread creation module: creates a download thread through the main thread according to the number of downloaded files and the task parallel requirements, and the download thread independently sends a download request to the remote server;

[0024] Multi-thread download module: the remote server sends the download file to the download thread according to the received download request, the download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread;

[0025] Asynchronous transmission module: transmits the successfully downloaded files to the remote server through an asynchronous mechanism;

[0026] The asynchronous transmission module and the multi-thread download module are run in parallel.

[0027] Among them, the thread creation module also includes: dynamically adjusting the maximum number of download requests by sending multiple preset numbers of download requests to the remote server, detecting the response of each download request, and dynamically adjusting the value of the semaphore according to the response speed of the remote server within the maximum number of download requests.

[0028] Wherein, the asynchronous transmission module is specifically:

[0029] The copy thread is triggered by an asynchronous mechanism, the copy thread reads the download file from the database and asynchronously transmits it to the remote server, the remote server returns a transmission completion notification to the copy thread after receiving the download file, and the copy thread sends the transmission completion notification to the download thread.

[0030] Among them, after the download thread completes parallel processing of the data of the downloaded file, it creates an identification file, and determines whether the downloaded file is completely downloaded by judging whether the number of the identification files is the same as the number of downloaded files. If the number of the identification files is not the same as the number of downloaded files, the downloaded file is not completely downloaded, and the download of the downloaded file continues through the breakpoint resume technology.

[0031] Among them, through range check, mutation check, consistency check, spatiotemporal consistency check and drift check, real-time monitoring and verification are carried out to see whether the data quality of the downloaded file meets the requirements of ocean forecasting and ocean assimilation.

[0032] The embodiment of the present invention realizes fast and reliable downloading and distribution of files by adopting the technical solutions of semaphore control, breakpoint resume, multi-threaded parallel downloading and asynchronous transmission, while ensuring the accuracy and reliability of downloaded data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A flow chart of a method for automatically downloading ocean data provided by an embodiment of the present invention;

[0035] Figure 2 An application flow chart of an automatic downloading method for ocean data provided by an embodiment of the present invention;

[0036] Figure 3 A timing diagram of multi-thread downloading provided by an embodiment of the present invention;

[0037] Figure 4 A timing diagram of an asynchronous transmission provided by an embodiment of the present invention;

[0038] Figure 5 A structural diagram of an automatic ocean data downloading system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0043] Embodiment 1

[0044] As attached Figure 1 As shown, the present invention provides a method for automatically downloading ocean data, comprising the following steps:

[0045] Data acquisition step S1: obtain download parameters and task parallel requirements, the download parameters include remote server information and the number of downloaded files; optionally, the download parameters also include the number of download attempts, download interval time and user information; optionally, create a download process log and a download failure log, record detailed information in the download process according to the download parameters, including the download status and error information of each file, and distinguish between normal operation logs and error logs through different log levels, such as I NFO and ERROR; the download process log records the download status and time of each file for subsequent maintenance and analysis; the download failure log records the errors and the number of failed attempts that occurred during the download process for troubleshooting;

[0046] Thread creation step S2: Create a download thread through the main thread according to the number of downloaded files and the task parallel requirements. The download thread independently sends a download request to the remote server; optionally, in conjunction with the reference to the attached Figure 2As shown, before the download request is sent, the expired cache data is cleaned up to release the storage space; optionally, before downloading the file, the download type is selected first, when the HYCOM type is selected, the HYCOM download is started; when the GFS type is selected, the GFS type is started; when the GDAS type is selected, the GDAS download is started;

[0047] Multi-thread downloading step S3: the remote server sends the download file to the download thread according to the received download request, the download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread;

[0048] Asynchronous transmission step S4: transmitting the successfully downloaded file to the remote server through an asynchronous mechanism;

[0049] The asynchronous transmission step S4 and the multi-thread downloading step S3 are performed in parallel.

[0050] Based on the above method, fast and reliable downloading and distribution of files are achieved through multi-threaded parallel downloading and asynchronous transmission. The number of download threads is controlled by semaphores to ensure the stability and efficiency of the download process.

[0051] Combined with Figure 3As shown, in order to intelligently determine the maximum number of simultaneous accesses allowed to a website, a semaphore is used to adjust the download thread so that the file can be downloaded at the maximum speed without being restricted. The maximum number of download requests is dynamically adjusted by sending a plurality of preset number of download requests to a remote server and detecting the response of each download request. Within the maximum number of download requests, the value of the semaphore is dynamically adjusted according to the response speed of the remote server. For example, when the response speed of the monitoring server and the returned status code is HTTP 429, it means "Too Many Download Requests". Requests", first send 1, 2, 4, 8 download requests, observe the response of each request, if the response time increases significantly or an error occurs, it can be inferred that the number of requests has approached or reached the limit; at this time, the main thread creates download thread 1, download thread 2, ..., download thread N according to the number of downloaded files and task parallel requirements, and the above download threads independently send download request 1, download request 2, ..., download request N to the remote server. Since it is a multi-threaded operation, each thread can work in parallel without blocking each other, thereby improving the download efficiency. This request is asynchronous, which means that the request of thread 1 will not wait for the request of thread 2 to be completed before initiating. The above threads process the returned data separately, so they do not interfere with each other. Multi-threading can effectively utilize network bandwidth and reduce the waiting time of single-thread download. After parallel download processing, the above threads send download success notifications to the main thread respectively; the remote server sends download file 1, download file 2, ..., download file N to the above threads respectively;

[0052] Among them, the semaphore is set to an integer value, which represents the number of concurrent download threads currently allowed. Every time a thread starts downloading, the value of the semaphore will decrease; when the download is completed, the value of the semaphore will increase. The value of the semaphore is dynamically adjusted according to the feedback from the server; for example, if it is found that the number of download threads is too large and the server response is slow, the semaphore value is reduced and the number of concurrent download threads is reduced; conversely, if it is found that more threads can be supported, the semaphore value is increased to ensure that all available bandwidth is used as much as possible for downloading without triggering server access restrictions.

[0053] When the download thread completes parallel processing of the data of the downloaded file, an identification file is created, and whether the downloaded file is completely downloaded is determined by determining whether the number of identification files is the same as the number of downloaded files. If the number of identification files is different from the number of downloaded files, the downloaded file is not completely downloaded, and the download of the downloaded file is continued through the breakpoint resume technology; if the number of identification files is different from the number of downloaded files, the downloaded file is completely downloaded, and the total download time is recorded; optionally, the identification file is a ".completed" file;

[0054] Furthermore, combined with Figure 4As shown, the downloaded files are transferred to the specified directory of the remote server through SSH and SFTP protocols, wherein the copy thread is triggered through the asynchronous mechanism, the copy thread reads the downloaded files from the database and asynchronously transfers them to the remote server, and the remote server returns the transfer completion notification to the copy thread after receiving the downloaded files, and the copy thread sends the transfer completion notification to the download thread. In this process, asynchronous transmission is combined with multi-threaded downloading, so that file download and transmission can be carried out in parallel, which improves the concurrency capability and resource utilization efficiency. Threads communicate and synchronize through queues or events to ensure the orderliness of operations, and through exception handling mechanism and data integrity confirmation, the stability of the system and the reliability of file transmission are guaranteed.

[0055] In order to ensure that the downloaded data maintains high quality and reliability during the acquisition and transmission process, the data quality of the downloaded files is monitored and verified in real time through range check, mutation check, consistency check, spatiotemporal consistency check and drift check to see if they meet the requirements of ocean forecasting and ocean assimilation. Among them, the range check is to verify whether the downloaded data is within a reasonable physical range; the mutation check is to detect whether there are unreasonable mutations in the data; the consistency check is to compare the consistency of data from different data sources or at different time points; the spatiotemporal consistency check is to check the consistency of the downloaded data in time and space to ensure the rationality of the data; the drift check is to monitor the long-term change trend of the data and detect whether there is drift.

[0056] The data quality control and data verification mechanisms integrated in the embodiment of the present invention ensure that the downloaded data meets the requirements of ocean forecasting and ocean assimilation, thereby improving the forecast accuracy.

[0057] Embodiment 2

[0058] Reference Figure 5 As shown, an embodiment of the present invention further provides an automatic downloading system for marine data, comprising:

[0059] Data acquisition module 1: obtains download parameters and task parallel requirements. Download parameters include remote server information and the number of downloaded files.

[0060] Thread creation module 2: Create download threads through the main thread according to the number of downloaded files and task parallel requirements, and the download threads independently send download requests to the remote server;

[0061] Multithreaded download module 3: The remote server sends the download file to the download thread according to the received download request. The download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread;

[0062] Asynchronous transmission module 4: transmits the successfully downloaded files to the remote server through an asynchronous mechanism;

[0063] The asynchronous transmission module 4 and the multi-thread download module 3 are operated in parallel.

[0064] Among them, the thread creation module 2 also includes: dynamically adjusting the maximum number of download requests by sending multiple preset numbers of download requests to the remote server, detecting the response of each download request, and dynamically adjusting the value of the semaphore according to the response speed of the remote server within the maximum number of download requests.

[0065] Among them, the asynchronous transmission module 4 is specifically:

[0066] The copy thread is triggered by an asynchronous mechanism, the copy thread reads the download file from the database and asynchronously transmits it to the remote server, and after receiving the download file, the remote server returns a transfer completion notification to the copy thread, and the copy thread sends the transfer completion notification to the download thread.

[0067] Among them, after the download thread completes parallel processing of the data of the downloaded file, it creates an identification file, and determines whether all the downloaded files are downloaded by judging whether the number of the identification files is the same as the number of downloaded files. If the number of the identification files is not the same as the number of downloaded files, the downloaded files are not fully downloaded, and the download of the downloaded files is continued through the breakpoint resume technology.

[0068] Among them, range check, mutation check, consistency check, spatiotemporal consistency check and drift check are used to monitor and verify in real time whether the data quality of the downloaded files meets the requirements of ocean forecasting and ocean assimilation.

[0069] It should be noted that the above is a reference to a method and system for automatically downloading ocean data, and the present invention is not limited thereto.

[0070] The embodiments of the present invention realize multi-threaded parallel downloading, fully utilize network bandwidth resources, greatly improve download speed, realize full automation of data download process, reduce human intervention, improve work efficiency, provide data verification and error handling mechanism, ensure the integrity and correctness of downloaded data, integrate data quality control and data verification mechanism, ensure that downloaded data meets ocean forecast and ocean assimilation requirements, improve forecast accuracy, and solve the technical problem that the prior art does not limit the number of download requests and the number of multiple threads, which easily causes obvious abnormalities in server response time when the number of multiple download threads exceeds the range of the number of parallel requests.

[0071] Finally, it should be noted that: the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0072] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A method for automatically downloading ocean data, characterized in that: The steps include: Data acquisition step: obtaining download parameters and task parallel requirements, wherein the download parameters include remote server information and the number of downloaded files; Thread creation step: creating a download thread through the main thread according to the number of downloaded files and the task parallel requirements, and the download thread independently sends a download request to the remote server; Multi-thread downloading step: the remote server sends the download file to the download thread according to the received download request, the download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread; Asynchronous transmission step: transmitting the successfully downloaded file to the remote server through an asynchronous mechanism; Wherein, the asynchronous transmission step and the multi-thread downloading step are performed in parallel.

2. The method for automatically downloading ocean data according to claim 1, characterized in that: The thread creation step also includes: dynamically adjusting the maximum number of download requests by sending a plurality of preset numbers of download requests to the remote server and detecting the response of each download request, and dynamically adjusting the value of the semaphore according to the response speed of the remote server within the maximum number of download requests.

3. The method for automatically downloading ocean data according to claim 1, characterized in that: The asynchronous transmission steps are specifically as follows: The copy thread is triggered by an asynchronous mechanism, the copy thread reads the download file from the database and asynchronously transmits it to the remote server, the remote server returns a transmission completion notification to the copy thread after receiving the download file, and the copy thread sends the transmission completion notification to the download thread.

4. The method for automatically downloading ocean data according to claim 1, characterized in that: Also includes: After the download thread completes parallel processing of the data of the downloaded file, an identification file is created, and whether all the downloaded files are downloaded is determined by judging whether the number of the identification files is the same as the number of downloaded files. If the number of the identification files is not the same as the number of downloaded files, the downloaded files are not downloaded in full, and the download of the downloaded files is continued through the breakpoint resume technology.

5. The method for automatically downloading ocean data according to claim 1, characterized in that: Also includes: Through range check, mutation check, consistency check, spatiotemporal consistency check and drift check, the data quality of the downloaded file is monitored and verified in real time to see whether it meets the requirements of ocean forecasting and ocean assimilation.

6. An automatic downloading system for marine data, characterized in that: include: Data acquisition module: obtains download parameters and task parallel requirements, wherein the download parameters include remote server information and the number of downloaded files; Thread creation module: creates a download thread through the main thread according to the number of downloaded files and the task parallel requirements, and the download thread independently sends a download request to the remote server; Multi-thread download module: the remote server sends the download file to the download thread according to the received download request, the download thread processes the data of the download file in parallel according to the preset signal quantity, saves the download file to the database, and sends a download success notification to the main thread; Asynchronous transmission module: transmits the successfully downloaded files to the remote server through an asynchronous mechanism; The asynchronous transmission module and the multi-thread download module are run in parallel.

7. The marine data automatic downloading system according to claim 6, characterized in that: The thread creation module also includes: dynamically adjusting the maximum number of download requests by sending a plurality of preset numbers of download requests to the remote server and detecting the response of each download request, and dynamically adjusting the value of the semaphore according to the response speed of the remote server within the maximum number of download requests.

8. The marine data automatic downloading system according to claim 6, characterized in that: The asynchronous transmission module is specifically: The copy thread is triggered by an asynchronous mechanism, the copy thread reads the download file from the database and asynchronously transmits it to the remote server, the remote server returns a transmission completion notification to the copy thread after receiving the download file, and the copy thread sends the transmission completion notification to the download thread.

9. The marine data automatic downloading system according to claim 6, characterized in that: Also includes: After the download thread completes parallel processing of the data of the downloaded file, an identification file is created, and whether all the downloaded files are downloaded is determined by judging whether the number of the identification files is the same as the number of downloaded files. If the number of the identification files is not the same as the number of downloaded files, the downloaded files are not downloaded in full, and the download of the downloaded files is continued through the breakpoint resume technology.

10. The marine data automatic downloading system according to claim 6, characterized in that: Also includes: Through range check, mutation check, consistency check, spatiotemporal consistency check and drift check, the data quality of the downloaded file is monitored and verified in real time to see whether it meets the requirements of ocean forecasting and ocean assimilation.

Images