Method for downloading and analyzing global marine meteorological data file

Abstract

The invention relates to the technical field of marine meteorological data file processing, in particular to a global marine meteorological data file downloading and analyzing method which comprises the following steps: S1, calculating a splicing downloading url; s2, starting multi-thread tasks, respectively downloading, and recording downloading tasks to form a downloading record table during downloading; s3, if downloading fails, the state of the task is set to be failed, if downloading succeeds, the state of the task is set to be successful, and meanwhile, the downloaded meteorological files are classified and stored; s4, traversing the downloading record table, obtaining tasks which fail in downloading, and starting multi-task downloading; s5, after downloading in the step S4 fails, the state of the downloading record table is set to be failed, the step S4 is continued, and if downloading succeeds, the file is stored; and S6, analyzing the acquired meteorological data. According to the invention, high-speed, high-efficiency and accurate analysis of global marine meteorological data files can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of processing marine meteorological data files, and in particular to a method for downloading and parsing global marine meteorological data files. Background Art

[0002] Marine meteorological data files are generally stored in nc files. Many meteorological research institutions around the world will encapsulate their real-time observations and predicted global marine meteorological data into nc files and store them on their own servers for all users around the world to download and use. Therefore, there will be many download channels and many download methods. If we want to download various types of global marine meteorological data files, we need to download them from different channels. Most meteorological download methods on the market are python downloads, which have a slow download speed. Then, the downloaded nc files are parsed to get the meteorological data we want. Secondly, the characteristic of global meteorological data files is that the amount of data is very large, because each marine meteorological data file stores a large amount of meteorological data. Finally, after the downloaded meteorological data nc file is obtained, this file cannot be opened for viewing with conventional software. Therefore, it is necessary to perform certain data parsing on the nc file to obtain the global marine meteorological data we want. Based on the huge amount of data mentioned above, the parsing of nc meteorological data files is also troubled. Some existing data parsing methods on the market cannot perform targeted data parsing, resulting in complex parsing and long parsing time. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a method for downloading and parsing global marine meteorological data files, which is used to solve the problems of slow download speed, complex parsing and long parsing time in the existing Python.

[0004] To solve the above problems, the technical solution of the present invention is as follows: A method for downloading and parsing global marine meteorological data files, comprising the following steps:

[0005] S1. Calculate and splice download URLs;

[0006] S2, start multi-threaded tasks, download separately, and record the download tasks to form a download record table during downloading;

[0007] S3. If the download fails, the status of the task is set to failure. If the download succeeds, the status of the task is set to success. At the same time, the downloaded meteorological files are classified and stored;

[0008] S4, traverse the download record table, obtain the tasks that failed to download, and start multi-task download;

[0009] S5. After the download fails in step S4, the status of the download record table is set to failure, and step S4 is continued. If the download is successful, the file is stored;

[0010] S6. Analyze the acquired meteorological data.

[0011] Furthermore, the URL is determined according to the type, time and address of the meteorological data file to be downloaded.

[0012] Furthermore, in step S2, the download URLs are decomposed into different download URLs according to the time, and then multiple thread tasks are started to download through Java code according to the decomposed download URLs.

[0013] Furthermore, in step S3, the classification storage stores different meteorological types in different folders, and stores the folders according to time.

[0014] Furthermore, the analysis in step S6 is performed by combining java code analysis and command line cdo analysis.

[0015] Furthermore, the Java code analysis is used for global meteorological data analysis.

[0016] Furthermore, the data parsed by the Java code is converted, calculated in the Java environment, and stored in the database.

[0017] Furthermore, the command line cdo parsing is used for parsing a certain point.

[0018] Furthermore, the command line cdo parses the meteorological data of a certain point through the input longitude and latitude coordinates and the meteorological data type.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. By adopting the Java code download method, the multi-threaded multi-task download method is started, and the download time is accurately controlled through the code, so that the download speed can be fast, the time control is flexible, and different download types can be realized, such as http, ftp, etc. At the same time, a failure re-download mechanism is created to ensure that the global marine meteorological data files can be downloaded at every point in the world at every time point.

[0021] 2. By combining Java code parsing and command line parsing, different parsing methods are adopted for different parsing requirements, which can achieve high-speed, efficient and accurate parsing of global marine meteorological data files. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the meteorological data download process of the present invention;

[0023] Figure 2 This is a schematic diagram of the process of steps S3 and S4 of the present invention;

[0024] Figure 3 It is a schematic diagram of the meteorological data analysis process of the present invention. DETAILED DESCRIPTION

[0025] like Figure 1 , Figure 2 , Figure 3 shown.

[0026] A method for downloading and parsing global marine meteorological data files comprises the following steps:

[0027] S1. Determine and calculate the download URL according to the type, time and address of the meteorological data file to be downloaded;

[0028] S2. Decompose the download URLs into different ones according to the time. Then, according to the decomposed download URLs, start multiple thread tasks to download through Java code. Each task downloads data according to the assigned different URLs. In this way, each task performs its own duties and can make full use of the server's bandwidth and CPU resources. At the same time, when allocating download tasks, it is necessary to maintain a download record table, which records the download time, download URL, and whether the download is successful.

[0029] S3. If the download fails, the download record table needs to be updated and the status of the task is set to failure. If the download is successful, the status in the download record table is set to success. At the same time, the downloaded meteorological files are classified and stored. Different meteorological types are stored in different folders, and they are stored in folders according to time. This is convenient for later file retrieval and later analysis of marine meteorological data files.

[0030] S4, traverse the download record table, obtain the tasks that failed to download, and start multi-task download;

[0031] S5. After the download fails in step S4, the status of the download record table is set to failure, and step S4 is continued. If the download is successful, the file is stored;

[0032] S6. parse the acquired meteorological data by combining java code parsing and command line cdo parsing.

[0033] The advantage of using Java code for parsing is that it can quickly and accurately parse global meteorological data, and the parsed data can be directly converted, calculated, and stored in the database in the Java environment, which is convenient and fast, and is more suitable for parsing global meteorological data files.

[0034] For parsing a certain point, the command line cdo method is used. This method can accurately parse the meteorological data of a certain point through the incoming latitude and longitude coordinates and meteorological data type, and the speed is also very fast. Combining these two methods, it is possible to quickly and accurately parse the global marine meteorological data.

[0035] The download of global ocean weather files in this application is implemented through Java code and deployed on the server. It can automatically and regularly download weather files without intervention. The downloaded weather files are stored in the server hard disk. The parsing is carried out by combining Java code and command line cdo. The command line cdo is directly implemented by the server plug-in.

[0036] The above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for downloading and parsing global marine meteorological data files, characterized in that: The following steps are involved: S1. Calculate and splice download URLs; S2, start multi-threaded tasks, download separately, and record the download tasks to form a download record table during downloading; S3. If the download fails, the status of the task is set to failure. If the download succeeds, the status of the task is set to success. At the same time, the downloaded meteorological files are classified and stored; S4, traverse the download record table, obtain the tasks that failed to download, and start multi-task download; S5. After the download fails in step S4, the status of the download record table is set to failure, and step S4 is continued. If the download is successful, the file is stored; S6. Analyze the acquired meteorological data.

2. The downloading and parsing method according to claim 1, characterized in that: The URL is determined by the type, time and address of the meteorological data file to be downloaded.

3. The downloading and parsing method according to claim 1, characterized in that: In step S2, the download URLs are decomposed into different ones according to the time, and then multiple thread tasks are started to download according to the decomposed download URLs through Java code.

4. The downloading and parsing method according to claim 1, characterized in that: In step S3, the classification storage stores different meteorological types in different folders, and stores them in different folders according to time.

5. The downloading and parsing method according to claim 1, characterized in that: The analysis in step S6 is performed by combining java code analysis and command line cdo analysis.

6. The downloading and parsing method according to claim 5, characterized in that: The java code analysis is used for global meteorological data analysis.

7. The downloading and parsing method according to claim 6, characterized in that: The data parsed by the Java code is converted, calculated in the Java environment, and stored in the database.

8. The downloading and parsing method according to claim 7, characterized in that: The command line cdo parsing is used for parsing at a certain point.

9. The downloading and parsing method according to claim 8, characterized in that: The command line cdo parses the meteorological data of a certain point through the input latitude and longitude coordinates and meteorological data type.

Images