A method and system for automatically updating and storing ERA5 source data

Abstract

The invention discloses an automatic updating and warehousing method and system for ERA5 source data, belonging to the technical field of data processing, comprising: configuring an operating environment and parameters, obtaining the start and end dates of downloaded data to generate an Era5 download time series, excluding the existing Era5 data dates to obtain a submission time series; submitting a download task in a multi-threaded manner to obtain an Era5 original data file; verifying the Era5 original data file by using a hash function to obtain Era5 first data; obtaining Era5 second data by pre-processing the Era5 first data; updating the obtained Era5 second data to a database file, and updating the start and end dates of the downloaded data; obtaining an evaluation coefficient by calculating a data update rate and a data integrity coefficient, comparing the evaluation coefficient with a standard evaluation value, and realizing evaluation of the system automation degree.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an automatic updating and warehousing method and system for ERA5 source data. Background Art

[0002] ERA5 is the latest global reanalysis dataset provided by the European Centre for Medium-Range Weather Forecasts (ECMWF), covering a variety of meteorological variables from the surface to the atmosphere. Due to its high resolution and long time series, ERA5 data has important application value in climate research, weather forecasting, and environmental monitoring.

[0003] Patent No. CN117633134A also discloses a weather process diagnosis method based on meteorological reanalysis data, including data downloading by the script download_era5_data.py, which can download ERA5 reanalysis data; data visualization by multiple ncl scripts, which can draw a multi-factor spatiotemporal change distribution map, and these two parts are combined into the run_scripts.sh script. By running the bash script, the corresponding python and ncl scripts are called to achieve the function of one-click data download and visualization; however, the patent has the following defects:

[0004] 1. Lack of automatic update function: CN117633134A does not implement the functions of "automatically monitoring data update status", "automatically submitting download tasks", and "automatically downloading and checking data integrity", which requires users to manually detect and start the data download process.

[0005] 2. There is a lack of evaluation on the automation of the automatic update and storage process of ERA5 source data. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for automatically updating and storing ERA5 source data to solve the deficiencies in the background technology.

[0007] The purpose of the present invention can be achieved by the following technical solution: A method for automatically updating and storing ERA5 source data, comprising the following steps:

[0008] Step 1: Register the CDS account URL and obtain the user key, download and configure the operating environment, create a virtual environment, install several dependent libraries, and put the main package main into the virtual environment;

[0009] Step 2: Obtain the start and end dates of the downloaded data and generate an Era5 download time series according to the preset time interval, and exclude the existing Era5 data dates to obtain the submission time series; submit the download task in a multi-threaded manner to obtain the Era5 original data file; wherein the preset time interval is daily or monthly;

[0010] Step 3: Calculate the hash value of the Era5 original data file by using the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data;

[0011] Using the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing abnormal values ​​of the Era5 first data, to obtain the Era5 second data;

[0012] Step 4: Create a database file, update the acquired Era5 second data to the database file, and update the start and end dates of the downloaded data to the script of the main program package;

[0013] Step 5: Periodically record the automatic update and storage process of Era5 source data on a monthly basis, obtain the evaluation coefficient by calculating the data update rate and data integrity coefficient, compare the evaluation coefficient with the standard evaluation value, and evaluate the degree of system automation.

[0014] Preferably, in step one, a file named .cdsapirc is created to save the url and key.

[0015] Preferably, in step 2, a date legality check is performed on the start and end dates of the downloaded data, including illegal date detection and latest download date verification;

[0016] The illegal date detection is specifically as follows: checking whether the input start and end dates conform to the standard date format, and if the input date format is wrong, automatically correcting it to a legal date;

[0017] Check whether the start date is earlier than or equal to the end date. If the start date is later than the end date, control the end date to be equal to the start date.

[0018] The latest download date check is specifically as follows: the input end date is compared with the current UTC time. If the end date is later than the set latest download date, the end date is automatically corrected to the set latest download date.

[0019] Preferably, in step three, the method for verifying the Era5 original data file is as follows:

[0020] The hash value of the Era5 original data file is calculated using the hash function SHA-256 to obtain the first hash value, and then compared with the second hash value provided by the ECMWF official website. If the first hash value is consistent with the second hash value, the verification is successful and the Era5 first data is obtained;

[0021] If the first hash value is inconsistent with the second hash value, the verification fails and the data is automatically re-downloaded until it succeeds or reaches the preset maximum number of retries; wherein the preset maximum number of retries is 3.

[0022] Preferably, in step 3, the method for removing outliers in the Era5 first data is as follows:

[0023] Traverse the Era5 first data to obtain the data set corresponding to each variable, calculate the mean and standard deviation of each data set; calculate the difference between all data points and the mean of the data set, set a normal threshold, compare the calculated difference with the normal threshold, and mark the data points that exceed the normal threshold as outliers for removal.

[0024] Preferably, in step 4, the created database file contains four types: airp, heat, evap, and wind;

[0025] Determine whether the data separate storage switch state is on or off. If the data separate storage switch state is on, output them to the corresponding 4 database files respectively; otherwise, output them all to the same database file.

[0026] Preferably, in step 5, the amount of data in the database before and after an update within a cycle is obtained, marked as CG;

[0027] The time required for each update of ERA5 source data within the acquisition cycle is marked as update duration T;

[0028] Add up all update durations T to get the total update duration ZT;

[0029] By formula The data update rate is calculated; where V is the data update rate and ZT is the total update time;

[0030] Obtain the amount of ERA5 data successfully imported into the database within the period and the total amount of original data that needs to be imported;

[0031] By formula The data integrity coefficient is calculated; where SW is the data integrity coefficient, CG is the amount of ERA5 data successfully imported into the database, and ZS is the total amount of original data.

[0032] Preferably, the evaluation coefficient is calculated by a formula based on the data update rate and the data integrity coefficient;

[0033] The calculation formula is: Where P is the evaluation coefficient, V is the data update rate, SW data integrity coefficient, is the weight coefficient of the data update rate, is the weight coefficient of the data integrity coefficient;

[0034] The obtained evaluation coefficient is compared with the standard evaluation value. If the evaluation coefficient is greater than or equal to the standard evaluation value, a green signal is generated; if the evaluation coefficient is less than the standard evaluation value, an orange signal is generated and an early warning is issued.

[0035] In order to solve the problem, the present invention also discloses an automatic update storage system for ERA5 source data, comprising:

[0036] The environment configuration module is used to register the CDS account URL and obtain the user key, download and configure the environment, create a virtual environment, install several dependent libraries, and put the main program package main into the virtual environment;

[0037] The data download module is used to obtain the start and end dates of the downloaded data and generate an Era5 download time series according to the preset time interval, and exclude the existing Era5 data dates to obtain the submission time series; submit the download task in a multi-threaded manner to obtain the Era5 original data file;

[0038] The data reading and preprocessing module is used to calculate the hash value of the Era5 original data file through the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data;

[0039] Using the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing abnormal values ​​of the Era5 first data, to obtain the Era5 second data;

[0040] The data update storage module is used to create a database file, update the acquired Era5 second data to the database file, and update the start and end dates of the downloaded data to the script of the main program package main;

[0041] The evaluation module is used to record the automatic update and storage process of Era5 source data on a monthly basis, obtain the evaluation coefficient by calculating the data update rate and data integrity coefficient, compare the evaluation coefficient with the standard evaluation value, and evaluate the degree of system automation.

[0042] Compared with the existing solutions, the present invention achieves the following beneficial effects:

[0043] By creating a separate, isolated environment to run the ERA5 source data automatic update repository system without affecting other parts of the system;

[0044] By determining the data range and time range to be downloaded and generating the corresponding download tasks, the start and end time of Era5 download is automatically corrected and updated to ensure the validity of the time, thereby avoiding download failures caused by time errors. Among them, the submission time series is obtained by excluding the existing Era5 data date, avoiding repeated downloads of existing data, which consumes more time and energy, improves download efficiency, and speeds up the acquisition of Era5 original data files.

[0045] By using hash functions to verify the Era5 original data files, the integrity and consistency of the data files can be verified to prevent data tampering or damage during transmission; the Era5 first data is pre-processed, and the data is initially cleaned and sorted to ensure the accuracy and availability of the downloaded data, preparing for subsequent storage operations;

[0046] By updating the acquired Era5 second data to the database file and updating the start and end dates of the downloaded data, the automation function of the system is realized;

[0047] By monitoring and evaluating the operating results of the entire system and obtaining the evaluation coefficient, we can understand the system's degree of automation and data processing capabilities, thus providing a basis for system optimization and improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] Figure 1 The present invention is a flowchart of a method for automatically updating and storing ERA5 source data.

[0050] Figure 2 The present invention is a module block diagram of an automatic update storage system for ERA5 source data. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0052] Embodiment 1, as Figure 1 As shown, a method for automatically updating and storing ERA5 source data includes the following steps:

[0053] Step 1: Register the CDS account URL and obtain the user key, download and configure the Python 3.6.8 environment, create a virtual environment, install several dependent libraries including but not limited to the cdsapi library, netCDF4, and numpy library, and put the main package main into the virtual environment;

[0054] Among them, create a file named .cdsapirc to save the url and key;

[0055] The main program package main includes but is not limited to setting download data parameters, time range and geographical area, and the specific contents are shown in Table 1;

[0056] Table 1 Introduction to the scripts in the main package main

[0057]

[0058] Step 2: Obtain the start and end dates of the downloaded data and generate an Era5 download time series according to a preset time interval, and exclude the existing Era5 data dates to obtain a submission time series; wherein the preset time interval is daily or monthly;

[0059] Submit the download task in multi-thread mode to obtain the Era5 original data file; the number of multi-threads is set to 20;

[0060] Furthermore, the python programming language is used to implement date legality detection on the start and end dates of the downloaded data, including illegal date detection and latest download date inspection; wherein the python programming language is written according to the specific application environment;

[0061] The illegal date detection is specifically as follows: checking whether the input start and end dates conform to the standard date format. If the input date format is incorrect, it will be automatically corrected to a legal date, such as "20230140", which will be automatically corrected to "20230131";

[0062] Check whether the start date is earlier than or equal to the end date. If the start date is later than the end date (such as 20230102>20230101), control the end date to be equal to the start date;

[0063] The latest download date check is specifically as follows: comparing the input end date with the current UTC time, and if the end date is later than the set latest download date, automatically correcting the end date to the set latest download date;

[0064] It should be noted that the latest download date set in the latest download date check is less than 6 days compared to the current UTC time; UTC time is Coordinated Universal Time, and Era5 data is released 5 days after UTC time. The extra day is to obtain a full 24 hours of data when submitting the download task;

[0065] In this embodiment, multi-threading technology is used to implement data download. Since the maximum number of queued tasks for a personal account is 20, the multi-threading is set to 20, which reduces the number of submissions, queue time, and improves download efficiency and speed; each thread will download the corresponding data according to the assigned time sequence to ensure efficient execution of the task.

[0066] Step 3: Calculate the hash value of the Era5 original data file by using the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data;

[0067] Among them, the method for verifying the Era5 original data file is as follows:

[0068] The hash value of the Era5 original data file is calculated using the hash function SHA-256 to obtain the first hash value, and then compared with the second hash value provided by the ECMWF official website. If the first hash value is consistent with the second hash value, the verification is successful and the Era5 first data is obtained;

[0069] If the first hash value is inconsistent with the second hash value, the verification fails and the data is automatically re-downloaded until it succeeds or reaches the preset maximum number of retries; wherein the preset maximum number of retries is 3.

[0070] Using the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing abnormal values ​​of the Era5 first data, to obtain the Era5 second data;

[0071] Among them, the method for removing outliers in the first data of Era5 is as follows:

[0072] Traverse the Era5 first data to obtain the data set corresponding to each variable, calculate the mean and standard deviation of each data set; calculate the difference between all data points and the mean of the data set, set a normal threshold, compare the calculated difference with the normal threshold, mark the data points exceeding the normal threshold as outliers and remove them, where the normal threshold is specifically specified according to the actual value;

[0073] For example, for the daily average temperature data set, the mean and standard deviation of all daily average temperature data are calculated and recorded as and ; For each daily average temperature data, calculate the difference between its value and the average value, recorded as , set the normal threshold to ,like , then the daily average temperature data point is an abnormal value; otherwise, the daily average temperature data point is a normal value;

[0074] Step 4: Create a database file containing four types: airp, heat, evap, and wind, which is used to update the acquired Era5 second data to the database file;

[0075] Exemplarily, the corresponding relationship shown in Table 2 saves the Era5 second data;

[0076] Table 2 Corresponding variables saved in each database file

[0077]

[0078] Further, it is determined whether the data separate storage switch state is on or off. If the data separate storage switch state is on, the data are output to the corresponding four database files respectively; otherwise, all of them are output to the same database file;

[0079] Update the start and end dates of the downloaded data to the script of the main package;

[0080] Embodiment 2, the present invention is an automatic update storage system for ERA5 source data, further comprising the following steps:

[0081] Step 5: Periodically record the automatic update and storage process of Era5 source data in monthly units, obtain the amount of data in the database before and after an update within the cycle, and mark it as CG;

[0082] The time required for each update of ERA5 source data within the acquisition cycle is marked as update duration T, in hours;

[0083] Add up all update durations T to get the total update duration ZT in hours;

[0084] By formula The data update rate is calculated; where V is the data update rate and ZT is the total update time;

[0085] Obtain the amount of ERA5 data successfully imported into the database within the period and the total amount of original data that needs to be imported;

[0086] By formula The data integrity coefficient is calculated; where SW is the data integrity coefficient, CG is the amount of ERA5 data successfully imported into the database, and ZS is the total amount of original data;

[0087] Based on the data update rate and the data integrity coefficient, the evaluation coefficient is calculated by a formula;

[0088] The calculation formula is:

[0089] Where P is the evaluation coefficient, V is the data update rate, SW data integrity coefficient, is the weight coefficient of the data update rate, is the weight coefficient of the data integrity coefficient;

[0090] The obtained evaluation coefficient is compared with the standard evaluation value. If the evaluation coefficient is greater than or equal to the standard evaluation value, a green signal is generated; if the evaluation coefficient is less than the standard evaluation value, an orange signal is generated and an early warning is issued; wherein, the high standard evaluation value is determined based on the historical update data of the system;

[0091] It should be noted that the green signal indicates the degree of automation of ERA5 source data update and storage; the orange signal indicates that the automation function of ERA5 source data update and storage is low, and it is still necessary to optimize the system configuration, increase hardware resources or improve the algorithm to improve the degree of automation of the system.

[0092] Embodiment 3, as Figure 2 As shown, the present invention provides an automatic update storage system for ERA5 source data, comprising:

[0093] The environment configuration module is used to register the CDS account URL and obtain the user key, download and configure the environment, create a virtual environment, install several dependent libraries, and put the main program package main into the virtual environment;

[0094] The data download module is used to obtain the start and end dates of the downloaded data and generate an Era5 download time series according to the preset time interval, and exclude the existing Era5 data dates to obtain the submission time series; submit the download task in a multi-threaded manner to obtain the Era5 original data file;

[0095] The data reading and preprocessing module is used to calculate the hash value of the Era5 original data file through the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data;

[0096] Using the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing abnormal values ​​of the Era5 first data, to obtain the Era5 second data;

[0097] The data update storage module is used to create a database file, update the acquired Era5 second data to the database file, and update the start and end dates of the downloaded data to the script of the main program package main;

[0098] The evaluation module is used to record the automatic update and storage process of Era5 source data on a monthly basis, obtain the evaluation coefficient by calculating the data update rate and data integrity coefficient, compare the evaluation coefficient with the standard evaluation value, and evaluate the degree of system automation.

[0099] In one or more embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the above-described embodiments of the invention are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.

[0100] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0102] It is obvious to a person skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatically updating and storing ERA5 source data, characterized in that: The following steps are involved: Step 1: Register a CDS account URL and obtain the user key, download and configure the runtime environment, create a virtual environment, install several dependent libraries, and place the main program package in the virtual environment; Step 2: Obtain the start and end dates of the downloaded data and generate an Era5 download time series according to the preset time interval. Exclude the existing Era5 data dates to obtain the submission time series; submit the download task in a multi-threaded manner to obtain the Era5 original data file; the preset time interval is daily or monthly; Step 3: Calculate the hash value of the Era5 original data file by using the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data; Use the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing outliers in the Era5 first data to obtain the Era5 second data; Step 4: Create a database file, update the acquired Era5 second data to the database file, and update the start and end dates of the downloaded data to the script of the main program package; Step 5: Periodically record the automatic update and storage process of Era5 source data on a monthly basis, calculate the data update rate and data integrity coefficient to obtain the evaluation coefficient, compare the evaluation coefficient with the standard evaluation value, and evaluate the degree of system automation, including: Get the amount of data in the database before and after an update within the cycle, marked as CG; Get the time required for each update of the ERA5 source data within the cycle, marked as update duration T; Add up all update durations T to get the total update duration ZT; By formula Calculate the data update rate; where V is the data update rate and ZT is the total update time; Obtain the amount of ERA5 data successfully imported into the database during the period and the total amount of original data that needs to be imported; By formula The data integrity coefficient is calculated; where SW is the data integrity coefficient, CG is the amount of ERA5 data successfully imported into the database, and ZS is the total amount of original data; Based on the data update rate and data integrity coefficient, the evaluation coefficient is calculated by the formula; The calculation formula is: ; Where P is the evaluation coefficient, V is the data update rate, SW data integrity coefficient, is the weight coefficient of the data update rate, is the weight coefficient of the data integrity coefficient; The obtained evaluation coefficient is compared with the standard evaluation value. If the evaluation coefficient is greater than or equal to the standard evaluation value, a green signal is generated; if the evaluation coefficient is less than the standard evaluation value, an orange signal is generated and an early warning is issued.

2. The method for automatically updating and storing ERA5 source data according to claim 1, characterized in that: In step 1, create a file named .cdsapirc to save the URL and key.

3. The method for automatically updating and storing ERA5 source data according to claim 1, characterized in that: In step 2, a date validity check is performed on the start and end dates of the downloaded data, including illegal date detection and latest download date verification; The illegal date detection is specifically as follows: checking whether the input start and end dates conform to the standard date format. If the input date format is incorrect, it is automatically corrected to a legal date; Check whether the start date is earlier than or equal to the end date. If the start date is later than the end date, control the end date to be equal to the start date. The latest download date check is specifically as follows: comparing the input end date with the current UTC time; if the end date is later than the set latest download date, the end date is automatically corrected to the set latest download date.

4. The method for automatically updating and storing ERA5 source data according to claim 1, characterized in that: In step 3, the method for verifying the Era5 original data file is as follows: Use the hash function SHA-256 to calculate the hash value of the Era5 original data file to obtain the first hash value, and compare it with the second hash value provided by the ECMWF official website. If the first hash value and the second hash value are consistent, the verification is successful and the Era5 first data is obtained; If the first hash value is inconsistent with the second hash value, the verification fails and the data is automatically re-downloaded until it succeeds or the preset maximum number of retries is reached; wherein the preset maximum number of retries is 3.

5. The method for automatically updating and storing ERA5 source data according to claim 4, characterized in that: In step 3, the method for removing outliers in the first data of Era5 is as follows: Traverse the Era5 first data to obtain the data set corresponding to each variable, calculate the mean and standard deviation of each data set; calculate the difference between all data points and the mean of the data set, set a normal threshold, compare the calculated difference with the normal threshold, and mark the data points that exceed the normal threshold as outliers for removal.

6. The method for automatically updating and storing ERA5 source data according to claim 5, characterized in that: In step 4, the database files created contain four types: airp, heat, evap, and wind; Determine whether the data separate storage switch state is on or off. If the data separate storage switch state is on, output it to the corresponding 4 database files respectively; Otherwise all output is to the same database file.

7. An automatic update and storage system for ERA5 source data, applying an automatic update and storage method for ERA5 source data according to any one of claims 1 to 6, characterized in that: include: The environment configuration module is used to register the CDS account URL and obtain the user key, download and configure the environment, create a virtual environment, install several dependent libraries, and place the main program package in the virtual environment; The data download module is used to obtain the start and end dates of the downloaded data and generate an Era5 download time series according to the preset time interval. It also excludes the existing Era5 data dates to obtain the submission time series; it submits the download task in a multi-threaded manner to obtain the Era5 original data file; The data reading and preprocessing module is used to calculate the hash value of the Era5 original data file through the hash function SHA-256, verify the Era5 original data file, and obtain the Era5 first data; Use the dependent library to read the Era5 first data and perform preprocessing, including standardizing the format and unit of the Era5 first data and removing outliers in the Era5 first data to obtain the Era5 second data; The data update and storage module is used to create a database file, update the acquired Era5 second data to the database file, and update the start and end dates of the downloaded data to the script of the main program package; The evaluation module is used to periodically record the automatic update and storage process of Era5 source data on a monthly basis, obtain the evaluation coefficient by calculating the data update rate and data integrity coefficient, and compare the evaluation coefficient with the standard evaluation value to evaluate the degree of system automation.

Images