CMIP6 data processing method based on historical meteorological data and optimal machine learning model

Abstract

The invention discloses a CMIP6 data processing method based on historical meteorological data and an optimal machine learning model, and the method can effectively evaluate the processing precision difference of the model for different types of meteorological data, thereby selecting the optimal model to obtain more credible future meteorological data changes. Comprising the following steps: extracting daily CMIP6 original data; establishing a simulation training set and a simulation test set for the CMIP6 data, and establishing an actual measurement training set and an actual measurement test set for the corresponding historical meteorological data in the same period; processing the simulation test set according to a machine learning model, and selecting an optimal machine learning model corresponding to different types of meteorological data; establishing an actual measurement training set according to all historical meteorological data, and establishing a simulation training set according to all historical period CMIP6 data; the optimal machine learning model is selected, a simulation test set is established according to the CMIP6 data in the future to-be-optimized period and serves as the input of the trained machine learning model, and the output result is the processing result of the optimal machine learning model on the CMIP6 data.

Description

Technical Field

[0001] The present invention belongs to the field of agricultural hydrology, and in particular relates to a method for processing CMIP6 data based on historical meteorological data and machine learning. Background Art

[0002] Future climate change will have an impact on food production that cannot be ignored. The World Climate Research Program (WCRP) has launched a new round of the International Coupled Model Intercomparison Project (CMIP6). It provides global meteorological data under different development models in this century, but due to its large scale, the accuracy of using CMIP6 data to influence crop yields in specific regions of China is inevitably insufficient. Therefore, in the actual application process, it is necessary to solve the problem of how to improve the prediction accuracy. Mastering the relevant climate data of specific regions can correct CMIP6 data to improve the credibility of CMIP6 data. CMIP6 is based on large-scale (such as continental scale) climate data with low spatial resolution, generally above 100 kilometers. In the actual application of CMIP6 data, CMIP6 needs to be downscaled, which is not sufficient for the actual situation of different geographical locations. The machine learning downscaling method is a common method to improve the accuracy of CMIP6 data. This method can convert low-resolution CMIP6 output data into high-resolution and more accurate climate data by using real or nearly real high-resolution historical climate data as a reference. Therefore, in the actual application process, it is necessary to explore the change characteristics of regional meteorological data through machine learning, so as to further optimize the CMIP6 data so that it retains the characteristics of different greenhouse gas emission patterns while being closer to the actual situation in the study area. Summary of the invention

[0003] In view of this, the present invention provides a method for processing CMIP6 data based on historical meteorological data and a machine learning model, comprising the following steps: S1: Extract daily CMIP6 raw data; S2: Use the acquired CMIP6 data to establish a simulated training set and a simulated test set, and use the corresponding historical meteorological data of the same period to establish a measured training set and a measured test set; S3: Process the simulated test set according to the machine learning model trained in S2, and select the best machine learning model corresponding to different types of meteorological data; S4: Establish a measured training set based on all historical meteorological data and a simulated training set based on all historical CMIP6 data; train the best machine learning model selected in S3, and establish a simulated test set based on the CMIP6 data of the future period to be optimized as the input of the trained machine learning model. The output result is the processing result of the best machine learning model on the CMIP6 data.

[0004] Furthermore, the S1 also includes: The CMIP6 daily meteorological data for historical and future periods are directly extracted based on the latitude and longitude geographical locations required for the research.

[0005] Furthermore, the S1 also includes: According to the characteristics of CMIP6 data, the units of CMIP6 meteorological data and historical meteorological data are unified.

[0006] Furthermore, in S2: The obtained historical CMIP6 data are divided into equal parts.

[0007] Specifically, in S2: Random forest, recurrent neural network, support vector machine, adaptive booster, decision tree, LASSO regression model, multivariate linear regression and ensemble learning model were trained based on simulated training sets and measured training sets.

[0008] Furthermore, in S2: Calculate the R between the meteorological data processed by machine learning and the measured test set 2 and RMSE.

[0009] Specifically, the direct extraction method is as follows: S21: Import multidimensional array database; S22: Search for files matching the pattern; S23: traverse the file and process the data; S24: Combine all results and save.

[0010] Specifically, the method for comprehensively selecting the best machine learning model corresponding to different types of meteorological data in S3 is as follows: S31: Arrange the historical and CMIP6 daily meteorological data of all test years in order into a column as machine learning input test data; S32: Build a neural network model: S33: compile model; S34: training model; S35: Use the model to make predictions; S36: Calculate evaluation indicators.

[0011] In a second aspect, a CMIP6 data processing device based on historical meteorological data and an optimal machine learning model is provided, specifically an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0012] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the steps of the method when executed by a processor.

[0013] Beneficial effects of the present invention: Different machine learning models have different processing results for meteorological data. When using machine learning to process CMIP6 data, how to select the most suitable machine learning model according to the type of meteorological data to be processed has become a hot topic for scholars to discuss. Therefore, using basic machine learning models, deep learning models, and integrated models to process CMIP6 data based on historical meteorological data in the same mode can effectively evaluate the differences in the processing accuracy of the above models for different types of meteorological data, so as to select the best model to obtain more reliable future meteorological data changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flow chart of this method; Figure 2a-2e Respectively represent the optimization of CMIP6 Tmax, Tmin, Rsds, pr, and SfcWind data by eight machine learning models; Figure 3 Taylor diagrams of optimization results of 8 machine learning models; Figure 4a-4l Results of the optimal machine learning model processing future CMIP6 data. DETAILED DESCRIPTION

[0015] In order to make the purpose, advantages and features of the present invention more obvious, the following specific implementation methods are combined with software codes to further describe the present invention in detail.

[0016] like Figure 1 -4. A specific embodiment of a method for processing CMIP6 data based on historical meteorological data and an optimal machine learning model is: in, Figure 4a-4l In the figure, AE represents the processing of meteorological data from Yinchuan (A), Zhongning (B), Qingtongxia (C), Litong (D), and Huinong (E).

[0017] The specific steps are: S1: Extract daily CMIP6 raw data in the form of a directly obtained .nc database file. Use a program edited based on python3.7 to directly extract the CMIP6 daily meteorological data (maximum and minimum temperatures, average temperature, wind speed, radiation, rainfall, etc.) for historical and future periods according to the latitude and longitude geographic locations required for the study. According to the characteristics of CMIP6 data, unify the units of CMIP6 meteorological data and historical meteorological data.

[0018] S2: The obtained historical CMIP6 data are equally divided to establish simulation training sets and simulation test sets, and the corresponding historical meteorological data of the same period are used to establish measured training sets and measured test sets. According to the simulation training sets and measured training sets, random forest (RF), recurrent neural network (RNN), support vector machine (SVM), adaptive booster (AdaBoost), decision tree (DTR), LASSO regression model (Lasso), multiple linear regression (MLR) and integrated learning model (IML) are trained.

[0019] Figure 2a to Figure 2e In the figure, lowercase letters represent optimizations of CMIP6 Tmax, Tmin, Rsds, pr, and SfcWind data (uppercase letters (A) to (H)) by eight machine learning models. The blue line represents the CMIP6 daily meteorological data extracted by longitude and latitude, the yellow line represents the daily measured data, and the green line represents the daily meteorological data after machine learning processing.

[0020] S3: The simulated test set is processed according to the 8 machine learning models of RF, DNN, DTR, SVM, Lasso, MLR, Adaboost, and IML trained in S2 to obtain the results after processing by the 8 machine learning models. The R between the meteorological data processed by the 8 machine learning models and the measured test set is calculated respectively. 2 and RMSE, and comprehensively select the best machine learning model corresponding to different types of meteorological data.

[0021] Figure 3 In the figure, capital letters (A) to (E) represent the Taylor diagrams of the processing results of 8 machine learning methods, namely RF, DNN, DTR, SVM, Lasso, MLR, Adaboost, and IML, on 5 types of meteorological data, namely Tmax (A), Tmin (B), sfcWind (C), rsds (D), and pr (E). The arc coordinates represent the R between the processed values ​​and the measured values. 2 ; The grey dotted line represents RMSE; the horizontal axis represents the standard deviation.

[0022] S4: Establish a measured training set based on all historical meteorological data, and a simulated training set based on all CMIP6 data from all historical periods. Train the best machine learning model selected in S3, and establish a simulated test set based on the CMIP6 data of the future period to be optimized as the input of the trained machine learning model. The output result is the processing result of the best machine learning model on the CMIP6 data.

[0023] like Figure 4a-4lThe calculation results, the blue line represents the future CMIP6 daily meteorological data extracted by longitude and latitude, the yellow line represents the daily meteorological data after machine learning processing, 126, 245, 585 represent the future CMIP6 meteorological data under the low emission, balanced, and high emission development paths (SSP1-2.6, SSP2-4.5, SSP5-8.5), A~E represent the five test points of Yinchuan, Zhongning, Qingtongxia, Litong, and Huinong, respectively. Figure 4a -c is the Rsds radiation data before and after processing using the Lasso regression model; Figure 4d -f is the Tmax maximum temperature data before and after processing with neural network; Figure gi is the Tmin minimum temperature data before and after processing with support vector machine; Figure jl is the sfcWind near-surface wind speed data before and after processing with random forest.

[0024] In a specific embodiment, a program edited based on python3.7 is used to directly extract the geographical location according to the latitude and longitude required for the study, and the method is as follows: S21: Import the following libraries: xarray (xr): used to process multidimensional array data, especially suitable for processing data in meteorology and earth science; pandas (pd): used to process and analyze structured data, especially tabular data; numpy (np): used to process numerical calculations, providing multidimensional array objects and various mathematical functions; glob: used to find file paths that match specific patterns.

[0025] S22: Find files that match the pattern: files = glob.glob("F:\\cmip6\\tas\\126\\ nc")glob.glob function finds all matches F:\\cmip6\\tas\\126\\ nc mode file paths and stores them in the files list.

[0026] S23: Traverse the file and process the data: for file in files: data = xr.open_dataset(file) data1 = data.interp(lon=106.77, lat=39.22).tas result = pd.DataFrame(data1) data_result.append(result) Open files: xr.open_dataset(file) opens each file and loads it as an xarray dataset.

[0027] Interpolation: data.interp(lon=106.77, lat=39.22).tas interpolates the data set and extracts the tas (temperature) data at a specific longitude (106.77) and latitude (39.22). Convert to DataFrame: pd.DataFrame(data1) converts the interpolated data to a pandas DataFrame.

[0028] Store results: Add the processing result of each file to the data_result list.

[0029] S24: Combine all results and save: Fresult = pd.concat(data_result, axis=1) pd.concat(data_result, axis=1) merges all DataFrames in the data_result list into one large DataFrame Fresult in column direction (axis=1).

[0030] Fresult.to_excel("D:\\Future dry hot wind\\Monthly TAS 126\\Huinong Monthly TAS 126.xlsx") Fresult.to_excel saves the merged DataFrame Fresult to an Excel file in the specified path.

[0031] In a specific embodiment, according to the characteristics of CMIP6 data, S1 unifies the units of CMIP6 meteorological data and historical meteorological data, and the method is as follows: use Python3.9 to program and process .nc files, extract data according to the longitude and latitude of the test site, obtain temperature data in K, rainfall data in mm / s, wind speed data in m / s, and radiation data in W / m2·s, and convert them into daily meteorological data according to 0℃=273.15K; 86400mm / s=1mm / day; 1W / m2·s=11.574MJ / m2·day.

[0032] In a specific embodiment, a measured training set and a measured test set are established in S2, and a machine learning model is trained according to the simulated training set and the measured training set, and the method is as follows: import pandas as pd import numpy as np import tensorflow as tf from sklearn.metrics import mean_squared_error from sklearn.metrics import mean_squared_error, r2_score Import the libraries required to run the code, numpy is used for numerical operations, tensorflow is used to build and train the neural network model, and mean_squared_error and r2_score in sklearn.metrics are used to evaluate the model performance. Arrange the historical and CMIP6 daily meteorological data of all training years in order as machine learning input training data.

[0033] x_train = np.array(data["simulated training set"]) y_train = np.array(data["actual training set"]) x_train = x_train.reshape(-1, 1) y_train = y_train.reshape(-1, 1) Extract the training and test data and convert them into NumPy arrays. x_train and y_train are the input features of the model. The data is reshaped into a two-dimensional array, where each sample is a one-dimensional array. This is to meet the expected format of the input data of the tensorflow model, that is, (num_samples, num_features). Here each sample has only one feature, so num_features=1.

[0034] In a specific embodiment, S3 comprehensively selects the best machine learning model corresponding to different types of meteorological data, and the method is as follows (taking the neural network algorithm as an example): S31: Arrange the historical and CMIP6 daily meteorological data of all test years in order and use them as machine learning input test data: x_test = np.array(data["Simulation test set"]) y_test = np.array(data["actual test set"]) x_test = x_test.reshape(-1, 1) y_test = y_test.reshape(-1, 1) S32: Build a neural network model: model = tf.keras.Sequential([ tf.keras.layers.Dense(64,activation='relu', input_shape=(1,)), tf.keras.layers.Dense(64,activation='relu'), tf.keras.layers.Dense(1) ]) Create a sequential model with two hidden layers, 64 neurons in each layer, and ReLU activation function. The output layer has one neuron, and because it is a regression problem, no activation function is used.

[0035] S33: Compile the model: model.compile(optimizer='adam', loss='mse') Configure the learning process of the model to use the Adam optimizer and mean squared error (MSE) as the loss function.

[0036] S34: Training model: model.fit(x_train, y_train, epochs=100, verbose=0) The model is trained for 100 epochs. verbose=0 means that log information during the training process is not printed.

[0037] S35: Use the model to make predictions: y_pred = model.predict(x_test) Use the trained model to predict the test set and get the predicted value y_pred.

[0038] S36: Calculate evaluation indicators: rmse1 = np.sqrt(mean_squared_error(x_test, y_test)) rmse2 = np.sqrt(mean_squared_error(y_pred, y_test)) r 2 = r 2 _score(y_test, y_pred) print("Original RMSE:", rmse1) print("RMSE after optimization:", rmse2) print("R-square score:", r 2 ) Original RMSE: the root mean square error between the simulated test set and the measured test set; RMSE after optimization: the root mean square error between the model prediction value and the measured test set; R-square score: The coefficient of determination of the model, which assesses the ability of the model to explain the variability of the data.

[0039] According to the RMSE and R of 8 machine learning models corresponding to each meteorological data 2 The output selects the best machine learning model for the specific meteorological data.

[0040] It should be pointed out that the output result described in S4 is the processing result of CMIP6 data by the best machine learning model. The method is as follows: a column of measured training sets is established based on all historical meteorological data, and a column of simulated training sets is established based on all historical CMIP6 data. The best machine learning model selected in S3 is trained, and a column of simulated test sets is established based on the CMIP6 data of the future period to be optimized as the input of the trained machine learning model. The processing method is the same as S3, but the measured test set is not input. The output result is saved as a new file, which is the processing result of CMIP6 data based on historical meteorological data and the best machine learning model.

[0041] It should be noted that any process or method description in the embodiments may be understood as representing a module, fragment or portion of a code including one or more executable instructions for implementing steps of a specific logical function or process, and that the scope of the preferred embodiments of the present invention includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0042] It should be noted that the logic and / or steps in the embodiments, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or in combination with these instruction execution systems, devices or equipment. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or equipment, or in combination with these instruction execution systems, devices or equipment. More specific examples of computer-readable media (non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0043] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0044] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0045] In addition, each functional module in the embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0046] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0047] The above embodiments describe the technical solutions of the present invention in detail. Obviously, the present invention is not limited to the described embodiments. Based on the embodiments of the present invention, people familiar with the technical field can also make various changes accordingly, but any changes that are equivalent or similar to the present invention belong to the scope of protection of the present invention.

[0048] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for processing CMIP6 data based on historical meteorological data and an optimal machine learning model, characterized in that: The steps include: S1: Extract daily CMIP6 raw data; S2: Use the acquired CMIP6 data to establish a simulated training set and a simulated test set, train the machine learning model based on the simulated training set and the measured training set, and use the corresponding historical meteorological data of the same period to establish a measured training set and a measured test set; S3: Process the simulated test set according to the learning model trained in S2, and select the best machine learning model corresponding to different types of meteorological data; S4: Establish a measured training set based on all historical meteorological data and a simulated training set based on all historical CMIP6 data; train the best machine learning model selected in S3, and establish a simulated test set based on the CMIP6 data of the future period to be optimized as the input of the trained machine learning model. The output result is the processing result of the best machine learning model on the CMIP6 data.

2. The method according to claim 1, characterized in that The S1 also includes: The CMIP6 daily meteorological data for historical and future periods are directly extracted based on the latitude and longitude geographical locations required for the research.

3. The method according to claim 1, characterized in that The S1 also includes: According to the characteristics of CMIP6 data, the units of CMIP6 meteorological data and historical meteorological data are unified.

4. The method according to claim 1, characterized in that: In S2: The obtained historical CMIP6 data are divided into equal parts.

5. The method according to claim 1, characterized in that In S2: Random forest, recurrent neural network, support vector machine, adaptive booster, decision tree, LASSO regression model, multivariate linear regression and ensemble learning model were trained based on simulated training sets and measured training sets.

6. The method according to claim 1, characterized in that In the S3: Calculate the R between the meteorological data processed by machine learning and the measured test set 2 and RMSE.

7. The method according to claim 2, characterized in that The direct extraction method is as follows: S21: Import multidimensional array database; S22: Search for files matching the pattern; S23: traverse the file and process the data; S24: Combine all results and save.

8. The method according to claim 6, characterized in that The method for comprehensively selecting the best machine learning model corresponding to different types of meteorological data in S3 is as follows: S31: Arrange the historical and CMIP6 daily meteorological data of all test years in order into a column as machine learning input test data; S32: Build a neural network model: S33: compile model; S34: training model; S35: Use the model to make predictions; S36: Calculate evaluation indicators.

9. A CMIP6 data processing device based on historical meteorological data and an optimal machine learning model, specifically an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images