Automatic deployment system and method for numerical climate simulation prediction model

Abstract

The invention discloses an automatic deployment system and method for a numerical climate simulation prediction model, and relates to the technical field of computer software application. The system comprises an initialization and prompt module, a system updating and dependent installation module, a file processing and environment variable setting module, a library file compiling and installation module, a WRF and WPS compiling and debugging module, an Anaconda3 installation and configuration module and a cleaning and ending module which are connected in sequence. According to the method, the integrated automatic full-process deployment of the numerical climate simulation and prediction model can be realized, the deployment efficiency is improved, the environment configuration success rate and the system safety are improved, the numerical climate simulation and prediction model is reduced from a professional level to a workstation level, and county-level weather stations and other basic units can also carry out high-precision numerical prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software application, and in particular to a numerical climate simulation prediction model automatic deployment system and method. BACKGROUND

[0002] The traditional numerical climate simulation prediction model deployment needs to manually install multiple dependent libraries such as GNU compiler, zlib, libpng, jasper, and the version compatibility requirement is strict. The manual deployment takes 1-2 days, and is prone to compilation failure (such as NetCDF and HDF5 version mismatch) due to library conflicts. The traditional deployment method needs to manually modify system configuration files such as bashrc / cshrc, which can easily damage the system and requires a system reinstallation, increasing the user's time cost. The traditional deployment lacks post-processing software, the data readability is poor, and the output data visualization cannot be realized. The above defects are difficult for users to start with, and have high threshold, which is not conducive to the deployment and application of mesoscale weather prediction models.

[0003] Therefore, a numerical climate simulation prediction model automatic deployment system and method are proposed to solve the problems existing in the prior art, which is a problem that needs to be solved by those skilled in the art. SUMMARY

[0004] Therefore, the present application provides a numerical climate simulation prediction model automatic deployment system and method, which can realize the integrated automatic full-process deployment of the numerical climate simulation prediction model, improve the deployment efficiency, and improve the environmental configuration success rate and system security.

[0005] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] A numerical climate simulation prediction model automatic deployment system comprises:

[0007] The initialization and prompting module, the system updating and dependency installation module, the file processing and environment variable setting module, the library file compilation and installation module, the WRF and WPS compilation and debugging module, the Anaconda3 installation and configuration module, and the cleaning and ending module are connected in sequence.

[0008] The system can optionally acquire the operating system type of the installation environment, initialize the installation according to the operating system type, record the deployment start time, and remind the user of the matters needing attention.

[0009] The system can optionally use apt-get update to update the system software package list and install system dependent software.

[0010] The system, optionally, the file processing and environment variable setting module is used for copying and decompressing the installation file, setting the environment variable, adding the environment variable to the.bashrc file, and applying the environment variable.

[0011] The system, optionally, the library file compiling and installing module is used for sequentially compiling and installing the netcdf, zlib, libpng and jasper libraries.

[0012] The system, optionally, the WRF and WPS compiling and debugging module is used for configuring and compiling the WRFV3, cleaning and configuring the WPS, then compiling the WPS, generating the namelist file of a certain area, running the WPS for data preprocessing, generating the met_em file and moving to the WRFV3 / run directory, and running the WRF for meteorological simulation debugging.

[0013] The system, optionally, the Anaconda3 installing and configuring module is used for running the Anaconda3 automatic installation and running script, activating the condabase environment, configuring the Anaconda3 mirror source, updating the conda to the latest version, creating the ncl_stable environment, and configuring the ncl_stable environment to the.bashrc.

[0014] The system, optionally, the cleaning and ending module is used for deleting the installation file of Anaconda3, releasing the memory, recording the installation ending time, prompting the user to restart the terminal, and then restarting the system to complete the deployment when the installation is successful, and simultaneously redirecting the standard output and the standard error output to the log file and the terminal to realize real-time recording of the log when the installation fails.

[0015] A numerical climate simulation prediction model automatic deployment method applied to the numerical climate simulation prediction model automatic deployment system, comprising:

[0016] S1, running the numerical climate simulation prediction model installation script, obtaining the operating system type of the installation environment, performing installation initialization, recording the deployment starting time, and reminding the user of the matters needing attention;

[0017] S2, updating the system software package list, and installing the system dependent software;

[0018] S3, copying and decompressing the installation file, setting and applying the environment variable;

[0019] S4, sequentially compiling and installing the netcdf, zlib, libpng and jasper libraries;

[0020] S5, configure and compile WRFV3, clean up and configure WPS, then compile WPS, compile and install zlib library again, run WPS for data preprocessing, and run WRF for meteorological simulation debugging;

[0021] S6, install Anaconda3, configure the mirror source of Anaconda3, update conda to the latest version, create ncl_stable environment and activate;

[0022] S7, when the installation is successful, delete the installation file of Anaconda3, release the memory, record the end time of the installation, prompt the user to restart the terminal, perform a restart countdown, and then restart the system to complete the deployment; when the installation fails, use the tee command to redirect the standard output and standard error output to the log file and the terminal at the same time, and realize real-time recording of the log.

[0023] The above method, optionally, the numerical climate simulation prediction model installation script in S1, automates manual operations, uses the flow control and command execution function of the shell script, and sequentially executes each step in a predetermined order.

[0024] According to the above technical solution, compared with the prior art, the present application provides a numerical climate simulation prediction model automatic deployment system and method, which has the following beneficial effects: the present application automates the entire installation and configuration process, from system update, dependency installation, file decompression, environment variable setting to compilation and running of each component, which can be automatically completed, and the user only needs to run the script to wait for the installation to be completed, which greatly improves the installation efficiency and reduces the possibility of human error; the present application automatically creates a log file to record all output information during the installation process, including the installation start time, system command execution result, etc., when problems occur, the user can quickly locate the problem by checking the log file, which is convenient for debugging and maintenance; the present application automatically detects whether the environment variable already exists in the.bashrc file, and if it does not exist, it is automatically added, the environment variable is directly set in the script, and the correctness and consistency of the environment variable during the installation process are ensured; the present application combines a visual system to support 10 types of standard meteorological product output, and can be deployed on Linux system and Windows subsystem. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description, and obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creating any creative labor.

[0026] Figure 1A structural diagram of a numerical climate simulation prediction model automatic deployment system provided by the present application is provided;

[0027] Figure 2 A flowchart of a numerical climate simulation prediction model automatic deployment method provided by the present application is provided. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] In the present application, relational terms such as first and second and the like are used only to distinguish one entity or operation from another, and do not necessarily require or imply that these entities or operations exist in any actual relationship or order, the term "comprising", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or equipment including the element.

[0030] Reference Figure 1 The present application discloses a numerical climate simulation prediction model automatic deployment system, comprising:

[0031] The initialization and prompting module, the system updating and dependency installation module, the file processing and environment variable setting module, the library file compiling and installation module, the WRF and WPS compiling and debugging module, the Anaconda3 installation and configuration module and the cleaning and ending module are connected in sequence.

[0032] Further, the initialization and prompting module acquires the operating system type of the installation environment, performs installation initialization according to the operating system type, records the starting time of deployment, and reminds the user of matters needing attention;

[0033] The present application will remind the user several times at the beginning that the installation process may take about 60 minutes, detailed prompt information, so that the user is mentally prepared, and anxiety caused by too long waiting time is avoided;

[0034] It can take hours or even days to manually install the WRF system, depending on the user's experience and system environment. The installation process can take about 60 minutes, and each step is automatically executed, significantly reducing the installation time.

[0035] Further, the system updates and dependency installation module uses apt-get update to update the system package list and install system dependencies.

[0036] The original script uses sudo apt-get for dependency installation, and in other systems such as CentOS, it can be replaced with yum or dnf package manager; for example, for CentOS system, replace sudo apt-get install tcsh m4build-essential gfortran libopenmpi-dev x11-apps ncview expect cmake qt5-default qtcreator graphicsmagick-imagemagick-compat with sudo yum install tcshm4build-essential gfortran libopenmpi-dev x11-apps ncview expect cmake qt5-default qtcreator graphicsmagick-imagemagick-compat (the specific package name may need to be adjusted according to the actual situation); when installing multiple dependent libraries, the original script installs in a fixed order, which can be optimized according to the actual dependency relationship between dependent libraries to reduce installation failures due to unmet dependencies; for example, some libraries may depend on specific versions of other libraries, so install these basic dependent libraries first, then install the upper layer dependent libraries.

[0037] Further, the file processing and environment variable setting module is used to copy and decompress the installation file, set the environment variable, and add the environment variable to the.bashrc file to apply the environment variable.

[0038] Further, the environment variable is automatically configured: automatically detect whether the environment variable already exists in the.bashrc file, and if not, automatically add it to ensure the correctness and consistency of the environment variable.

[0039] Further, the original script defines a series of environment variables manually and adds them to the.bashrc file. The improved version can automatically detect the system environment using a script, intelligently determine and set appropriate environment variables based on the system type (such as different versions of Linux systems) and installed software. For example, by detecting the architecture of the system (32-bit or 64-bit), the -m64 parameter in CFLAGS and FFLAGS can be automatically adjusted.

[0040] Using configuration files to manage environment variables: The settings of environment variables are centralized in a special configuration file, and the script reads this configuration file to set the environment variables. This facilitates the management and modification of environment variables, and when deploying in different environments, only the configuration file needs to be modified, without the need to make extensive changes in the script.

[0041] Further, the library file compilation and installation module is used to compile and install the netcdf, zlib, libpng, and jasper libraries in sequence.

[0042] Further, the WRF and WPS compilation and debugging module is used to configure and compile WRFV3; clean and configure WPS, then compile WPS; generate a namelist file for a certain region; run WPS for data preprocessing, generate a met_em file and move it to the WRFV3 / run directory; run WRF for meteorological simulation debugging.

[0043] Further, the WRF and WPS compilation parameter adjustment: The original script uses fixed compilation options; in different application scenarios, the compilation parameters can be adjusted according to hardware resources and simulation requirements; for example, if the computer is equipped with a GPU, the GPU acceleration-related compilation options can be enabled to improve the efficiency of the model; when compiling WRFV3, the parameters of. / configure can be modified to enable GPU support (the specific parameters depend on the WRF version and GPU type).

[0044] The running parameters of namelist.wps and namelist.input in the original script are fixed settings, which can be dynamically adjusted according to different meteorological simulation requirements; for example, when simulating the meteorological conditions of different regions, the grid resolution (dx, dy), simulation time (start_date, end_date), etc. can be adjusted according to the geographical characteristics of the region.

[0045] Further, the Anaconda3 installation and configuration module is used to run the Anaconda3 automatic installation and running script, activate the conda base environment, configure the Anaconda3 mirror source, update conda to the latest version, create the ncl_stable environment, and configure the ncl_stable environment to the.bashrc.

[0046] Further, the cleaning and ending module is used for deleting the installation file of Anaconda3, releasing the memory, recording the ending time of installation, prompting the user to restart the terminal, and then restarting the system to complete the deployment when the installation is successful; when the installation fails, the standard output and the standard error output are simultaneously redirected to the log file and the terminal to realize real-time recording of the log;

[0047] In the manual installation process, various errors such as command input error and file path error are prone to occur due to a large number of operation steps, and the error rate is high; the present application reduces human intervention and reduces the error rate through automatic execution, and the log recording function can help the user quickly locate and solve problems, further improving the success rate of installation.

[0048] Referring to Figure 2 The numerical climate simulation prediction model automatic deployment method is applied to the numerical climate simulation prediction model automatic deployment system, and comprises the following steps:

[0049] S1, running the numerical climate simulation prediction model installation script, obtaining the operating system type of the installation environment, performing installation initialization, recording the starting time of deployment, and reminding the user of precautions;

[0050] S2, updating the system software package list, and installing system dependent software;

[0051] S3, copying and decompressing the installation file, setting and applying the environment variable;

[0052] S4, sequentially compiling and installing the netcdf, zlib, libpng and jasper libraries;

[0053] S5, configuring and compiling WRFV3, cleaning and configuring WPS, then compiling WPS, again compiling and installing the zlib library, running WPS for data preprocessing, and running WRF for meteorological simulation debugging;

[0054] S6, installing Anaconda3, configuring the mirror source of Anaconda3, updating conda to the latest version, creating an ncl_stable environment and activating it;

[0055] S7, when the installation is successful, deleting the installation file of Anaconda3, releasing the memory, recording the ending time of installation, prompting the user to restart the terminal, performing a restart countdown, and then restarting the system to complete the deployment; when the installation fails, using the tee command to simultaneously redirect the standard output and the standard error output to the log file and the terminal to realize real-time recording of the log.

[0056] The numerical climate simulation prediction model automatic deployment method covers the whole process from system updating, dependency installation, library file compilation to WRF and WPS (WRF Preprocessing System) configuration, compilation and running, and provides a one-stop solution.

[0057] Further, the numerical climate simulation prediction model installation script in S1 automates manual operation, utilizes the process control and command execution function of the shell script, and sequentially executes each step in a predetermined order.

[0058] In one specific embodiment, the traditional manual installation is compared with the installation of the present application:

[0059] Experimental environment: hardware environment: use the same configuration of computer, including CPU (Intel Core i7-8700K), memory (16GB DDR4), hard disk (500GB SSD); software environment: operating system is Ubuntu 18.04LTS.

[0060] Installation content: install WRF3.7 and related dependencies, including tcsh, m4, build-essential, gfortran, libopenmpi-dev and a series of software packages, as well as library files required for configuring and running WRF and WPS (such as netcdf, zlib, libpng, jasper, etc.).

[0061] Traditional manual installation: according to the guidance of WRF official document, manually execute each installation step in turn, including downloading software package, decompression, configuration, compilation and installation, manually setting environment variable, manually editing configuration file;

[0062] Installation of the present application: use the provided 1-WRF3.7AutoInstall.sh script for installation.

[0063] Traditional manual installation: the average installation time of multiple tests is 180 minutes, and frequent manual input of commands is required during the installation process, and due to the tedious steps, operation errors are prone to occur, which may cause the installation to be interrupted, and the user needs to perform part of the steps again, further prolonging the installation time;

[0064] Installation of the present application: the average installation time of multiple tests is 50 minutes, close to the expected 60 minutes of the script, and the script automatically executes each step, reducing the manual waiting and operation time, and greatly improving the installation efficiency.

[0065] The installation success rate of traditional manual installation: in 10 tests, 7 times are successfully installed, and the success rate is 70%, and the main reasons for failure include command input error, dependency library version incompatibility, environment variable setting error, etc.

[0066] The installation success rate of the installation of the application: in 10 tests, 9 times are successfully installed, the success rate is 90%, and the failure is mainly due to network problems, which can be solved by re-running the script, and the automatic installation script reduces human operation errors and improves the installation success rate.

[0067] Traditional manual installation: the configurations after manual installation by different personnel are different, for example, the settings of environment variables may not be completely the same, and the parameters in the configuration file may also be different due to different personal understandings, which may lead to inconsistent results in subsequent model running;

[0068] The installation of the application: the configuration after each installation is completely consistent, because the script adds environment variables and generates configuration files according to fixed settings, whether it is the setting of environment variables or the parameter in the namelist.wps and namelist.input configuration file, it is uniformly generated and modified by the script, which ensures the consistency of the environment and configuration after each installation, and provides a guarantee for the stable operation of the subsequent model.

[0069] Each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The above-described system and system embodiment are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0070] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A numerical climate simulation prediction model automatic deployment system, characterized by: include: The initialization and prompt module, system update and dependency installation module, file processing and environment variable setting module, library file compilation and installation module, WRF and WPS compilation and debugging module, Anaconda3 installation and configuration module and cleanup and end module are connected in sequence.

2. The automatic deployment system of a numerical climate simulation prediction model according to claim 1, characterized in that: The initialization and prompt module obtains the operating system type of the installation environment, performs installation initialization based on the operating system type, records the deployment start time, and reminds users of precautions.

3. The automatic deployment system of a numerical climate simulation prediction model according to claim 1, characterized in that: System update and dependency installation module, use apt-get update to update the system package list and install system dependent software.

4. The automatic deployment system of a numerical climate simulation prediction model according to claim 1, characterized in that: The file processing and environment variable setting module is used to copy and decompress installation files, set environment variables, add environment variables to the .bashrc file, and apply environment variables.

5. The automatic deployment system of numerical climate simulation prediction model according to claim 1, characterized in that: The library file compilation and installation module is used to compile and install the netcdf, zlib, libpng, and jasper libraries in sequence.

6. The automatic deployment system of a numerical climate simulation prediction model according to claim 1, characterized in that: The WRF and WPS compilation and debugging module is used to configure and compile WRFV3; clean and configure WPS, then compile WPS; generate a namelist file for a certain area; run WPS to preprocess the data, generate the met_em file and move it to the WRFV3 / run directory; and run WRF to debug the meteorological simulation.

7. The automatic deployment system of a numerical climate simulation prediction model according to claim 1, characterized in that: The Anaconda3 installation and configuration module is used to run the Anaconda3 automatic installation script, activate the condabase environment, configure the Anaconda3 image source, update conda to the latest version, create the ncl_stable environment, and configure the ncl_stable environment to .bashrc.

8. The automatic deployment system of numerical climate simulation prediction model according to claim 1, characterized in that: The cleanup and end module is used to delete the Anaconda3 installation files, free up memory, record the installation completion time, prompt the user to restart the terminal, and then restart the system to complete the deployment when the installation is successful. When the installation fails, the standard output and standard error output are redirected to the log file and terminal at the same time to achieve real-time log recording.

9. A method for automatically deploying a numerical climate simulation prediction model, applied to the system for automatically deploying a numerical climate simulation prediction model according to any one of claims 1 to 8, comprising: S1. Run the numerical climate simulation prediction model installation script to obtain the operating system type of the installation environment, perform installation initialization, record the deployment start time, and remind users of precautions; S2. Update the system software package list and install system dependent software; S3. Copy and unzip the installation files, set and apply environment variables; S4. Compile and install the netcdf, zlib, libpng, and jasper libraries in sequence; S5. Configure and compile WRFV3, clean and configure WPS, then compile WPS, compile and install the zlib library again, run WPS for data preprocessing, and run WRF for meteorological simulation debugging; S6. Install Anaconda3, configure the Anaconda3 mirror source, update conda to the latest version, create the ncl_stable environment and activate it; S7. If the installation is successful, delete the Anaconda3 installation file, release memory, record the installation completion time, prompt the user to restart the terminal, start the restart countdown, and then restart the system to complete the deployment. If the installation fails, use the tee command to redirect the standard output and standard error output to the log file and terminal at the same time to achieve real-time log recording.

10. The method for automatic deployment of a numerical climate simulation prediction model according to claim 9, characterized in that: The numerical climate simulation prediction model installation script in S1 automates manual operations and uses the process control and command execution functions of shell scripts to execute each step in a predetermined order.

Images