Python-Fortran hybrid programming for deep integration of artificial intelligence and scientific numerical calculation

Abstract

The application provides a method for realizing deep fusion of artificial intelligence and scientific numerical calculation by Python-Fortran hybrid programming, which comprises four parts of environment setting, recompilation of original Fortran module, Fortran-to-Python and Python-to-Fortran interface, Python master control design and operation, and realizes deep fusion of AI and scientific numerical calculation designed by Python. According to the technical scheme of the application, any existing Fortran scientific numerical calculation module can be directly called by artificial intelligence, any artificial intelligence algorithm obtained by machine learning can be directly called by a scientific calculation module, any scientific calculation module coded in Fortran can be directly called by AI designed in Python without re-coding in Python, and any AI algorithm coded in Python can be directly called by a scientific calculation module without re-coding in Fortran.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for achieving deep integration of artificial intelligence and scientific numerical computation through Python-Fortran hybrid programming. Background Technology

[0002] Since the birth of ENIAC (Electronic Numerical Integrator and Computer), the world's first electronic numerical integrator and computer, in the 1950s, various scientific computing methods based on physics principles, especially numerical simulations of Earth's fluids and weather and climate evolution, have developed rapidly. Numerical simulation discretizes the dynamic and thermodynamic equations controlling atmospheric changes onto a grid system and performs time integration of partial differential equations to obtain the state evolution of the environment. The accuracy of numerical simulation is limited by the resolution of the basic fluid units in the discrete computation. Currently, due to the continuous improvement of high-performance computing capabilities, the resolution of numerical simulations is constantly improving and is approaching the kilometer level. The biggest challenge in high-resolution simulation is the need to accurately describe small-scale processes such as turbulence, a problem that artificial intelligence (AI) can solve. On the other hand, based on end-to-end big data fitting, AI, as a data-driven machine learning method, has a powerful ability to extract complex relationships, but it also lacks scientific impetus and has poor comprehensibility and interpretability. AI needs scientific input. The deep integration of AI and scientific computing will inevitably promote the development of both AI and science.

[0003] However, AI is designed and developed using the Python language, while numerical simulations for scientific computing are developed using scientific computing languages, primarily Fortran. Although a Fortran-to-Python compilation protocol has been proposed, deep integration between the two still faces barriers, and systematically mixing Python and Fortran programming to facilitate online integrated computing for both scientific computing and AI has not yet been achieved. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a method for achieving deep integration of artificial intelligence and scientific numerical computation through Python-Fortran hybrid programming.

[0005] This invention is achieved through the following technical solution: a method for deeply integrating artificial intelligence and scientific numerical computation using Python-Fortran hybrid programming, specifically including the following steps:

[0006] Step S1: General f2py protocol framework

[0007] The f2py protocol provides a basic framework for integrating Python and Fortran code to perform hybrid computing.

[0008] Step S2: Python-Fortran hybrid computation process based on f2py

[0009] Step S2.1, Environment Setup

[0010] Starting with the Python environment: Enter "conda create -n f2py python=3.11" to create the environment, then execute "conda activate f2py" to activate the environment and "conda install numpy=1.26.4" to install the required software; for non-standard environment variables, use the "source" command to supplement the configuration.

[0011] Step S2.2: Create a Python-callable Fortran module PCF

[0012] After completing the environment configuration, add the "-fPIC" option to the compiler options used by the original Fortran program to directly perform recompilation; once all the relatively independent Fortran code has been compiled as DLL candidate files, it can be linked with the Python main controller PMC.

[0013] The Fortran subroutine with plug-in functionality is packaged and encapsulated. Then, all PCF modules are integrated into this subroutine, and a signature file named appname.pyf containing a clear structure of the plug-in is generated by the command line.

[0014] The final result will be a complete DLL file; this file is a DDL "shared object" that can be called by Python, in which all subroutines and functions can be called by PMC; all information about the logical structure of appname.cpython-311-x86_64-linux-gnu.so will be recorded in the signature file appname.pyf;

[0015] Step S2.3: Create a Fortran-callable Python function

[0016] Create an interface to convert Python-encoded functions into public subroutines that can be called in Fortran; the interface file contains the function definitions and the contents that are passed from the PMC to the Fortran application as parameters via a plugin; the interface file is compiled together with step S2.1 to generate the signature file appname.pyf;

[0017] Step S3, PMC Architecture

[0018] Python includes three core components: import statements, model integration wires, and the interface for step S2.3; import statements include importing predefined selected Fortran modules from appname or directly importing all modules in appname; import statements include various modules required by PMC, including the MPI module that provides information on the process element identifier PE-id, and all required predefined artificial intelligence algorithms;

[0019] Step S4: Implementation of Python-Fortran hybrid computation

[0020] The parallel computing characteristics of Python-Fortran hybrid computing are demonstrated by using the mpirun command. This run command line is inserted into the existing run_script file of the Fortran model, and the job is submitted to the background job queue using the qsub command to run on a large-scale MPI computing node. Ensure that the environment configuration described in step S2.1 has been completed before the mpirun command is executed.

[0021] Through the above steps, the basic platform for Python-Fortran hybrid modeling and data assimilation, named Hf2pMDA, was completed.

[0022] As a preferred option, several candidate DLL files are generated during the compilation process in step S2.1. The packaging and encapsulation commands will appear as a series of lines similar to the initial creation commands described above, as follows:

[0023] $FC="mpif90" CC="mpicc" CXX="mpicxx" ldshared="mpif90" ldflags="-no-ipo -Wl,-Wl,--export-dynamic" f2py -c appname.pyf app_plugs.F90 libapp1.alibapp2.a ...

[0024] -lincdf - L / ... / software / netcdf3 / lib - L / ... / openmpi / lib --backenddistutils

[0025] The option "-no-ipo" disables cross-module optimization to reduce potential uncertainties during optimization; while the option "-Wl" ensures that all functions associated with the DLL file are called correctly; in this case, the openmpi / lib file needs to be recompiled using the "-fPIC" label.

[0026] As a preferred approach, step S4 demonstrates the parallel computing characteristics of the Python-Fortran hybrid computing using the mpirun command, as detailed below:

[0027] $ mpirun -n xx python appname_main.py>appname.log

[0028] Where “xx” represents the number of parallel computing cores required by the program, and appname.log is the file that records the program's running logs.

[0029] By employing the above technical solutions, this invention has the following beneficial effects compared to existing technologies:

[0030] 1. Any scientific computing module coded in Fortran can be directly called by AIs designed in Python, without the need to recode it in Python;

[0031] 2. Any AI algorithm coded in Python can be directly called by modules that perform scientific computing without needing to be recoded in Fortran.

[0032] Other additional aspects and advantages of the invention will be described in detail in the following description section, or may be learned by practice of the invention. Attached Figure Description

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0034] Figure 1 The design logic flowchart for the Python-Fortran hybrid modeling and data assimilation (Hf2pMDA) system architecture is shown, which is completed in four steps: environment setup, Fortran code recompilation, Fortran and Python interface design, and Python main controller design. The necessary integral calculations are for time integral control of the execution mode.

[0035] Figure 2 This diagram illustrates Hf2pMDA, a modeling platform that deeply integrates AI and scientific numerical simulation through Python-Fortran hybrid programming. In the diagram, f2p represents Fortran to Python, p2f represents Python to Fortran, p2p represents Python to Python, and Fortran scientific simulation refers to scientific computing encoded in Fortran. Detailed Implementation

[0036] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0037] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0038] The following is combined Figures 1 to 2 The present invention provides a detailed description of a method for achieving deep integration of artificial intelligence and scientific numerical computation using Python-Fortran hybrid programming, as described in an embodiment of the present invention.

[0039] like Figure 1 As shown, this invention proposes a method for deeply integrating artificial intelligence and scientific numerical computation through Python-Fortran hybrid programming, specifically including the following steps:

[0040] Step S1: General f2py protocol framework

[0041] The f2py protocol provides a basic framework for integrating Python and Fortran code for hybrid computing. Since Python is an interpreted language, its programs can run directly without pre-compilation, while compiled languages ​​such as Fortran require pre-compilation. The natural way to achieve hybrid computing of Python and Fortran is to import callable Fortran (PCF) modules into the Python master controller (PMC) as callable objects, while allowing Fortran programs in scientific computing to call artificial intelligence algorithms written in Python as subroutines or functions. To this end, the Python-Fortran hybrid modeling and data assimilation (Hf2pMDA) infrastructure designed in this study needs to solve three key problems: (1) The computing environment needs to support the running of both Python and Fortran languages; (2) The Fortran model code needs to be recompiled into PCF modules; (3) An application-oriented interface needs to be designed to connect the PCF modules with the PMC, so that Fortran programs in scientific computing can call artificial intelligence algorithms written in Python.

[0042] Step S2: Python-Fortran hybrid computation process based on f2py

[0043] Step S2.1, Environment Setup

[0044] f2py itself does not include specific environment variable settings. However, to minimize interference from other computational tasks on Python-Fortran modeling and development, we strongly recommend using environment configuration tools such as conda to create an independent runtime environment for hybrid Python-Fortran applications. This environment must include environment variables for both Fortran and Python. The specific steps can begin with the Python environment: enter "conda create -n f2py python=3.11" to create the environment, then execute "condaactivate f2py" to activate the environment and "conda install numpy=1.26.4" to install the required software (corresponding to software version numbers 3.11 and 1.26.4 respectively). For non-standard environment variables that are only needed in specific applications, use the "source" command to supplement the configuration.

[0045] Step S2.2: Create a Python-callable Fortran module PCF

[0046] To enable Python calls to existing Fortran code, a new "-fPIC" tag needs to be added during compilation, making the code a candidate dynamic link library (DLL) with "position-independent" characteristics. After completing the environment configuration, add the "-fPIC" tag to the compiler options used by the original Fortran program, and you can directly perform recompilation. Once all the relatively independent Fortran code has been compiled as DLL candidate files, it can be linked with the Python main controller (PMC).

[0047] To reduce the complexity of the infrastructure between PMC and PCF modules, a Fortran subroutine with plug-in functionality needs to be packaged and encapsulated. This subroutine effectively organizes the PCF modules and provides several plug-ins for PMC. Then, all PCF modules are integrated into this subroutine, and a signature file named appname.pyf containing a clear plug-in structure is generated via the command line. The following command line is used to create the signature file named appname.pyf:

[0048] "$f2py app_plugs.F90 -m appname -h -- overwrite-signatureappname.pyf "

[0049] The `app_plugs.F90` module consists of subroutines or functions that organize the PCF module, serving as a plugin for the PMC. During compilation, several candidate DLL files are generated (e.g., libapp1.a, libapp2.a, ... etc.). The packaging and encapsulation commands will appear as a series of lines similar to the initial creation commands described above, as follows:

[0050] $FC="mpif90" CC="mpicc" CXX="mpicxx" ldshared="mpif90" ldflags="-no-ipo -Wl,-Wl,--export-dynamic" f2py -c appname.pyf app_plugs.F90 libapp1.alibapp2.a ...

[0051] -lincdf - L / ... / software / netcdf3 / lib - L / ... / openmpi / lib --backenddistutils

[0052] The option "-no-ipo" disables cross-module optimization to reduce potential uncertainties during optimization; while the option "-Wl" ensures that all functions associated with the DLL file are called correctly; in this case, the openmpi / lib file needs to be recompiled using the "-fPIC" label.

[0053] Finally, since the above commands typically do not display any error messages, they will eventually generate a complete DLL file named appname.cpython-311-x86_64-linux-gnu.so. This file is a DDL "shared object" that can be called by Python, in which all subroutines and functions can be called by PMC. All information about the logical structure of appname.cpython-311-x86_64-linux-gnu.so is recorded in the signature file appname.pyf, which is used not only as an import name in PMC but also provides efficient guidance for building PMC.

[0054] Step S2.3: Create a Fortran-callable Python function

[0055] To enable Fortran code to call algorithms written in Python, we need an interface (e.g., named callback_python.F90) to create public subroutines (e.g., named python_algorithm1, python_algorithm2, etc.) that convert Python-encoded functions into Fortran-callable subroutines. Simultaneously, the interface file (app_plug.F90) contains function definitions (e.g., named python_foo1, python_foo2, etc.) and parameters passed from PMC to the Fortran application via the plugin. The interface and interface file are compiled together in step S2.1 to generate the signature file appname.pyf. Once the interface is successfully built, all subroutines representing Python algorithms can be called in any Fortran application using the regular "use" statement. Detailed information about FCP applications will be provided when applying Hf2pMDA to a specific NWP model, as described in Section 4.

[0056] Step S3, PMC Architecture

[0057] A Project Management Python controller (PMC, such as a system named appname_main.py) should include three core components: import statements, model integration threads, and the interface for step S2.3. Import statements include "from appnameimport module_list" (importing predefined selected Fortran modules from appname) or directly "import appname" (importing all modules in appname). Import statements include various modules required by the PMC, such as MPI modules that provide process element identifiers (PE-ids), and predefined artificial intelligence algorithms. To minimize the need for Python recoding of existing Fortran models, the PMC typically keeps the code concise during model integration, as Fortran models can easily call Python artificial intelligence algorithms.

[0058] Step S4: Implementation of Python-Fortran hybrid computation

[0059] The parallel computing characteristics of Python-Fortran hybrid computing are demonstrated using the `mpirun` command, as follows:

[0060] $ mpirun -n xx python appname_main.py>appname.log

[0061] Where “xx” represents the number of parallel computing cores required by the program, and appname.log is the file that records the program's running log; insert this run command line into the existing run_script file of the Fortran model, and submit the job to the background job queue using the qsub command to run it on a large-scale MPI computing node; however, it is necessary to ensure that the environment configuration described in step S2.1 has been completed before the mpirun command is executed;

[0062] Through the above steps—environment configuration, creating a Fortran module that can call Python, developing a Python algorithm that can call Fortran, and designing a simple Python main control program—we have completed the following: Figure 2 The Python-Fortran hybrid modeling and data assimilation platform shown is named Hf2pMDA.

[0063] In the description of this invention, the term "a plurality of" refers to two or more. Unless otherwise explicitly defined, the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. The terms "connection," "installation," "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0064] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0065] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for implementing deep fusion of artificial intelligence and scientific numerical calculation by Python-Fortran hybrid programming, characterized in that, Specifically comprising the following steps: Step S1, general f2py protocol framework The f2py protocol provides a basic framework for integrating Python and Fortran code for hybrid computing; Step S2, Python-Fortran hybrid computing process based on f2py Step S2.1, environment setting Start from the Python environment: input "conda create -n f2py python=3.11 ”to create the environment, then execute "conda activate f2py”to activate the environment and "conda install numpy=1.26.4 ”to install the required software; for non-standard environment variables, use the "source”command to supplement the configuration; Step S2.2, create Python callable Fortran module PCF After completing the environment configuration, add the "-fPIC”tag to the compiler options used in the original Fortran program, and then directly execute the recompilation; when all relatively independent Fortran codes are compiled as DLL candidate files, they can be linked with the Python main controller PMC; Packaging the Fortran subroutine with plug-in function, which can organize PCF modules and provide several plug-ins for PMC; then integrate all PCF modules into the subroutine, and generate a signature file named appname.pyf containing the clear structure of the plug-in through the command line; Use the following command line to create a signature file named appname.pyf: "$f2py app_plugs.F90 -m appname -h -- overwrite-signature appname.pyf ” Where app_plugs.F90 is composed of subroutines or functions that organize PCF modules and serve as plug-ins for PMC; the command line for packaging the candidate DLL file generated during compilation as the signature file of appname.pyf is as follows: $FC="mpif90” CC="mpicc” CXX ="mpicxx” ldshared="mpif90” ldflags="-no-ipo-Wl,-Wl,--export-dynamic” f2py -c appname.pyf app_plugs.F90 libapp1.alibapp2.a... -lincdf - L / ... / software / netcdf3 / lib - L / ... / openmpi / lib --backenddistutils The option "-no-ipo" turns off the cross-module optimization function to reduce the uncertainty that may occur during the optimization process; and the option "-Wl" ensures that all functions associated with the DLL file can be correctly called; at this time, the openmpi / lib file needs to be recompiled with the "-fPIC" tag; Finally, since the above command usually does not display any error prompt, a complete DLL file named appname.cpython-311-x86_64-linux-gnu.so is finally generated; this file is a DDL "shared object" that can be called by Python, and all subprograms and functions can be called by PMC; all information about the logical structure of appname.cpython-311-x86_64-linux-gnu.so is recorded in the signature file appname.pyf; Step S2.3, creating a Fortran-callable Python function An interface is created to convert the Python-coded function into a Fortran-callable common subroutine; at the same time, the interface file app_plug.F90 contains the definition of the function and the transmission of the function as a parameter from PMC to the Fortran application through the plug-in; the interface and the interface file are compiled together through step S2.1 to generate the signature file appname.pyf; once the interface is successfully built, all subprograms representing Python algorithms can be called in any Fortran application through the conventional "use" statement; Step S3, PMC architecture Python includes three core components: import statements, model integration harnesses, and interfaces of step S2.3; import statements include importing pre-defined selected Fortran modules from appname or directly importing all modules in appname; import statements include various modules required by PMC, including MPI modules that provide process element identification PE-id information, and defined artificial intelligence algorithms; Step S4, implementation of Python-Fortran hybrid computing The parallel computing characteristics of Python-Fortran hybrid computing are exhibited through the mpirun command; insert this run command line into the existing run_script file of the Fortran model, and submit the job to the background job queue through the qsub command to run it on a large-scale MPI computing node; Ensure that the environment configuration described in step S2.1 has been completed before the mpirun command is executed; Through the above steps, the construction of the Python-Fortran hybrid modeling and data assimilation basic platform is completed, which is named Hf2pMDA.

2. The method for deep integration of artificial intelligence and scientific numerical calculation by Python-Fortran hybrid programming implementation according to claim 1, characterized in that, The step S4 exhibits the parallel computing characteristics of Python-Fortran hybrid computing through the mpirun command, and the specific command is as follows: $ mpirun -n xx python appname_main.py > appname.log where "xx" is the number of parallel computing cores required by the program, and appname.log is the file that records the program running log.

Images