A lifecycle management system and method for scientific computing programs

Abstract

This invention discloses a full lifecycle management system and method for scientific computing programs. The system includes a build environment subsystem and a production environment subsystem. The former provides computer resources for the build process of the scientific computing program throughout its lifecycle, while the latter provides computer resources for the testing and deployment processes. This invention, through a full lifecycle management method for scientific computing programs, operates on corresponding computing resources, encompassing a series of steps including querying, triggering scheduling, build execution, result distribution, and test deployment. It automatically generates the executable file of the scientific computing program and configures its runtime dependencies. Simultaneously, it automatically generates a corresponding description file recording the entire lifecycle process. Furthermore, based on version management of these description files and their sets, it achieves full lifecycle traceability and cross-platform migration and deployment of scientific computing programs in a high-performance computing environment.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to a full lifecycle management system and method for scientific computing programs. Background Technology

[0002] In modern large-scale scientific engineering and big data fields, research on complex systems often requires large-scale simulations and data analysis in high-performance computing (HPC) environments, involving multiple coupled programs and multiple stages. In these studies, initial data often needs to undergo processing through one or more scientific programs to arrive at the final conclusions. As the diversity of scientific programs used increases, the reliability of the scientific conclusions produced decreases, and even problems of unreproducibility arise—the so-called "computational reproducibility crisis." To address this issue, research has focused on metadata related to "scientific computing results," hoping to utilize mature tools to record the scientific data processing process and the origin of data at each stage, thus providing "provenance" information for the entire data flow.

[0003] The traceability mechanism of scientific data processing based on scientific programs is built upon the assumption that consistent results can be obtained in different computing environments. This implicitly assumes that scientific computing programs generating scientific data will behave consistently across different environments. However, in practical applications, the possibility of this assumption being violated cannot be ignored. Many scientific programs are distributed within the scientific community in the form of source code, requiring compilation into executable files before they can be used as modules in a scientific workflow to perform computational tasks. Parameter selection during compilation can potentially alter the actual behavior of a module, and simply recording the data generation process cannot guarantee the traceability of computational results. The same data processing steps may produce inconsistent results due to differences in the compilation process of module programs in different environments. Furthermore, the reproduction of the computing environment is a crucial factor in ensuring the "reproducibility" of computational results. Complete "reproducibility" should involve constructing and executing the data flow based on the traceability information of the computation process to verify the consistency of the results. This requires that a complete traceable system for generating the data flow not only records its execution process but also records the construction and generation process of a rich and comprehensive "module library." In a high-performance computing cluster environment, the lifecycle of scientific software can be divided into five stages: source code, building, testing, deployment, and invocation (see...). Figure 1The source code is provided to HPC environment administrators by developers through source code repositories or other channels; administrators build and test the code, deploying the results to the HPC software repository; users then use shell commands or workflow engines to invoke programs from the repository to complete computational tasks. Therefore, to obtain complete traceability information for scientific data results, it is necessary to incorporate the entire lifecycle construction process of the scientific program itself into the traceability system of the scientific data flow. This allows for the automatic construction of the entire lifecycle of the scientific program from the source code, recording traceability information for each stage of the lifecycle.

[0004] Container-based environment encapsulation and scheduling can be a potential solution, using containers to allocate computing resources and provide a well-defined execution environment for applications. However, this approach disrupts traditional scientific workflows, requiring the introduction of new technologies to assist in implementation, undoubtedly adding extra work and a higher barrier to entry for researchers. Furthermore, introducing containers into complex data analysis within scientific workflows requires careful consideration, as direct methods increase network usage and runtime. Moreover, while containers isolate and encapsulate complex runtime environments, establishing the internal environment still requires addressing the essence of the program code and the original construction process. The lack of recordability and traceability of these fundamental construction processes can lead to inconsistencies in the environments packaged by different containers. Finally, the container itself is a black box, which is undoubtedly a drawback for the operability of data science.

[0005] Therefore, in scientific computing within existing high-performance cluster environments, a universal method is needed that, without altering the original data processing workflow, records traceability information of the entire lifecycle of a scientific program's construction process, and enables the migration and reproduction of the construction environment of a scientific program or scientific program repository across different high-performance cluster environments based on this traceability information. Summary of the Invention

[0006] To address the above issues, this invention provides a broader and more universal solution for a full lifecycle management system for scientific computing programs in high-performance computing environments. This system integrates the entire lifecycle of scientific program construction into the traceability system of scientific data processing workflows, enabling traceability and reproducibility of the scientific program construction process in high-performance computing environments and migrating it across different environments. This effectively solves the problem of missing metadata regarding the traceability of the scientific program construction process in the scientific data processing workflow system, further promoting the improvement of the "computation reproducibility crisis."

[0007] The technical solution of this invention encompasses HPC package management methods, continuous integration concepts, and container virtualization technology. These methods, technologies, and concepts include not only hardware deployment but also software application configuration and secondary development. Therefore, the characteristic of this invention lies in: encapsulating multiple functional modules that implement functions, the process steps for executing these modules, and the hardware system for implementing these processes. A lifecycle management system for scientific computing programs in a high-performance computing environment describes the construction of the hardware system that implements these processes. A lifecycle management method for scientific computing programs in a high-performance computing environment explains in detail how to implement the functions of multiple modules through process steps running on the hardware system, and also describes the implementation functions encapsulated in each functional module.

[0008] This invention provides a full lifecycle management system for scientific computing programs in a high-performance computing environment. The system includes a build environment subsystem and a production environment subsystem. The build environment subsystem is used to complete the construction process of the scientific program, and the production environment subsystem is used to complete the testing and deployment process of the scientific program.

[0009] The build environment subsystem includes a query server, a continuous integration server, an execution server, and local shared storage. The query server, continuous integration server, and execution server provide computer resources. Their operation configurations are based on user requirements, executing a series of queries, triggers, and executions. This automatically generates a one-to-one description file recording the entire lifecycle of the software package, providing complete traceability information of the module's build process. Simultaneously, the executable files and module environment of the software package are initially deployed in the corresponding directory of the local shared storage.

[0010] The query server is configured to provide an environment based on user needs, and includes a query module and a distribution module.

[0011] The query module is used to receive user requests and query whether the scientific software package requested by the user exists.

[0012] The distribution module is configured to distribute and deploy the following execution results in different production environments within the production environment subsystem.

[0013] The continuous integration server is configured to provide an environment for the continuous integration platform, on which a private file management repository and a continuous integration system are deployed.

[0014] The execution server, configured as the execution environment for the software package lifecycle, needs to interact with the continuous integration server. It can consist of multiple physical nodes operating different operating systems, or it can be a virtualization container on the same server encapsulating different operating system environments. It is used to automatically execute a series of processes including software package initialization, configuration parsing, dependency checking, source code acquisition, compilation and installation, and local deployment, and then deploys the results to shared storage.

[0015] The description file, using a key-value tree data structure and stored in YAML or JSON format, records information throughout the entire lifecycle of the scientific software, serving as a basis for module traceability. Its key-value pairs primarily include:

[0016]

[0017] The local shared storage provides shared storage resources to store installed scientific programs and stack environments. It is physically linked to the query server; if the execution server consists of actual physical nodes, it is physically linked to those nodes; if it consists of virtual container nodes, the specific directories on the shared storage are mounted and mapped into the container.

[0018] The production environment subsystem comprises multiple high-performance cluster environments with different operating systems. Each high-performance cluster environment may have a different operating system and includes at least three basic units: a login node, multiple compute nodes, and shared storage. Operations within each high-performance environment are executed concurrently. The operation configuration on these environments is used to deploy the executable files and module description files generated in the build system to the shared storage of the corresponding production system. Furthermore, tasks are submitted to the compute nodes for testing via the login node. Upon successful testing, the deployment process within that environment is completed.

[0019] The high-performance computing environment includes at least three basic units: a login node, multiple computing nodes, and shared storage.

[0020] The login node includes a testing module, which is configured to submit test cases to the computing node. The computing node includes a computing module, which is configured to accept the test cases submitted by the testing module to test whether the software modules synchronized to the shared storage in the construction subsystem can run successfully.

[0021] According to one embodiment of the present invention, the shared storage node includes public storage resources.

[0022] According to another aspect of the present invention, a method for full lifecycle management of scientific computing programs in a high-performance computing environment includes a query step, a triggering and scheduling step, a build and execution step, a result distribution step, and a test and deployment step; wherein,

[0023] Step 1: Query Steps

[0024] The query step described above runs in the environment provided by the query server mentioned earlier. The query step includes a query source, a query method, and a query destination. The query method determines whether the query source exists for the query destination, and the query result is fed back to trigger the construction step.

[0025] Furthermore, the query source is a requirement proposed by a user or maintainer, which includes both local deployment requirements and remote deployment requirements; the query method is a set of query function files or scripts; the query destination is an initialized scientific software description file repository directory stored in shared storage. Furthermore, the scientific software description file repository is a collection of local description files synchronized with a remote file repository.

[0026] Furthermore, the scientific software description file records the entire lifecycle of the scientific software's construction process and corresponds one-to-one with the scientific software.

[0027] Step 2: Scheduling Steps

[0028] The aforementioned triggering and scheduling steps run within the environment provided by the continuous integration server described above. The scheduling trigger uses continuous integration methods, including a version management module and a scheduling module.

[0029] Furthermore, the version management module is a general version management platform; the version management platform can be a version management service directly deployed on the continuous integration server, or it can be a version management service encapsulated in a container.

[0030] Furthermore, establish corresponding project repositories on the version management platform.

[0031] Furthermore, the project repository serves as the working directory, and it also contains branches corresponding to different HPC environments.

[0032] The scheduling module is a general continuous integration system. When the project repository receives a trigger signal, it will trigger the continuous integration system to schedule the corresponding executor to execute the work process according to the agreed workflow process in the working directory.

[0033] Step 3: Build the execution steps

[0034] Furthermore, the aforementioned construction and execution steps run in the environment provided by the execution server described above; the execution server is a collection of different operating system environments, and the scheduling module mentioned above determines which execution environment to select to run the execution steps.

[0035] Furthermore, the construction and execution steps include the following stages in sequence:

[0036] Initialization phase [defInitialize()]: Based on the information from the scientific program in the query module phase, initialize and obtain the specific name, version, and tool connection information of the underlying environment.

[0037] The `[defCheckModule()]` stage checks whether the scientific program exists in the local stack environment, based on the version information and toolchain of the specific scientific program obtained during the initialization phase.

[0038] The configuration parsing phase [defMakeConfig()] aims to generate the configuration file for the scientific program. If the user provides the file, it is used directly; otherwise, a similar file of a similar version is searched for in the system's build repository, and an attempt is made to generate the program's configuration file.

[0039] The Dependency List phase [defDependList()] further lists the module's dependency environment and dependency libraries based on the program configuration file generated in the previous phase, and checks the completeness of the dependency environment.

[0040] The source code retrieval phase [defFetchSource()]: For a specific scientific program object, retrieves the source code of the scientific program and returns its source code location or link.

[0041] The compilation and installation phase [defInstall()]: For a specific scientific program object, it installs its dependent environment and the program itself, generates an executable file, and deploys its module management environment.

[0042] Local deployment phase [defDeploy()]: For a specific scientific program object, the results of the previous phases are gathered in the description file and deployed in the local directory repository of the description file, so that the description file and the scientific program are in one-to-one correspondence, and the entire "pipeline" construction process is recorded.

[0043] Furthermore, the build process will vary depending on the user's needs. For local deployment, the build process only needs to be executed on an execution server identical to the local environment. For remote deployment, the build process will select the execution server and execution procedure based on the query results.

[0044] Furthermore, for the aforementioned off-site deployment requirement, it is necessary to first check whether the corresponding description file exists in the description file. If it exists, the execution server with the same environment as the off-site environment is directly scheduled to execute the build process. If it does not exist, the build process needs to be executed in the executor server with the same environment as the local environment. After the build result is pushed to the description file repository, the execution server with the same environment as the off-site environment is scheduled to execute the build process again, using the repository as the source.

[0045] Step 3 of the construction execution process includes:

[0046] Step 3.1: Perform the initialization of the scientific program;

[0047] Step 3.2: Check if the scientific procedure already exists;

[0048] Step 3.3:

[0049] If the result of step 3.2 is that the function exists, proceed directly to step 3.7: execute the Deploy(fullname) function;

[0050] If the result of step 3.2 does not exist, proceed directly to steps 3.4-3.7:

[0051] Step 3.4: Execute the MakeConfig(fullname) function;

[0052] Step 3.5: Execute the DependList(fullname) function;

[0053] Step 3.6: Execute the FetchSource(fullname) function;

[0054] Step 3.7: Execute the Deploy(fullname) function;

[0055] The `Initialize(name, version, tag)` function initializes the scientific program, where `name` represents the name of the scientific program, `version` represents the version of the scientific program, and `tag` represents the default toolchain dependencies of the scientific program. This initialization function initializes the information of the scientific program retrieved from the query step to obtain the full name of the scientific program and its final supported toolchain versions.

[0056] The CheckModule(fullname) function checks scientific programs. The parameter fullname represents the full name of the scientific program. This function is used to check whether the package exists in the current stack environment.

[0057] MakeConfig(fullname) is a function parameter for generating the configuration file for a scientific program. Fullname represents the full name of the scientific program. This function attempts to generate the configuration file for the scientific program: if the user provides the file, it is used directly; otherwise, it searches for a similar file of a similar version in the system's build repository and attempts to generate the configuration file for the program.

[0058] The DependList(fullname) function is used to resolve the dependencies of scientific programs. The parameter fullname represents the full name of the scientific program. This function resolves the program's dependency libraries based on the configuration file of the scientific program or the configuration file of a similar version of the scientific computing program.

[0059] FetchSource(fullname) is a function that retrieves the source code of a scientific program. The parameter fullname represents the full name of the scientific program. This function retrieves the source code of the scientific program according to the method and address described in the configuration file of the scientific program and returns the final location where the source code is saved.

[0060] The Install(fullname) function is used to install and build scientific programs. The parameter fullname represents the full name of the scientific program. This function is used to complete the installation process of the scientific program and generate the corresponding scientific program environment management files.

[0061] Deploy(fullname) is the function to deploy a scientific program. The parameter fullname represents the full name of the scientific program. This function is used to collect the results information from the above stages, record the results information in the description file of the scientific program, and deploy the description file in the specified file repository.

[0062] The functions mentioned above are encapsulated in a unified toolset to implement the functional functions of each stage. Starting from the essence of scientific programs, the entire life cycle of scientific programs is decomposed according to the functions of different stages. During the process, the corresponding functions are called according to the corresponding needs, so that when the construction and execution steps are completed, the executable file of the scientific program and the environment configuration that the program depends on can be generated. At the same time, a module description file containing the one-to-one correspondence of the program construction process is generated.

[0063] Step 4: Result Distribution Step

[0064] Furthermore, the result distribution step operates in the environment provided by the query server described above, and the distribution step includes the distribution data source, the distribution method, and the distribution destination.

[0065] Furthermore, the distribution data source is the deployment result from step 3, including the installed scientific software programs and their corresponding description files. The distribution method is a set of functional functions or scripts that implement the distribution; the distribution destination is the corresponding installation directory on shared storage in the production environment subsystem.

[0066] Furthermore, the description file is a newly generated result file in step 3, stored in the corresponding location within the description file repository directory mentioned in step 1.

[0067] Furthermore, the installed scientific software program is another result package directory generated in step 3, stored in a predefined installation directory.

[0068] Furthermore, the corresponding installation directory on the shared storage in the production environment subsystem must have the same directory structure as the software package generated in step 3.

[0069] Step 5: Test Deployment Steps

[0070] Furthermore, the test deployment steps are run in various HPC environments within the production environment subsystem. The test deployment steps include a test module and a deployment module.

[0071] Furthermore, the testing module tests the availability of the software packages distributed in the production environment in step 3. The testing module includes existence testing of the scientific software packages, environment loading feasibility testing, and computational functionality feasibility testing. The test cases are user-provided basic cases; depending on the scale of the scientific software's operation, the choice is either direct testing on a small-scale local login node or a large-scale implementation submitted to the computing node through the scheduling system.

[0072] Furthermore, the deployment module is a set of scripts that determine whether to execute based on the results of the test module. If the test is successful, the module code is executed, and the system is deployed as a formally usable scientific software environment; if the test fails, the module is not executed, and the process exits.

[0073] Advantages of this invention:

[0074] 1. The system and method of this invention are applied in the field of science and technology in general high-performance clusters. Targeting the full lifecycle management of scientific programs in complex high-performance clusters, the system and method described in this invention introduce the concept of continuous integration to dynamically track the entire lifecycle process of scientific programs. In addition to generating executable files for deploying scientific programs and module management environments, the results of each stage of the lifecycle are clearly recorded in lightweight description files that correspond one-to-one with the scientific program. This makes up for the lack of metadata in the scientific data analysis process system regarding the traceability of the construction process of the scientific program itself.

[0075] 2. The description files described in the system and method of the present invention introduce the concept of version management, and manage the collection of description files through a repository, thereby transforming the management of complex scientific programs into the management of lightweight file repositories, making the package management of scientific software in the entire production environment traceable.

[0076] 3. The description file repository described in the system and method of the present invention contains and records the construction process and method of scientific programs in each file, and thus the entire repository represents the software stack state of the production environment; using this repository as the source, the installation state of scientific software can be reproduced in a new platform environment, thereby realizing the cross-platform portability of scientific software.

[0077] The system and method of this invention are currently being practically applied in the management of complex physical modules in magnetic confinement fusion integrated modeling. They dynamically generate description files that correspond one-to-one with each physical module, compensating for the lack of traceability information regarding the module construction process in traditional scientific workflow-based integrated modeling systems and promoting the integrity of metadata traceability information for integrated modeling results. Furthermore, since the input and output interfaces of each physical module are fixed, the data interfaces of each physical module are also written into the description file during implementation. This ensures that the description file contains not only construction information but also interface information. Calls to physical modules are transformed from workflow engine calls to description file parsing; the description file acts as an intermediary bridge, indirectly solving the traditional separation between the basic operating environment and program calls. Simultaneously, the complete description file repository containing construction information enables cross-platform portability of the integrated modeling system and its included physical modules. This is of great significance for addressing the growing demand for traceability and reproducibility of scientific data in magnetic confinement fusion integrated simulation research. Furthermore, the versatility of this system and method allows it to be extended to the application of scientific computing programs in any field based on high-performance clusters. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the life cycle stages of the scientific procedure described in this invention.

[0079] Figure 2 This is a schematic diagram of the hardware framework for the entire lifecycle of the scientific computing program of this invention;

[0080] Figure 3 This is a flowchart of the scientific computing program lifecycle management method of the present invention;

[0081] Figure 4 This is a flowchart illustrating the "pipeline-style" construction and execution process implementation method in this invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort and innovative ideas should fall within the scope of protection of this application.

[0083] This invention generally relates to the field of high-performance computing (HPC) environment application technology, and more specifically, to the full lifecycle management of complex scientific program repositories in HPC environments, such as... Figure 1 This invention presents the complete lifecycle of a scientific program. Regarding the complete lifecycle, the lifecycle described in this invention comprises five phases (which can be divided into five stages: source code, build, test, deployment, and invocation), which are considered together as a lifecycle line.

[0084] According to one embodiment of the present invention, a management system for implementing the entire lifecycle of scientific computing programs in a high-performance computing environment is proposed, the overall hardware architecture of which is shown in the figure below. Figure 2 The system includes a build environment subsystem and a production environment subsystem.

[0085] In the build environment subsystem, the user sends a request 1 for the required scientific computing program. The query server checks if the user's request exists in the local shared storage via step 5. If it does not exist, the request goes to the continuous integration server via step 2. The continuous integration server determines the specific requirements and triggers the corresponding execution operations. Then, the request goes to the execution server via step 3. The execution server automatically performs a series of automated processes, including initialization, program module checks, configuration parsing, dependency listing, source code acquisition, compilation and installation, and local deployment. The results are then saved in the corresponding directory in the shared storage via step 4.

[0086] Based on the results generated in steps 2, 3, and 4, the query server distributes the results to a corresponding high-performance cluster within the production environment subsystem via step 6. The execution server can consist of multiple physical nodes operating different operating systems, or it can be a virtualized container encapsulating different operating system environments on the same server. The build environment subsystem can be located in the same network environment as a high-performance cluster within the production environment subsystem. The high-performance cluster environment included in the production environment subsystem can be different clusters depending on different operating systems. The functional modules and implementation methods described below all utilize this hardware resource.

[0087] The flowchart of the method for managing the entire lifecycle of scientific computing programs in a high-performance computing environment, implemented on the aforementioned hardware architecture, is as follows: Figure 3 As shown, the method includes the following steps: query step, triggering step, build execution step, result distribution step, and test deployment step;

[0088] Step 1: Query Steps

[0089] The query step described above runs within the query module of the environment provided by the query server mentioned earlier. The query step includes a query source, a query method, and a query destination. The query method determines whether the query source exists for the query destination, and the query result is then fed back to trigger the construction step.

[0090] Step 2: Scheduling Steps

[0091] The scheduling step described above runs within the environment provided by the continuous integration server. The scheduling principle is based on mainstream continuous integration concepts, and this step includes a version management module and a scheduling module. The scheduling module supports multiple environment branches, each containing a configuration file describing that branch's environment; the scheduling module triggers the corresponding execution environment according to requirements.

[0092] Step 3: Build the execution steps

[0093] The build and execution steps described above run within the environment provided by the execution server mentioned earlier. The execution server is a collection of different physical or virtual operating system environments. The scheduling module in the scheduling step above determines which executor to select to run the execution module. If the executor is a virtual container environment, the local working directory is mounted and mapped to the actual production environment's working directory, ensuring directory consistency between the build and production environments. Different execution environments within the execution module share a locally maintained source code repository and module description file repository.

[0094] Furthermore, the execution module is a series of "pipeline" build and execution processes, encapsulated in a modular toolset. It includes the following stages:

[0095] First: Initialization phase [defInitialize()]: Based on the information from the scientific program in the query module phase, initialize and obtain the specific name, version, and supporting basic environment tool connection information.

[0096] Second: Check the program module stage [defCheckModule()]: Based on the version information and toolchain of the specific scientific program obtained in the initialization stage, check whether the scientific program exists in the local stack environment.

[0097] Third: Configuration parsing phase [defMakeConfig()]: The purpose of this phase is to generate the configuration file for the scientific program; if the user provides the file, it is used directly; if not, a similar file of a similar version is searched for in the system's build repository, and the program's configuration file is generated.

[0098] Fourth: List Dependencies [defDependList()]: Based on the program configuration file generated in the previous stage, further list the module's dependent environment and dependent libraries, and check the completeness of the dependent environment.

[0099] Fifth: Source code retrieval stage [defFetchSource()]: For a specific scientific program object, retrieve the source code of the scientific program and return its source code location or link.

[0100] Sixth: Compilation and installation phase [defInstall()]: For a specific scientific program object, install its dependent environment and the program itself, generate an executable file and deploy its module management environment.

[0101] Seventh: Local Deployment Phase [defDeploy()]: For a specific scientific program object, the results of the previous phases are gathered in the description file and deployed in the local directory repository of the description file. This ensures a one-to-one correspondence between the description file and the scientific program, and records the entire "pipeline" build process, such as... Figure 4 As shown.

[0102] Step 4: Result Distribution Step

[0103] The result distribution step operates in the distribution module within the environment provided by the query server mentioned above, distributing the results of step 3, including the executable file of the installed scientific software program, the software management environment, and the description file corresponding to the scientific program, to the production environment subsystem.

[0104] Step 5: Test Deployment Steps

[0105] The test deployment steps are run in the corresponding high-performance cluster environments within the production environment subsystem. The test deployment steps include a test module and a deployment module.

[0106] Furthermore, the testing module tests the availability of the software packages distributed in the production environment in step 3. Upon successful testing, the deployment module is run to complete the final deployment.

[0107] This invention expands upon traditional HPC package management, incorporating continuous integration principles and container technology into its implementation mechanism. It addresses some drawbacks of container-based migration and reproducibility, enabling full lifecycle management of scientific programs in high-performance computing environments. The execution results of this dynamic process are recorded in lightweight description files, serving as a comprehensive supplement to the metadata of the scientific computing program's construction process. Furthermore, by mapping the collection of description files to a file repository using version control principles, the management of complex scientific programs is transformed into the management of a lightweight file repository, making package management of scientific software traceable throughout the production environment. Moreover, each file contains and records the scientific program's construction process and methods, thus the entire repository represents the software stack state of the production environment. Using this repository as a source, the lifecycle of scientific software can be reproduced in new platform environments, thereby achieving cross-platform portability of scientific software.

[0108] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A full life cycle management system for scientific computing programs, characterized in that, include: The environment subsystem is built to complete the process of querying, triggering, executing, and distributing results for scientific computing programs. The production environment subsystem is used to complete the testing and final deployment of scientific computing programs in different production environments. The aforementioned build environment subsystem includes a query server, a continuous integration server, an execution server, and local shared storage. The query server, continuous integration server, and execution server provide computer resources, and the operation configuration on them takes user requirements as input and executes a series of query, trigger, and execution processes. They automatically generate description files that record the entire lifecycle of the software package to record complete traceability information of the module's build process. At the same time, they initially deploy the executable files and module environment of the software package in the corresponding directory in the local shared storage. The production environment subsystem includes one or more high-performance computing cluster environments with different operating system environments. The operation configuration on it is used to deploy the executable files and module description files generated in the build system to the shared storage of the corresponding production system, and further submit tasks to the computing nodes for testing through the login node; when the test is successful, the deployment process in this environment is realized. The continuous integration server in the build environment subsystem includes a version management module and a scheduling module. The triggering module supports multiple environment branches for different operating systems. Each branch contains a configuration file describing the environment of that branch. The triggering module selects the corresponding environment branch according to the requirements and schedules it to the execution module in the execution step. The description files automatically generated in the build environment subsystem that record the entire lifecycle of scientific software packages adopt a key-value tree data structure and are stored in YAML or JSON file format. The value corresponding to each key is the result of a series of dynamic processes running on the execution server in the build environment subsystem. The collection of multiple description files is managed through a version control system. The description file repository containing the build information process will determine the software stack environment of a high-performance computing cluster. The high-performance cluster environment in the production environment subsystem is either a local high-performance cluster within the same network space as the construction environment subsystem, or multiple geographically distributed high-performance computing clusters with different operating systems.

2. A method for full life cycle management of scientific computing programs, characterized by, Includes the following steps: Step 1: Query Step. Execute the query module provided by the query server to check if the scientific program exists. Step 2, the triggering step, includes a version management module and a triggering module, which triggers and schedules the appropriate environment branch based on the query results; Step 3, Build Execution Steps: This includes multiple executors, each of which is a local physical node or a virtual encapsulated container. Execution modules run on the executors, and these modules can execute a series of predefined "pipeline" build execution processes. The implementation of this "pipeline" process is encapsulated in a modular toolset. Step 4, the result distribution step, includes a distribution module that distributes the build results to the corresponding production environment; Step 5, test deployment step: the results of the temporary distribution are tested in the corresponding production environment and the final deployment is completed, wherein the test module and the deployment module are contained; The "pipeline" process executed in the build execution step of step 3 comprises the following stages: An initialization stage: according to the information of the scientific program in the query module stage, the specific name, version and tool chain information of the supported underlying environment of the scientific program are further refined; A program checking module stage: according to the version information and tool chain of the specific scientific program refined in the initialization stage, it is checked whether the scientific program exists in the local stack environment; A configuration analysis stage: a configuration file of the scientific program is generated; if the user provides the configuration file, it is directly used; if not, a similar file of a similar version is searched from the build repository in the system, and the configuration file of the program is tried to be generated; A dependency listing stage: according to the configuration file of the scientific program generated in the last stage, the dependent environment and dependent library of the scientific program are further listed, and the integrity of the dependent environment is checked; A source code acquisition stage: for a specific scientific program object, the source code of the scientific program is acquired and its source code position or link is returned; A compilation and installation stage: for a specific scientific program object, the dependent environment and the program itself are installed, an executable file is generated and the module management environment thereof is deployed; A local deployment stage: for a specific scientific program object, the results of the previous stages are collected in a description file, and the deployment is made in a local directory repository of the description file, so that the description file and the scientific program are one-to-one corresponding, and the whole "pipeline" construction process is recorded.

Images