A configuration method, device, storage medium and product for a development environment

Abstract

The present application discloses a configuration method, device, storage medium and product for a development environment, and relates to the field of server technology. The method comprises: obtaining customized configuration information of a development environment to be created; obtaining third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, and obtaining hardware constraints corresponding to hardware configuration information; determining target framework configuration information corresponding to the framework based on each third-party library constraint, hardware constraint and framework initial configuration information; searching for target image configuration information from a pre-configured image library based on the hardware configuration information; and constructing an environment configuration file for the development environment based on the target framework configuration information, target third-party library configuration information, target image configuration information and hardware configuration information. The present application can solve the problem of high dependency version conflict rate in complex scenarios of deploying a development environment, which leads to a long development environment construction cycle.

Description

Technical Field

[0001] The present application relates to the field of server technology, and in particular to a configuration method, device, storage medium, and product for a development environment. Background Art

[0002] With the development of artificial intelligence (AI), applied across various fields such as image recognition, natural language processing, and autonomous driving, AI is profoundly changing our lives. As AI technology continues to gain popularity, building an efficient, stable, and easy-to-use AI development environment is becoming increasingly important. A complete AI development environment typically includes hardware infrastructure, software platforms, data resources, and development tools.

[0003] Related technologies typically utilize virtual environment tools to create independent development environments, use cloud service providers to provide virtual machine images pre-installed with mainstream frameworks, and employ static full-build strategies for containerized builds. However, these approaches typically obtain virtual machine images pre-installed with mainstream frameworks through offline component packaging, resulting in versions lagging behind the community. They can only handle Python-level dependencies and are unable to manage other underlying components. Cross-language dependencies require manual intervention, which can lead to compatibility issues across multiple dependencies, high dependency version conflicts in complex scenarios, and lengthy development environment build cycles. Summary of the Invention

[0004] The present application provides a configuration method, device, storage medium and product for a development environment, so as to at least solve the problem in related technologies of high dependency version conflict rate in complex scenarios of deploying development environments, resulting in a long development environment construction cycle.

[0005] The present application provides a method for configuring a development environment, which includes: obtaining a custom configuration file for the development environment to be created; parsing the custom configuration file to obtain configuration information corresponding to the development environment, the configuration information including initial configuration information of a framework for constructing the development environment, hardware configuration information having a mapping relationship with the framework, and third-party library initial configuration information corresponding to each third-party library in at least one third-party library; obtaining third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, and obtaining hardware constraints corresponding to the hardware configuration information, wherein both the third-party library constraints and the hardware constraints are used to constrain the framework.

[0006] Based on each third-party library constraint, hardware constraint, and the framework initial configuration information, determine the target framework configuration information corresponding to the framework; based on the hardware configuration information, search the target image configuration information corresponding to the hardware configuration information from the pre-configured image library; based on the target framework configuration information, target third-party library configuration information, target image configuration information, and hardware configuration information, construct an environment configuration file corresponding to the development environment, and the environment configuration file is used to create the development environment.

[0007] The present application also provides a configuration device for a development environment, including: an acquisition module for obtaining a custom configuration file of the development environment to be created. The acquisition module is also used to parse the custom configuration file to obtain configuration information corresponding to the development environment, the configuration information including the initial configuration information of the framework for building the development environment, the hardware configuration information having a mapping relationship with the framework, and the third-party library initial configuration information corresponding to each third-party library in at least one third-party library. The acquisition module is also used to obtain the third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, as well as to obtain the hardware constraints corresponding to the hardware configuration information. The third-party library constraints and hardware constraints are both used to constrain the framework.

[0008] The processing module is configured to determine the target framework configuration information corresponding to the framework based on each third-party library constraint, hardware constraint, and the framework's initial configuration information. The processing module is also configured to search a preconfigured image library for target image configuration information corresponding to the hardware configuration information based on the hardware configuration information. The processing module is also configured to construct an environment configuration file corresponding to the development environment based on the target framework configuration information, target third-party library configuration information, target image configuration information, and hardware configuration information. The environment configuration file is used to create the development environment.

[0009] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for configuring a development environment when executing the computer program.

[0010] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the configuration method of any of the above-mentioned development environments are implemented.

[0011] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned configuration methods for the development environment when executed by a processor.

[0012] Through this application, since the target framework configuration information corresponding to the framework can be determined based on third-party libraries, hardware, and frameworks, it can be determined that not only Python-level dependencies can be processed, but also other underlying-level components can be managed. In addition, the constructed environment configuration file includes target framework configuration information, target third-party library configuration information, target image configuration information, and hardware configuration information. Therefore, the development environment can be created directly based on the environment configuration file, avoiding the need for manual intervention in third-party library configuration information, hardware configuration information, and framework configuration information, avoiding configuration compatibility issues of multiple dependencies (i.e., third-party libraries, hardware information, and frameworks), avoiding high dependency version conflict rates in complex scenarios, and reducing the time-consuming cycle of developing the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 A flowchart of a method for configuring a development environment provided in an embodiment of the present application;

[0015] Figure 2 A flowchart of another method for configuring a development environment provided in an embodiment of the present application;

[0016] Figure 3 A flowchart of another method for configuring a development environment provided in an embodiment of the present application;

[0017] Figure 4 A flowchart of another method for configuring a development environment provided in an embodiment of the present application;

[0018] Figure 5 A structural block diagram of a configuration device for a development environment provided in an embodiment of the present application;

[0019] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0022] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0023] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the configuration method of the development environment depends, the specific application environment architecture or specific hardware architecture is described herein.

[0024] The embodiments of the present application are applicable to scenarios where AI development environments are constructed. For example, they are suitable for configuration and deployment in large model development environments and big data development environments, thereby achieving software and hardware compatibility and intelligent configuration of different components or dependency packages in complex environments.

[0025] An AI development environment typically includes hardware infrastructure, software platforms, data resources, and development tools. Hardware infrastructure includes high-performance computing servers, graphics accelerators, and mass storage devices. Software platforms include operating systems, programming languages, deep learning frameworks, and data processing tools. Data resources include training and testing datasets, and model parameters. Development tools include code encoders, debuggers, and version control tools.

[0026] In related technologies, the development environment is usually built based on the following three solutions. The first is a virtual environment tool solution, which uses virtual environment tools to achieve package-level dependency isolation by creating an independent Python environment, and relies on manually written environment configuration files (environment.yml) or version management files (requirements.txt). The second is a pre-built cloud image solution, which uses virtual machine images with pre-installed mainstream frameworks provided by cloud service providers, and users can directly start instances for use. It achieves rapid deployment through offline packaging of components such as Compute Unified Device Architecture (CUDA) and PyTorch. The third is a containerized build solution, which defines the layered image build steps through a container image build file (Dockerfile) and uses container technology to encapsulate environmental dependencies.

[0027] Among them, virtual environment tools can only handle Python-level dependencies and cannot manage underlying components such as CUDA drivers and graphics processing unit (GPU) firmware. Cross-language dependencies also require manual intervention. However, because AI involves a wide variety of deep learning frameworks and third-party library versions, manual intervention can lead to compatibility issues between multiple dependency versions, resulting in high dependency version conflicts in complex scenarios. GPU drivers, CUDA versions, and computing power compatibility must be manually matched, making the development environment configuration process time-consuming and error-prone. Furthermore, the environment configuration processes for local development machines, cloud servers, and container clusters are inconsistent, resulting in high migration costs and insufficient cross-platform support for the development environment.

[0028] To address the above technical issues, an embodiment of the present application provides a method for configuring a development environment. This method uses a unified syntax to describe environmental requirements, combines custom configuration files with dependencies between third-party libraries, frameworks, and hardware, determines the third-party library versions, framework versions, and hardware driver versions required for the development environment, and automatically generates a lightweight, cross-platform environment configuration file to build an AI development environment. This approach solves the problems of version lag, rigid hardware adaptation, and inconsistent cross-platform environment configuration processes in traditional pre-built image solutions.

[0029] The configuration method for a development environment provided in an embodiment of the present application is performed by a configuration device for the development environment. The configuration device for the development environment can be any device with computing and communication capabilities. For example, the configuration device for the development environment can be a server or a cloud server. Optionally, the configuration device for the development environment has a display interface and input / output functions, wherein the display interface is used to display a custom configuration file input by the user. The configuration device for the development environment includes a user interface module, an intelligent parsing module, an environment construction module, and a test verification module.

[0030] The embodiment of the present application provides a configuration method for a development environment, such as Figure 1 As shown, Figure 1 A flow chart of a method for configuring a development environment provided in an embodiment of the present application is provided. The method for configuring a development environment includes the following steps:

[0031] S101: Obtain a custom configuration file for the development environment to be created.

[0032] The custom configuration file is a configuration file in the Declarative Environment Description Language (DEDL) format. It can include definitions of basic environment parameters, such as the operating system version, programming language version (Python version), and time zone required for the development environment. It can also include hardware requirements and the target format of the environment configuration file. For example, hardware requirements can include the graphics card model and the graphics card's memory size. The target format can be a container image format (Dockerfile), a management software package file (Kubernetes YAML), or a conda environment file. A custom configuration file can look like this:

[0033] environment:

[0034] model_arch: "Transformer" # Model architecture type

[0035] frameworks:

[0036] - "PyTorch>=2.0.0"# Version constraint

[0037] dependencies:

[0038] - "transformers>=4.35.0"

[0039] - "deepspeed==0.12.0"

[0040] hardware:

[0041] gpu: "XXX" # target GPU model

[0042] cuda: ">=11.8"# CUDA version requirement

[0043] memory: "32GB"# Minimum video memory

[0044] CPU: "X86_64"

[0045] The user interface module of the development environment configuration device includes a pre-configured DEDL-formatted configuration file template. This template is used to convert user or developer input into a custom configuration file. DEDL-formatted custom configuration files allow users and developers to describe their desired development environment configuration requirements (also known as the target state) rather than specific steps. Custom configuration files use a YAML- or JSON-compatible syntax.

[0046] Exemplarily, the configuration device of the development environment responds to the user's input operation, obtains the configuration requirements of the development environment input by the user, converts the configuration requirements based on the configuration file template, and obtains the custom configuration file.

[0047] S102: Parse the custom configuration file to obtain configuration information corresponding to the development environment.

[0048] The configuration information includes initial configuration information of a framework for building a development environment, hardware configuration information having a mapping relationship with the framework, and initial configuration information of a third-party library corresponding to each third-party library in at least one third-party library.

[0049] The initial framework configuration information for the development environment includes the initial configuration information for the deep learning framework required by the user-configured development environment. The deep learning framework is PyTorch. For example, the initial framework configuration information may include the version range of PyTorch.

[0050] Hardware configuration information includes the server's hardware configuration. For example, this information may include the server's graphics card configuration, CUDA driver configuration, and central processing unit (CPU) configuration. Graphics card configuration information may include the graphics card model or graphics memory. The server's hardware and framework have a mapping relationship, also known as a dependency.

[0051] The initial configuration information for third-party libraries is for at least one third-party library that depends on the deep learning framework. This initial configuration information can be a user-configured version range for the third-party library. These third-party libraries can include the Transformers library and the DeepSpeed ​​library. For example, if the Transformers version range is greater than or equal to 4.35.0, the system must install a Transformers library version greater than or equal to 4.35.0; if the DeepSpeed ​​version range is 0.12.0, the system must install the exact DeepSpeed ​​library version 0.12.0.

[0052] In one example, an intelligent parsing module of a configuration device of a development environment parses a custom configuration file to obtain configuration information corresponding to the development environment.

[0053] S103 , obtaining third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, and obtaining hardware constraints corresponding to the hardware configuration information.

[0054] Among them, third-party library constraints and hardware constraints are both used to constrain the framework.

[0055] It is understandable that third-party library constraints and hardware constraints can constrain the version of the deep learning framework.

[0056] The target third-party library configuration information is the configuration information of the third-party library required by the development environment.

[0057] In one example, the intelligent parsing module of the configuration device of the development environment obtains third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, and obtains hardware constraints corresponding to the hardware configuration information.

[0058] In one example, the configuration device of the development environment queries the latest configuration information and dependency information of each third-party library in real time through the Python Package Index (PyPI) or the container image (NVIDIA GPU CLOUD, NGC) package metadata repository of the deep learning software stack.

[0059] The dependency information includes the constraints of the third-party libraries and multiple underlying libraries that each third-party library depends on.

[0060] Furthermore, the configuration device of the development environment determines the target third-party library configuration information based on the latest configuration information of each third-party library and the initial configuration information of the third-party library that have been queried.

[0061] S104 , determining target framework configuration information corresponding to the framework based on each third-party library constraint, hardware constraint, and the framework initial configuration information.

[0062] The target framework configuration information is the configuration information of the deep learning framework required by the development environment.

[0063] In one example, the configuration device of the development environment can build a dependency graph based on each third-party library constraint, hardware constraint, and framework initial configuration information; based on the dependency graph, determine the target framework configuration information corresponding to the framework.

[0064] Among them, the dependency graph can be shown as follows:

[0065] PyTorch (>=2.0.0) #Pytorch version in the framework's initial configuration information

[0066] ├── transformers (>=4.35.0)

[0067] │└── Pytorch (>=2.1.0)# transformers 4.35.0 depends on Pytorch>= 2.1.0

[0068] └── deepspeed (==0.12.0)

[0069] └── PyTorch (specific version) #deepspeed12.0 depends on a specific PyTorch version

[0070] CUDA (>=11.8)

[0071] ├── PyTorch (>=2.0.0) CUDA-compatible PyTorch version

[0072] └── deepspeed (0.12.0) CUDA-compatible version of deepspeed

[0073] GPU (specific model)

[0074] └── CUDA (>=2.0.0) CUDA version compatible with the GPU

[0075] Among them, the dependency graph can be used to verify the version compatibility between third-party libraries, hardware, and frameworks.

[0076] In one example, a configuration device of a development environment traverses a dependency graph to search for a combination of version requirements of third-party libraries, hardware, and frameworks, and determines the optimal versions that meet the requirements of the third-party libraries, hardware, and frameworks.

[0077] For example, the constraint condition of the third-party library Transformers is Pytorch>=2.1.0, the hardware CUDA constraint condition is Pytorch>=2.0.0, and the constraint condition in the initial configuration information of the framework is Pytorch>=2.0.0; according to the Transformers constraint: Pytorch>=2.1.0, the CUDA constraint: Pytorch>=2.0.0, and the framework constraint: Pytorch>=2.0.0, the merged effective constraint is Pytorch>=2.1.0. Therefore, the configuration device of the development environment traverses the dependency graph to find the version requirement combination of the third-party library, hardware, and framework, and determines that the optimal version that meets the framework is Pytorch=2.1.0.

[0078] In one example, a configuration device of a development environment determines target framework configuration information corresponding to the framework based on each third-party library constraint, hardware constraint, and framework initial configuration information.

[0079] S105 , searching for target image configuration information corresponding to the hardware configuration information from a pre-configured image library according to the hardware configuration information.

[0080] The pre-configured image library is the NGC image library.

[0081] The target image configuration information may be a target image label or name.

[0082] Exemplarily, the configuration device of the development environment searches for target image configuration information corresponding to the hardware configuration information from a pre-configured image library according to the hardware configuration information.

[0083] S106 , constructing an environment configuration file corresponding to the development environment according to the target framework configuration information, the target third-party library configuration information, the target image configuration information, and the hardware configuration information.

[0084] Among them, the environment configuration file is used to create a development environment.

[0085] In one example, a configuration device of a development environment combines target framework configuration information, target third-party library configuration information, target image configuration information, and hardware configuration information to form an environment configuration file corresponding to the development environment.

[0086] Based on the above Figure 1 The method can obtain configuration information corresponding to the development environment based on the custom configuration file of the development environment to be created; and determine the target framework configuration information corresponding to the framework based on the third-party library constraints corresponding to each third-party library initial configuration information, the hardware constraints corresponding to the hardware configuration information, and the framework initial configuration information; according to the hardware configuration information, search for the target image configuration information corresponding to the hardware configuration information from the preconfigured image library; and construct an environment configuration file corresponding to the development environment based on the target framework configuration information, the target third-party library configuration information, the target image configuration information, and the hardware configuration information.

[0087] Since the target framework configuration information corresponding to the framework can be determined based on third-party libraries, hardware, and frameworks, it can be determined that not only Python-level dependencies can be handled, but other underlying-level components can also be managed. In addition, the constructed environment configuration file includes target framework configuration information, target third-party library configuration information, target image configuration information, and hardware configuration information. Therefore, the development environment can be created directly based on the environment configuration file, avoiding the need for manual intervention in third-party library configuration information, hardware configuration information, and framework configuration information, avoiding configuration compatibility issues of multiple dependencies (i.e., third-party libraries, hardware information, and frameworks), avoiding high dependency version conflicts in complex scenarios, and reducing the time-consuming cycle of developing the development environment.

[0088] In an optional example, based on the above embodiment, as described above, the custom configuration file is parsed to obtain the configuration information corresponding to the development environment. The following method steps are specifically referred to. Figure 2As shown, Figure 2 A flow chart of another method for configuring a development environment provided in an embodiment of the present application includes:

[0089] S201: Searching for framework initial configuration information corresponding to a first key field in a custom configuration file.

[0090] The framework initial configuration information indicates the configuration information of the framework supporting the model architecture. The model architecture may be a Transformer model architecture. The framework initial configuration information includes the initial framework version constraints of the framework. The first key field may be frameworks.

[0091] S202: Search the custom configuration file for at least one third-party library initial configuration information corresponding to the second key field.

[0092] The initial configuration information of each third-party library includes the version information of each third-party library that has a mapping relationship with the framework. The second key field may be dependencies.

[0093] S203: Search the custom configuration file for hardware configuration information corresponding to the third key field.

[0094] The hardware configuration information includes initial architecture version information, which is the CUDA driver version information that has a dependency relationship with the graphics card. The third key field may be hardware.

[0095] In an optional example, based on the above embodiment, as described above, the third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information are obtained, and the hardware constraints corresponding to the hardware configuration information are obtained. The following method steps are specifically referred to. Figure 3 As shown, Figure 3 A flow chart of another method for configuring a development environment provided in an embodiment of the present application includes:

[0096] S301, query the latest version information of the i-th third-party library.

[0097] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0098] S302: Check whether the latest version information is consistent with the initial configuration information of the i-th third-party library.

[0099] S303: When the latest version information is inconsistent with the initial configuration information of the i-th third-party library, the i-th third-party library constraint corresponding to the i-th third-party library initial configuration information is determined based on the latest version information, and the i-th latest version information is determined as the target third-party library configuration information corresponding to the i-th third-party library.

[0100] Wherein, i is a positive integer.

[0101] It can be understood that the target third-party library configuration information is the latest version information of the third-party library.

[0102] For example, taking the i-th third-party library as transformers and the latest version information as version 4.35.0, the framework version range that the 4.35.0 version of transformers depends on is Pytorch>=2.1.0, and the i-th third-party library initial configuration information is Pytorch>=2.0.0, the configuration device of the development environment determines that the i-th third-party library constraint corresponding to the i-th third-party library initial configuration information is Pytorch>=2.1.0 based on the latest version information, and determines the i-th latest version information as the target third-party library configuration information corresponding to the i-th third-party library as version 4.35.0.

[0103] Optionally, the configuration device of the development environment detects that the latest version information is consistent with the initial configuration information of the i-th third-party library, and based on the i-th third-party library initial configuration, determines the i-th third-party library constraint corresponding to the i-th third-party library initial configuration information.

[0104] The i-th third-party library constraint corresponding to the i-th third-party library initial configuration information is the version range of the framework required by the i-th third-party library of the latest version information.

[0105] Understandably, the latest version information can ensure that the development environment always uses the latest version of the community's third-party library features and security patches.

[0106] In some optional implementations, the configuration device of the development environment extracts initial architecture version information of the computing unified device architecture from the hardware configuration information; and determines hardware constraints corresponding to the initial architecture version information based on the initial architecture version information.

[0107] The hardware constraint corresponding to the initial architecture version information is the version range of the framework required for calculating the unified device architecture of the latest version information.

[0108] In an optional example, based on the above embodiment, as described above, according to each third-party library constraint, hardware constraint, and framework initial configuration information, the target framework configuration information corresponding to the framework is determined as follows. For details, see Figure 4 Shown, including:

[0109] S401 , detecting whether there is a conflict between each third-party library constraint and the initial framework version constraint.

[0110] Each third-party library constraint corresponds to the first framework version information, and the initial framework version constraint corresponds to the second framework version information.

[0111] In one example, when the first framework version information is greater than or equal to the second framework version information, the configuration device of the development environment determines that there is no conflict between each third-party library constraint and the initial framework version constraint; when the first framework version information is less than the second framework version information, it is determined that there is a conflict between each third-party library constraint and the initial framework version constraint.

[0112] S402 : When there is no conflict between all third-party library constraints and the initial framework version constraints, determine the first highest framework version information as candidate target framework configuration information according to all third-party library constraints and the initial framework version constraints.

[0113] For example, taking the framework version information corresponding to the first third-party library constraint as Pytorch>=2.1.0, the framework version information corresponding to the second third-party library constraint as Pytorch>=1.9.0, and the framework version information corresponding to the initial framework version constraint as Pytorch>=2.0.0 as an example, when there is no conflict between all third-party library constraints and the initial framework version constraint, then according to all third-party library constraints Pytorch>=2.1.0, Pytorch>=1.9.0 and the initial framework version constraint Pytorch>=2.0.0, the first highest framework version information is determined to be Pytorch>=2.1.0 as the candidate target framework configuration information.

[0114] S403: Detect whether there is a conflict between the candidate target framework configuration information and the framework version constraint in the hardware constraint condition.

[0115] For example, taking the framework version information corresponding to the hardware constraint condition as Pytorch>=2.1.1 and the candidate target framework configuration information as Pytorch>=2.1.0 as an example, the configuration device of the development environment determines that there is no conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint condition.

[0116] S404: When it is determined that there is no conflict between the candidate target framework configuration information and the framework version constraint in the hardware constraint condition, the second highest framework version information is determined as the target framework configuration information according to the candidate target framework configuration information and the hardware constraint condition.

[0117] For example, taking the framework version information corresponding to the hardware constraint condition as Pytorch>=2.1.1 and the candidate target framework configuration information as Pytorch>=2.1.0 as an example, the configuration device of the development environment determines that the second highest framework version range is Pytorch>=2.1.1 according to the candidate target framework configuration information Pytorch>=2.1.1 and the hardware constraint condition Pytorch>=2.1.0, and based on the second highest framework version range, any version number in the second highest framework version range is used as the target framework configuration information.

[0118] S405 , when it is determined that there is a conflict between at least one third-party library constraint and the initial framework version constraint, or when it is determined that there is a conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint, an alarm message is generated.

[0119] The warning information is used to instruct the user to modify the custom configuration file.

[0120] In one example, when there is a conflict between at least one third-party library constraint and the initial framework version constraint, the configuration device of the development environment generates alarm information according to the version information of the at least one third-party library and the at least one third-party library constraint.

[0121] For example, taking at least one third-party library as transformers and the version information of transformers>=4.35.0, the transformers constraint is Pytorch>=2.1.0, and the initial framework version constraint is Pytorch==2.0.0 as an example, it is obvious that there is a conflict between at least one third-party library constraint and the initial framework version constraint. According to at least one third-party library constraint Pytorch>=2.1.0, an alarm information is generated to recommend upgrading the initial framework configuration information in the custom configuration file to Pytorch>=2.1.0, or downgrading the transformers version information to <4.35.0.

[0122] In another example, when it is determined that there is a conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint condition, the configuration device of the development environment generates an alarm message according to the hardware constraint condition.

[0123] For example, taking the candidate target framework configuration information as Pytorch>=2.1.0 and the hardware constraint condition as Pytorch<2.1.0 as an example, it is obvious that there is a conflict between the version constraint on the framework in the candidate target framework configuration information and the hardware constraint condition. According to the hardware constraint condition Pytorch<2.1.0, an alarm information is generated to recommend downgrading the initial configuration information version information of the framework to Pytorch<2.1.0.

[0124] In some optional embodiments, the configuration information also includes the target format of the environment configuration file. The hardware configuration information also includes the graphics card model of the server.

[0125] In one example, when the target format is a management software package file format, the configuration device of the development environment determines the target image configuration information corresponding to the graphics card model and the initial architecture version information from a preconfigured image library based on the graphics card model and the initial architecture version information.

[0126] Exemplarily, when the graphics card model is the first model and the initial architecture version information CUDA version is greater than or equal to 11.8, the target image configuration information corresponding to the first model and the initial architecture version information is determined to be nvcr.io / nvidia / pytorch:22.10 - py3 from the pre-configured image library NGC.

[0127] In another example, when the target format is a container image format, the configuration device of the development environment determines the toolkit version of the Compute Unified Device Architecture Toolkit based on the graphics card model and the initial architecture version information; and determines the target image configuration information based on the toolkit version and the target framework configuration information.

[0128] As you can understand, NGC has multiple PyTorch-related container images, each pre-installed with a specific version of PyTorch and the CUDA toolkit. NGC provides different versions of the CUDA toolkit, and the system will select the appropriate CUDA version to install based on the graphics card model and initial architecture version, or use the image pre-installed with that version.

[0129] In this embodiment of the present application, the environment configuration file includes system dependency information corresponding to the server's operating system version. The same environment configuration file can be generated in various formats, such as Dockerfile, conda environment files, and Kubernetes YAML, to adapt to different deployment scenarios. Furthermore, this embodiment of the present application supports the generation of fixed-version configuration files to ensure consistency between development and production environments.

[0130] In some optional implementations, the configuration device of the development environment generates a container image build file corresponding to the environment configuration file based on the environment configuration file.

[0131] The container image build file is used to deploy network applications across environments. This file can be a Dockerfile. A Dockerfile is used to build a Docker image and contains a series of instructions that define the image's content and configuration.

[0132] In one example, a configuration device of a development environment obtains a target image corresponding to the target image configuration information based on the target image configuration information included in the environment configuration file; adds system dependency instructions corresponding to the system dependency information to the target image, and installs a framework corresponding to the target framework configuration information and at least one third-party library corresponding to the target third-party library configuration information, as well as sets a working directory and copies the project files of the project to be executed in the development environment to generate a container image build file.

[0133] Exemplarily, the configuration device of the development environment obtains the target image nvcr.io / nvidia / pytorch:22.10-py3 corresponding to the target image configuration information according to the target image configuration information included in the environment configuration file; adds the system dependency instruction apt-get or yum command corresponding to the system dependency information in the target image, and uses the pip command to install the framework corresponding to the target framework configuration information and at least one third-party library corresponding to the target third-party library configuration information, and sets the working directory and copies the project files of the project to be executed in the development environment to the target image to generate a container image build file.

[0134] Optionally, if the Dockerfile needs to define a startup command, the configuration device of the development environment can add the corresponding instructions in the Dockerfile. An example of generating a container image build file is as follows:

[0135] # Select the appropriate base image based on GPU model and CUDA version

[0136] FROM nvcr.io / nvidia / pytorch:22.10-py3

[0137] # Set the working directory

[0138] WORKDIR / app

[0139] # Install system dependencies

[0140] RUN apt-get update&&apt-get install -y \

[0141] build-essential \

[0142] &&rm -rf / var / lib / apt / lists / *

[0143] # Install Python dependencies

[0144] RUN pip install --no-cache-dir \

[0145] torch>=2.0,<2.2 \

[0146] transformers>=4.35.0 \

[0147] deepspeed==0.12.0

[0148] # Copy the project files to the working directory

[0149] COPY . / app

[0150] # Define the startup command

[0151] CMD ["python", "main.py"]

[0152] Optionally, the development environment's configuration device can optimize images through redundant stripping, layered construction, and on-demand tailoring. For example, the development environment's configuration device can analyze dependencies and automatically exclude non-essential toolchains to minimize image size; alternatively, the development environment's configuration device can retain only runtime dependencies and remove compilation tools and intermediate files; alternatively, the development environment's configuration device can support user-defined tailoring rules, retaining only the PyTorch core library and optionally removing documentation and test cases.

[0153] The development environment configuration device installs and deploys the user's required development environment step by step according to the environment configuration file. The environment deployment is successful. The environment configuration file is used when the user migrates and redeploys the development environment, directly using the fixed version of the configuration file to ensure the environment is consistent.

[0154] Furthermore, after the development environment is successfully deployed, the configuration device of the development environment builds a first container image based on the container image build file; after starting the container based on the first container image, a test script is created; the test script is added to the container to build a second container image, and the container test graphics card is run to output the test results; the actual environment configuration file is compared with the configuration information; if the actual environment configuration file is consistent with the configuration information, it is determined that the development environment configuration is successful.

[0155] The test script is used to check the environment configuration of the development environment. The test results include the actual environment configuration file of the development environment.

[0156] For example, using a Docker container as an example, a test script is run inside the Docker container to verify the Python library version, CUDA version, and GPU availability. This is done by importing the corresponding third-party library and obtaining its version information, for example, by executing the nvcc --version command using the subprocess module to verify the CUDA version. For example, a test script used to check the CUDA version of the development environment is shown below:

[0157] try:

[0158] result = subprocess.run(['nvcc', '--version'], capture_output=True,text=True)

[0159] cuda_output = result.stdout

[0160] if 'release 11.8' not in cuda_output:

[0161] print("CUDA version verification failed, current version information: {cuda_output}, required version:>=11.8")

[0162] else:

[0163] print("CUDA version verification passed")

[0164] except FileNotFoundError:

[0165] print("nvcc not found, unable to verify CUDA version")

[0166] It can be understood that the configuration information defined in EDEL is compared to detect whether the development environment meets user requirements.

[0167] It is understandable that the environment configuration file generated by the embodiment of the present application only requires the user to declare or configure a custom file, and there is no need to pay attention to how to install and deploy it. In addition, the present application can automatically generate a complete development environment configuration file based on the mapping relationship or dependency relationship between the framework and third-party libraries and hardware information. The same environment configuration file generates scripts in multiple formats under different environments to adapt to different deployment scenarios. It also supports the generation of fixed versions of configuration files to ensure consistency under different deployment environments.

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0169] The embodiment of the present application also provides a configuration device for a development environment, such as Figure 5 As shown, Figure 5 This is a structural block diagram of a configuration device for a development environment provided in an embodiment of the present application; the configuration device for the development environment includes: an acquisition module 501, which is used to obtain a custom configuration file for the development environment to be created.

[0170] The acquisition module 501 is also used to parse the custom configuration file to obtain configuration information corresponding to the development environment. The configuration information includes the initial configuration information of the framework for building the development environment, the hardware configuration information that has a mapping relationship with the framework, and the initial configuration information of the third-party library corresponding to each third-party library in at least one third-party library.

[0171] The acquisition module 501 is also used to obtain the third-party library constraints and target third-party library configuration information corresponding to each third-party library initial configuration information, and to obtain the hardware constraints corresponding to the hardware configuration information. The third-party library constraints and hardware constraints are both used to constrain the framework.

[0172] The processing module 502 is used to determine the target framework configuration information corresponding to the framework according to each third-party library constraint condition, hardware constraint condition, and the initial configuration information of the framework.

[0173] The processing module 502 is further configured to search the pre-configured image library for target image configuration information corresponding to the hardware configuration information according to the hardware configuration information.

[0174] The processing module 502 is further configured to construct an environment configuration file corresponding to the development environment based on the target framework configuration information, the target third-party library configuration information, the target image configuration information, and the hardware configuration information. The environment configuration file is used to create the development environment.

[0175] In some optional embodiments, the acquisition module 501 is specifically used to search for framework initial configuration information corresponding to the first key field from the custom configuration file, where the framework initial configuration information is used to indicate configuration information of the framework that supports the model architecture; search for at least one third-party library initial configuration information corresponding to the second key field from the custom configuration file; and search for hardware configuration information corresponding to the third key field from the custom configuration file.

[0176] In some optional implementations, each third-party library initial configuration information includes version information of each third-party library that has a mapping relationship with the framework; the acquisition module 501 is further specifically used to query the latest version information of the i-th third-party library, where i is a positive integer; detect whether the latest version information is consistent with the i-th third-party library initial configuration information; when the latest version information is inconsistent with the i-th third-party library initial configuration information, determine the i-th third-party library constraint corresponding to the i-th third-party library initial configuration information based on the latest version information, and determine the i-th latest version information as the target third-party library configuration information corresponding to the i-th third-party library.

[0177] In some optional implementations, the acquisition module 501 is further specifically configured to extract initial architecture version information of the unified device architecture from the hardware configuration information; and determine hardware constraints corresponding to the initial architecture version information based on the initial architecture version information.

[0178] In some optional implementations, the framework initial configuration information includes the framework's initial framework version constraints; the processing module 502 is specifically used to detect whether there is a conflict between each third-party library constraint and the initial framework version constraint; when there is no conflict between all third-party library constraints and the initial framework version constraint, the first highest framework version information is determined as the candidate target framework configuration information based on all third-party library constraints and the initial framework version constraint; detect whether there is a conflict between the candidate target framework configuration information and the version constraints on the framework in the hardware constraints; when it is determined that there is no conflict between the candidate target framework configuration information and the version constraints on the framework in the hardware constraints, the second highest framework version information is determined as the target framework configuration information based on the candidate target framework configuration information and the hardware constraints.

[0179] In some optional embodiments, the processing module 502 is further specifically configured to generate an alarm message when it is determined that there is a conflict between at least one third-party library constraint and the initial framework version constraint, or when it is determined that there is a conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint, and the alarm message is used to instruct the modification of the custom configuration file.

[0180] In some optional implementations, the configuration information also includes a target format of the environment configuration file; the hardware configuration information also includes a graphics card model of the server. When the target format is a management software package file format, the processing module 502 is further specifically used to determine, from a preconfigured image library, target image configuration information corresponding to the graphics card model and the initial architecture version information based on the graphics card model and the initial architecture version information.

[0181] In some optional embodiments, when the target format is a container image format, the processing module 502 is further specifically used to determine the toolkit version of the computing unified device architecture toolkit based on the graphics card model and the initial architecture version information; and determine the target image configuration information based on the toolkit version and the target framework configuration information.

[0182] In some optional implementations, the custom configuration file is a configuration file in a declarative environment description language format.

[0183] In some optional implementations, the processing module 502 is further configured to generate a container image build file corresponding to the environment configuration file based on the environment configuration file, and the container image build file is used to deploy network applications across environments.

[0184] In some optional implementations, the environment configuration file includes system dependency information corresponding to the operating system version of the server; the processing module 502 is further specifically used to obtain a target image corresponding to the target image configuration information based on the target image configuration information included in the environment configuration file; add system dependency instructions corresponding to the system dependency information to the target image, and install the framework corresponding to the target framework configuration information and at least one third-party library corresponding to the target third-party library configuration information, as well as set the working directory and copy the project files of the project to be executed in the development environment to generate a container image build file.

[0185] In some optional embodiments, the processing module 502 is further used to build a first container image based on the container image build file; after starting the container based on the first container image, create a test script, and the test script is used to check the environment configuration of the development environment; add the test script to the container to build a second container image, and run the container test graphics card to output the test results, which include the actual environment configuration file of the development environment; compare the actual environment configuration file with the configuration information; if the actual environment configuration file is consistent with the configuration information, it is determined that the development environment configuration is successful.

[0186] For the description of the features in the embodiment corresponding to the configuration device of the development environment, please refer to the relevant description of the embodiment corresponding to the configuration method of the development environment, and no further details will be given here.

[0187] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, Figure 6 The hardware structure diagram of an electronic device provided in an embodiment of the present application is shown. The electronic device includes a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to perform the steps of any of the above-mentioned configuration method embodiments of the development environment.

[0188] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned development environment configuration method embodiments when running.

[0189] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0190] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned configuration method embodiments of the development environment are implemented.

[0191] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned configuration method embodiments of the development environment are implemented.

[0192] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] The above is a detailed introduction to the configuration method, device, computer equipment and storage medium of a development environment provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A method for configuring a development environment, characterized in that: The method comprises: Get the custom configuration file for the development environment to be created; Parsing the custom configuration file to obtain configuration information corresponding to the development environment, the configuration information including initial configuration information of a framework for building the development environment, hardware configuration information having a mapping relationship with the framework, and initial configuration information of a third-party library corresponding to each of at least one third-party library; Obtaining third-party library constraints and target third-party library configuration information corresponding to each of the third-party library initial configuration information, and obtaining hardware constraints corresponding to the hardware configuration information, wherein both the third-party library constraints and the hardware constraints are used to constrain the framework; Determining target framework configuration information corresponding to the framework according to each of the third-party library constraints, the hardware constraints, and the framework initial configuration information; According to the hardware configuration information, searching for target image configuration information corresponding to the hardware configuration information from a pre-configured image library; An environment configuration file corresponding to the development environment is constructed based on the target framework configuration information, the target third-party library configuration information, the target image configuration information, and the hardware configuration information. The environment configuration file is used to create the development environment.

2. The method according to claim 1, characterized in that The step of parsing the custom configuration file to obtain configuration information corresponding to the development environment includes: Searching for the framework initial configuration information corresponding to the first key field from the custom configuration file; Searching for at least one piece of the third-party library initial configuration information corresponding to the second key field from the custom configuration file; The hardware configuration information corresponding to the third key field is searched from the custom configuration file.

3. The method according to claim 1, characterized in that Each of the third-party library initial configuration information includes version information of each of the third-party libraries having a mapping relationship with the framework; The obtaining of the third-party library constraint condition and the target third-party library configuration information corresponding to each of the third-party library initial configuration information includes: Query the latest version information of the i-th third-party library, where i is a positive integer; Check whether the latest version information is consistent with the initial configuration information of the i-th third-party library; When the latest version information is inconsistent with the i-th initial configuration information of the third-party library, the i-th third-party library constraint corresponding to the i-th initial configuration information of the third-party library is determined based on the latest version information, and the i-th latest version information is determined as the target third-party library configuration information corresponding to the i-th third-party library.

4. The method according to claim 3, characterized in that The obtaining of the hardware constraint conditions corresponding to the hardware configuration information includes: Extracting initial architecture version information of a computing unified device architecture from the hardware configuration information; The hardware constraint condition corresponding to the initial architecture version information is determined according to the initial architecture version information.

5. The method according to claim 4, characterized in that The framework initial configuration information includes an initial framework version constraint of the framework; Determining target framework configuration information corresponding to the framework based on each of the third-party library constraints, the hardware constraints, and the framework initial configuration information includes: Detecting whether there is a conflict between each of the third-party library constraints and the initial framework version constraints; When there is no conflict between all third-party library constraints and the initial framework version constraints, determining the first highest framework version information as the candidate target framework configuration information according to all the third-party library constraints and the initial framework version constraints; Detecting whether there is a conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint condition; When it is determined that there is no conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint condition, the second highest framework version information is determined as the target framework configuration information according to the candidate target framework configuration information and the hardware constraint condition.

6. The method according to claim 5, characterized in that The method further comprises: When it is determined that there is a conflict between at least one of the third-party library constraints and the initial framework version constraint, or when it is determined that there is a conflict between the candidate target framework configuration information and the version constraint on the framework in the hardware constraint, an alarm message is generated, and the alarm message is used to instruct the modification of the custom configuration file.

7. The method according to claim 5, characterized in that The configuration information further includes the target format of the environment configuration file; the hardware configuration information further includes the graphics card model of the server. When the target format is a management software package file format, searching for target image configuration information corresponding to the hardware configuration information from a preconfigured image library based on the hardware configuration information includes: According to the graphics card model and the initial architecture version information, the target image configuration information corresponding to the graphics card model and the initial architecture version information is determined from the pre-configured image library.

8. The method according to claim 7, characterized in that When the target format is a container image format, searching for target image configuration information corresponding to the hardware configuration information from a preconfigured image library according to the hardware configuration information includes: Determine a toolkit version of a Compute Unified Device Architecture toolkit according to the graphics card model and the initial architecture version information; The target image configuration information is determined according to the toolkit version and the target framework configuration information.

9. The method according to claim 1, characterized in that The custom configuration file is a configuration file in a declarative environment description language format.

10. The method according to claim 7, characterized in that The method further comprises: A container image construction file corresponding to the environment configuration file is generated according to the environment configuration file, and the container image construction file is used to deploy network applications across environments.

11. The method according to claim 10, characterized in that The environment configuration file includes system dependency information corresponding to the operating system version of the server; and generating a container image construction file corresponding to the environment configuration file according to the environment configuration file includes: According to the target image configuration information included in the environment configuration file, obtaining a target image corresponding to the target image configuration information; Add system dependency instructions corresponding to the system dependency information to the target image, install the framework corresponding to the target framework configuration information and at least one third-party library corresponding to the target third-party library configuration information, set a working directory and copy the project files of the project to be executed in the development environment to generate the container image build file.

12. The method according to claim 10, characterized in that The method further comprises: Building a first container image based on the container image building file; After starting a container based on the first container image, creating a test script, wherein the test script is used to check the environment configuration of the development environment; Adding the test script to the container to build a second container image, running the container to test the graphics card, and outputting the test results, wherein the test results include the actual environment configuration file of the development environment; comparing the actual environment configuration file with the configuration information; If the actual environment configuration file is consistent with the configuration information, it is determined that the development environment configuration is successful.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for configuring a development environment as claimed in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the configuration method of the development environment according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the configuration method of the development environment according to any one of claims 1 to 12 are implemented.

Images