Image generation method, system, medium and application for machine learning components

Abstract

The present invention belongs to the field of image generation technology, and discloses an image generation method, system, medium and application for machine learning components, including: installing a docker tool; generating a new jupyter container; modifying the jupyter controller code to make the newly generated jupyter container have a privileged mode; starting the docker service in the newly generated jupyter container, and using the docker tool to create an image. The image generation method provided by the present invention uses the docker tool in the jupyter environment of the kubeflow container platform, and all jupyter environments created by the platform can directly use the docker tool to perform related operations such as creating an image and deploying a docker container, thereby realizing the one-stop operation requirements of algorithm personnel for debugging + building + deploying in the jupyter environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and in particular relates to an image generation method, system, medium and application for a machine learning component, and specifically relates to a method for building an image using a docker service in a machine learning jupyter component. Background Art

[0002] Currently, in the existing mode of the Jupyter environment, there are two ways for current algorithm researchers to use Jupyter for modeling:

[0003] (1) The local client or server uses Python to start a Jupyter server and then access it through a local browser.

[0004] (2) Private cloud and public cloud deployment: The JupyterLab service provided by platforms such as Baidu BML and Huawei ModelArts is deployed through containers on the public cloud, or deployed in the notebook server in the Kubeflow private cloud, and users access it through a local browser.

[0005] In order to solve the above technical problems, the prior art CN201110073978.7 is a programming platform system programmed with a description language, which defines system requirements as corresponding function keys and defines corresponding program blocks. The program blocks include two parts, one is a program syntax block, and the other is a program variable block. Each program variable block corresponds to a calculation mode system. In actual application, the program variable block will perform calculations according to the characteristics described by the user when selecting such requirements, thereby obtaining a high-level computer program system of the corresponding variables.

[0006] Prior art 2 CN201910738817.1 discloses a natural language programming method and device based on a strong static scripting language, the method comprising: a front-end compiler obtains a first script and meta-information from a database, compiles the first script into a loaded editor renderer, and verifies the content in the editor renderer according to the meta-information, wherein the meta-information includes at least type information; generates a second script by editing the verified editor renderer, and stores the second script in a database; a back-end interpreter obtains the second script and meta-information from a database, compiles the second script into a runtime object, and verifies the runtime object according to the meta-information.

[0007] Prior art three CN201910804907.6 discloses a method for constructing a natural language programming interface NLI for a software project and a method for generating code. The code generation method is as follows: 1) encapsulate each software project as a primitive in NLI, including: functional feature description of the primitive, API call mode, object parameters and other parameters; 2) determine the abstract syntax tree node type corresponding to the primitive; each node type specifies the child nodes and attributes of the node of this type; after binding each child node and attribute of the node to the corresponding Java code text area, during the process of completing the parameters of the NLI primitive, the projection editor generates the abstract syntax tree of the code; 3) starting from the root node of the abstract syntax tree, recursively transform each node, insert the attributes and child nodes in the abstract syntax tree node into the vacant part of its API call mode, and finally, generate the Java code corresponding to the primitive.

[0008] Through the above analysis, the problems and defects of the prior art are as follows:

[0009] (1) The first mode of the prior art is generally suitable for individual testing by algorithm personnel. In this mode, users can directly use Jupyter to deploy the Docker service on the machine. However, in order to ensure the stability of the environment, it is generally used by a single user, and multiple algorithm personnel will not use the same Jupyter service, which increases costs.

[0010] (2) In the second mode of the prior art, the jupyter service provided by cloud vendors or kubeflow can support the creation of multiple jupyter services based on one platform, but the docker tool cannot be used in jupyter, which makes the image generation efficiency low.

[0011] The significance of solving the above problems and defects is: algorithm personnel can create enough Jupyter containers for independent modeling based on limited hardware resources, and use Docker tools to build images and deploy Docker services in the container, which can be completed directly in the Jupyter container, integrating debugging, building and deployment. Summary of the invention

[0012] In view of the problems existing in the prior art, the present invention provides a mirror generation method, system, medium and application for machine learning components.

[0013] The present invention is implemented as follows: a method for generating an image in a machine learning component is applied to a client, and the method for generating an image in a machine learning component includes:

[0014] By generating new open source engines in the language programming environment created by machine learning containers;

[0015] The machine learning open source engine container is a notebook platform for algorithm modeling deployed based on the k8s cluster environment. Users can create an interactive development environment based on jupyter on this platform.

[0016] The main operation process of the open source engine is:

[0017] (1) Log in to the notebook platform;

[0018] (2) Create a notebook service (select the Jupyter image to be used, mounted data volumes and other resources);

[0019] (3) Enter the created notebook service to perform the Jupyter interactive development environment.

[0020] The process of starting a new open source engine container is: code debugging is performed in a language programming environment. After the code debugging is completed, a Dockerfile file can be written to build the code and dependent environment into a Docker image.

[0021] The debugging part is mainly code debugging. Users can perform model training, data preprocessing and other related Python operations in the Jupyter interactive development environment.

[0022] The construction part is mainly to build a docker image. Users can write a dockerfile file based on the debugged code in the jupyter interactive development environment, and build the relevant code and environment into a docker image for storage.

[0023] The deployment part mainly uses the built docker image to create a docker container to deploy and publish services.

[0024] Further, applied to the client, the image generation method for the machine learning component also includes:

[0025] Step 1: Install the docker tool and generate a new jupyter container image;

[0026] Step 2: Modify the Jupyter controller code to enable the newly generated Jupyter container to have privileged mode;

[0027] Step 3: Start the Docker service in the newly generated Jupyter container and use the Docker tool to create the image.

[0028] ps: Use service docker start to start the docker service in the jupyter container.

[0029] Furthermore, in step 1, the docker tool is installed based on the jupyter image.

[0030] Furthermore, in step 2, the controller modification method is to modify the template file referenced by the controller when creating the service, add the code related to the privileged mode, and build the modified code into a new notebook controller image for use. Using the privileged mode allows the container to use most of the capabilities of the host machine, including some kernel features and device access, similar to system-level tools such as systemctl service.

[0031] Furthermore, in step three, the docker service is started under the condition that the newly generated jupyter container has a privileged mode.

[0032] Another object of the present invention is to provide an image generation system for a machine learning component used in the image generation method for a machine learning component, the image generation system for the machine learning component comprising:

[0033] (1) Basic environment module: The basic environment that supports the use of this system is the open source kubernetes cluster + notebook service environment.

[0034] (2) The new programming language program image module is used to install the docker tool on the basis of the original jupyte program image to generate a new jupyter image. Using the container created by the image, the docker tool can be called in the jupyter program to perform related operations such as docker image creation and container service creation. The core technology of this system is dockerin docker.

[0035] (3) The programming language controller module performs underlying code modification based on the original notebook controller and adds a container privilege mode module. After the modification, the Jupyter container created by the notebook controller will carry the privilege mode, and system-level commands such as systemctl service can be used in the Jupyter container.

[0036] (4) When the open source engine startup module, the new programming language program image module, and the programming language controller module both meet the conditions, it supports starting the docker service in the jupyter container, and then uses the docker tool to execute the image generation method for the machine learning component.

[0037] Another object of the present invention is to provide a client, which is used to implement the image generation method used in the machine learning component.

[0038] Another object of the present invention is to provide an application of the mirror generation method used in a machine learning component in software development in the financial, transportation, and public safety fields.

[0039] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows:

[0040] The present invention uses the docker tool in the kubeflow (a machine learning toolkit for Kubernetes. Kubeflow is a set of technology stacks running on K8S. The technology stack includes many components. The relationship between the components is relatively loose. We can use them together or use some of them separately) container platform's jupyter (a combination of (Julia, a high-performance dynamic high-level programming language for scientific computing), Python and R (a language and operating environment for statistical analysis and drawing, with a font shape similar to Jupiter, and the languages ​​it currently supports far exceed these three. More than 40 programming languages, such as C++, C#, MATLAB, Spark (Scala), are all supported by Jupyter) environment, and all jupyter environments created by the platform can be directly mirrored using the docker tool.

[0041] The present invention starts the Docker container and other related operations, solving the one-stop operation requirements of algorithm personnel for debugging + building + deploying in the Jupyter environment.

[0042] Compared with the prior art, after algorithm personnel complete code debugging in Jupyter, they need to migrate the code to a server that can use Docker services and then build an image, and then use the built image deployment service to verify the effect. The present invention allows algorithm personnel to use Docker services to build images directly in the current Jupyter environment after debugging the code, as well as to deploy Docker services in a one-stop manner, avoiding the problem of code migration and switching back and forth between different platforms to release services. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a method for generating an image in a machine learning component provided by an embodiment of the present invention.

[0044] Figure 2 It is a rendering of the image generation method for a machine learning component provided in an embodiment of the present invention.

[0045] Figure 3 is a schematic diagram of an image generation system for a machine learning component provided by an embodiment of the present invention;

[0046] In the figure: 1. Basic environment module; 2. New programming language program image module; 3. Programming language controller module; 4. Open source engine startup module.

[0047] Figure 4 This is a comparison diagram of the mainstream solutions in the Jupyter modeling industry provided by the embodiments of the present invention.

[0048] Figure 5 It is an operation flow chart of the image generation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In view of the problems existing in the prior art, the present invention provides a mirror image generation method for a machine learning component. The present invention is described in detail below with reference to the accompanying drawings.

[0051] The present invention can use docker (an open source application container engine) in jupyter; when using the jupyter environment, algorithm personnel use docker to encapsulate their own business code and environment into an image, start the container in jupyter, and verify the scenario of the image function.

[0052] The machine learning open source engine container is a notebook platform for algorithm modeling deployed based on the k8s cluster environment. Users can create an interactive development environment based on jupyter on this platform.

[0053] The main operation process of the open source engine is:

[0054] 1. Log in to the notebook platform;

[0055] 2. Create a notebook service (select the jupyter image to be used, mounted data volumes and other resources);

[0056] 3. Enter the created notebook service to start the Jupyter interactive development environment. Figure 1 As shown, the image generation method for a machine learning component provided by an embodiment of the present invention includes the following steps:

[0057] S101, install the docker tool based on the original jupyter image to generate a new jupyter image.

[0058] S102, modify the original jupyter controller code so that the newly generated jupyter container has a privileged mode.

[0059] S103, use the service docker start command to start the docker service in the jupyter container, and then use the docker tool.

[0060] The Jupyter controller modification method is to modify the template file referenced by the controller when creating a service, add the code related to the privileged mode, and build the modified code into a new notebook controller image for use. Using the privileged mode allows the container to use most of the capabilities of the host machine, including some kernel features and device access, similar to system-level tools such as systemctl service.

[0061] The Docker service is started under the condition that the newly generated Jupyter container has privileged mode.

[0062] Ordinary technicians can also use other steps to implement the image generation method for machine learning components provided by the present invention. Figure 1 The image generation method for a machine learning component provided by the present invention is only a specific embodiment.

[0063] In one embodiment of the present invention, Figure 2 It is a rendering of the image generation method for a machine learning component provided in an embodiment of the present invention.

[0064] In one embodiment of the present invention, Figure 3 As shown, the image generation system for a machine learning component provided by an embodiment of the present invention includes:

[0065] Basic environment module 1 is used to support the use of the basic environment as an open source kubernetes cluster and notebook service environment;

[0066] New programming language program image module 2 is used to install the docker tool based on the original jupyter image; generate a new jupyter image; use the container created by the image to call the docker tool in the jupyter program to perform docker image creation and related operations to create container services;

[0067] Programming language controller module 3 is used to modify the underlying code based on the original notebook controller and add a container privilege mode module. The Jupyter container created by the modified notebook controller carries the privilege mode and uses systemctl service in the Jupyter container. These two system-level commands are used within the privilege mode function range.

[0068] The open source engine startup module 4 is used to support the image to start the docker service in the jupyter container and then use the docker tool when both the new programming language program image module and the programming language controller module meet the conditions.

[0069] The effect of the embodiment of the present invention is very good, and the comparison with the current mainstream solution is as follows:

[0070] The comparison results of the mainstream solutions in the Jupyter modeling industry are shown in Table 1.

[0071] Table 1 Comparison of mainstream solutions in the jupyter modeling industry

[0072] Jupyter deployment method Support template creation Support the use of docker tools Support multi-tenant use server no yes no K8s Cluster no yes no Cloud vendors yes no yes kubeflow yes no yes The present invention yes yes yes

[0073] Comparison effect chart Figure 4 shown.

[0074] The operation flow chart of this embodiment is as follows Figure 5 shown.

[0075] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0076] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A mirror generation method for a machine learning component, characterized in that: Applied to the client, the image generation method for the machine learning component includes: By generating new open source engines in the language programming environment created by machine learning containers; Start a new open source engine service, perform one-stop operation procedures for debugging, building, and deploying in a language programming environment, and perform image creation and deployment of Docker containers; Applied to the client, the image generation method for the machine learning component also includes: Step 1: Install the docker tool and generate a new jupyter container. Step 2: Modify the Jupyter controller code to enable the newly generated Jupyter container to have privileged mode; Step 3: Start the Docker service in the newly generated Jupyter container and use the Docker tool to create an image. In the step 1, the docker tool is installed on the basis of the jupyter image; in the step 3, the docker service is started under the condition that the newly generated jupyter container has a privileged mode.

2. The image generation method for a machine learning component according to claim 1, characterized in that: The operation process of the open source engine is as follows: (1) Log in to the notebook platform; (2) Create a notebook service, select the Jupyter image to be used, and mount the data volume resources; (3) Enter the created notebook service to perform the Jupyter interactive development environment.

3. The image generation method for a machine learning component according to claim 1, characterized in that: The operation process of starting a new open source engine service is: code debugging is performed in a language programming environment. After the code debugging is completed, a Dockerfile file can be written to build the code and the dependent environment into a Docker image.

4. The image generation method for a machine learning component according to claim 1, characterized in that: The debugging part is code debugging. Users can perform Python operations related to model training and data preprocessing in the Jupyter interactive development environment. The construction part is to build a docker image. Users can write a dockerfile file based on the debugged code in the jupyter interactive development environment to build the relevant code and environment into a docker image for storage; The deployment part is to use the built docker image to create a docker container to deploy and publish services.

5. An image generation system for a machine learning component that implements the image generation method for a machine learning component according to any one of claims 1 to 4, characterized in that: Applied to the client, the image generation system used in the machine learning component includes: Basic environment module, which supports the use of the system's basic environment as an open source kubernetes cluster + notebook service environment; The new programming language program image module is used to install the docker tool on the basis of the original jupyte program image to generate a new jupyter image. Using the container created by the image, the docker tool can be called in the jupyter program to perform docker image creation and create container service related operations. The programming language controller module is based on the original notebook controller. The underlying code is modified and the container privilege mode module is added. After the modification, the Jupyter container created by the notebook controller will carry the privilege mode, and the systemctl service system-level command can be used in the Jupyter container. When the open source engine startup module, the new programming language program image module, and the programming language controller module both meet the conditions, it supports starting the docker service in the jupyter container and then using the docker tool.

6. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the image generation method for a machine learning component as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the image generation method for a machine learning component as described in any one of claims 1 to 4.

8. A client, characterized in that: The client is used to implement the image generation method for a machine learning component as described in any one of claims 1 to 4.

Images