A method and system for distributed online deployment of a Python model

Abstract

The application discloses a kind of Python model distributed online deployment method and system, data processing and feature engineering code converter designed by the application convert the logic of offline data processing and feature engineering of Python model training front-end into online Python code;Model engine designed by the application can quickly load various different kinds of Python models such as machine learning, deep learning, natural language processing etc.Through the Python model distributed deployment method and system provided by the application, Python model can be quickly deployed online, and can easily meet the requirements of high concurrency, high availability and fault tolerance etc.Algorithm engineers only need to spend time and effort on modeling, thereby improving the model development efficiency and shortening the model iteration cycle.

Description

Technical Field

[0001] This invention belongs to the field of model deployment technology, and in particular relates to a distributed online deployment method and system for Python models. Background Technology

[0002] With the accumulation of massive amounts of data and the continuous development of AI theory and technology, machine learning and deep learning models have been applied to various fields such as industry, finance, retail, automotive, and healthcare. Python, as the most commonly used tool for data analysis and model training, is widely used in the development of models across various fields. After a Python model is developed, it needs to be deployed online. Currently, there are two methods for deploying Python models, and the process is detailed below. Figure 1 .

[0003] 1. Algorithm engineers use Flask to build simple web services, which use Python to load models and provide services to the outside world.

[0004] 2. The algorithm engineer delivers the Python model file, model training logic, and model calling documentation to the deployment engineer. The deployment engineer then converts the model into a specific format, such as a PMML file, and loads the model using a Java or C++ program to provide services to the outside world.

[0005] Because online model deployment has requirements such as high concurrency, high availability, and high fault tolerance, and model updates and releases in different domains may be very frequent, and offline data processing and feature engineering also need to be implemented during online prediction; therefore, both of the above Python model deployment methods have their own shortcomings.

[0006] The first approach, where algorithm engineers use Flask to build simple web services, has several drawbacks. First, Flask's performance is poor and cannot meet performance requirements when the call volume is high. Second, online model deployment has certain requirements for high concurrency, high availability, and fault tolerance, making it inefficient for algorithm engineers to handle the deployment work. Finally, offline data processing and feature engineering also require algorithm engineers to implement an online program, further increasing the workload of Python model deployment.

[0007] The second method involves algorithm engineers and deployment engineers collaborating to deploy the model online. First, the algorithm engineer delivers the Python model file, model training logic, and model calling documentation to the deployment engineer. Then, the deployment engineer needs to convert offline data processing and feature engineering into a deployable program and convert the model into a deployable form. This involves certain communication costs. Moreover, when converting the model into a deployable form, the deployment engineer needs to ensure the consistency between the offline training and online prediction results of the Python model, which places high demands on the deployment engineer's skills. Therefore, this method is time-consuming, labor-intensive, and inefficient. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a distributed online deployment method and system for Python models, thus solving the following technical problems:

[0009] 1. How to meet the deployment and online requirements of any different types of Python models;

[0010] 2. How to achieve the requirements of high concurrency, high availability, and fault tolerance for Python online models;

[0011] 3. How to achieve hot updates of model versions without affecting business logic calls;

[0012] 4. How to enable offline data processing and feature engineering to be quickly deployed online while ensuring consistency between offline and online logic and data accuracy, thereby improving operational efficiency.

[0013] The objective of this invention is achieved through the following technical solution:

[0014] According to a first aspect of this specification, a method for distributed online deployment of a Python model is provided, the method comprising:

[0015] The logic of offline data processing and feature engineering before training Python models is converted into online Python code through a data processing and feature engineering code converter.

[0016] Load different types of Python models through the model engine;

[0017] The Python model is deployed online using a distributed deployment framework, including:

[0018] The distributed file storage system is used to store the Python model files uploaded by users, so that each model engine can load the Python model files.

[0019] Calculate the number of model engines currently required based on the concurrent call volume and the number of requests each model engine can handle;

[0020] When deploying a new Python model, set the number of instances of the model, y. Based on the resource usage of the server where the model engine is located, select the y model engines with the most abundant resources to deploy the model.

[0021] Furthermore, the code converter is used by users to configure the logic of offline data processing and feature engineering before model training through an interface, and stores the user configuration in JSON format; it reads the user configuration and generates online Python code with the same logic based on the logic of offline data processing and feature engineering.

[0022] Furthermore, the code converter employs an online Python code computation method based on Jython to convert Python code into bytecode and then run it in the JVM.

[0023] Furthermore, the model engine is used to load different kinds of Python models, including:

[0024] Model file initialization: Based on the Python model file uploaded by the user, load the Python model into memory to generate the model prediction object;

[0025] Model input parameter and model prediction result identification: Parse the contents of the Python model file, convert it into a Python object, and identify the model input parameters and model prediction results by reading the Python object variables;

[0026] Model prediction: When requested data is distributed to the model engine, the received data is converted into the input parameter format of the Python model. Based on the input parameter data, the model prediction object generated in memory is called to obtain and return the model prediction result.

[0027] Furthermore, users encapsulate the Python model into the format defined by the model template, then compress it into a ZIP archive, and upload the Python model file archive. The model engine then decompresses the model file archive for further processing.

[0028] Furthermore, the model engine has a built-in Python algorithm package to support the loading of Python models. If an algorithm package not included in the model engine is needed, the user must upload the algorithm package along with the model file.

[0029] Furthermore, the distributed deployment framework load balances all requests calling the model to each model engine, increasing the number of model instances when the request concurrency is high and decreasing the number of model instances when the request concurrency is low.

[0030] Furthermore, the distributed deployment framework stores model deployment information on Zookeeper. When a model engine exits abnormally, it retrieves the model deployment information stored on the abnormal model engine, and then selects the model engine with the most sufficient resources based on the server resource usage of the normal model engine. The model deployed on the abnormal model engine is then migrated to other normal model engines. The model call requests being processed on the abnormal model engine are forwarded to other normal model engines.

[0031] Furthermore, the distributed deployment framework stores model deployment information on ZooKeeper. When a user updates the model version, the model files on each model engine are updated sequentially. While a model file on one model engine is being updated, the models on other model engines can still be used. If the update is successful, the model file on the next model engine will be updated. If the update fails, it will automatically roll back to the previous version of the model file.

[0032] According to a second aspect of this specification, a distributed online deployment system for Python models is provided. This system is based on a distributed deployment framework and includes a code converter and a model engine.

[0033] The code converter is used to convert the logic of offline data processing and feature engineering before training a Python model into online Python code;

[0034] The model engine is used to load different types of Python models;

[0035] The distributed deployment framework implements distributed online deployment of Python models, including:

[0036] The distributed file storage system is used to store the Python model files uploaded by users, so that each model engine can load the Python model files.

[0037] Calculate the number of model engines currently required based on the concurrent call volume and the number of requests each model engine can handle;

[0038] When deploying a new Python model, set the number of instances of the model, y. Based on the resource usage of the server where the model engine is located, select the y model engines with the most abundant resources to deploy the model.

[0039] The beneficial effects of this invention are:

[0040] 1. With the distributed deployment method and system for Python models provided by this invention, algorithm engineers can quickly deploy Python models and pre-processing, feature engineering, etc. online; no deployment engineer intervention is required, thereby reducing labor costs; by converting online Python code into bytecode and then running it in the JVM, the running efficiency is improved.

[0041] 2. Through the distributed deployment method and system for Python models provided by this invention, Python models can be quickly deployed and launched, easily meeting the requirements of high concurrency, high availability and fault tolerance. Algorithm engineers only need to spend their time and energy on modeling, thereby improving model development efficiency and shortening the model iteration cycle. Attached Figure Description

[0042] Figure 1 Flowchart for deploying an existing Python model;

[0043] Figure 2 A flowchart of a distributed online deployment process for a Python model is provided as an exemplary embodiment.

[0044] Figure 3 A configuration interface for a data processing and feature engineering code converter is provided as an exemplary embodiment;

[0045] Figure 4 A distributed deployment architecture diagram provided for an exemplary embodiment. Detailed Implementation

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0048] This invention provides a distributed online deployment method and system for Python models. The model deployment flowchart is shown below. Figure 2 The technical solution and implementation details of the present invention will be described in detail below.

[0049] First, a data processing and feature engineering code converter was designed and implemented. The key steps of this code converter are as follows:

[0050] 1. This code converter has built-in common mathematical functions, time functions, string functions, array functions, regular expression functions, and various common operators.

[0051] 2. Supports users configuring the logic for offline batch data processing and feature engineering before the model is built in a user interface. The user's configuration is then stored in the system in JSON format. The configuration interface for this code converter can be found here. Figure 3 .

[0052] 3. This code converter can read the user's configuration and generate online Python code with the same logic as offline data processing and feature engineering, thereby ensuring the consistency and accuracy of offline and online data.

[0053] 4. This code converter provides an online Python code calculation method based on Jython, which converts Python code into bytecode and then runs it in the JVM, reducing the code compilation time each time it runs, thereby improving the efficiency of online execution.

[0054] Secondly, a Python model engine was designed and implemented that can load models of any different types. This model engine can quickly load various types of Python models, such as machine learning, deep learning, and natural language processing models. The model engine implements the following five methods:

[0055] 1. Model file initialization: Users need to package the Python model into the format defined by the template provided by the system, and then compress it into a ZIP archive. Then, upload the compressed model file archive to the system. This method can decompress the compressed model file archive and load the Python model into memory to generate the model prediction object. The model engine has most of the commonly used Python algorithm packages built-in to support the loading of Python models. If you need to use an algorithm package that is not in the model engine, you only need to upload the algorithm package and the model file together to the system.

[0056] 2. Model input parameter identification: This method can parse the content of the model file packaged according to the model template, convert it into a Python object, and identify the parameter value, parameter type, and parameter reference value of the Python model input parameters by reading the Python object variables.

[0057] 3. Model prediction output identification: This method can parse the content of the model file packaged according to the model template, convert it into a Python object, and identify the field names and field types of the Python model output by reading the Python object variables.

[0058] 4. Online data processing and feature engineering calculations: This method can convert offline data processing and feature engineering logic into online Python code based on the user's configuration in the code converter, thus enabling offline data processing and feature engineering to be quickly deployed online.

[0059] 5. Model Prediction: When requested data is distributed to the model engine, this method can receive the data, convert it into the input parameter format of the Python model, call the model prediction object generated in memory based on the input parameter data, and obtain and return the model prediction result.

[0060] Then, a distributed model deployment framework was designed and implemented. This framework aims to solve the problems of high concurrency, high availability, fault tolerance, and hot update of model versions for Python online models. Its architecture diagram is shown below. Figure 4 The core functionality of this framework is as follows:

[0061] 1. This framework uses the distributed file storage system HDFS to store Python model files, so that each model engine can load the model files.

[0062] 2. This framework allows you to set a minimum number of model engines, x. When the concurrent call volume is low, only x model engines are started. When the concurrent call volume is high, the framework can calculate the required number of model engines based on the concurrent call volume and the number of requests each model engine can handle, thus dynamically adjusting the number of model engines to save server resources. The framework assigns an access address to each model engine, which can be used to access all models deployed on that engine. A model engine can load multiple different models, and a model can be deployed on multiple different model engines. When deploying a new model, you can set the number of instances y for that model. The framework can then select the y model engines with the most abundant resources based on the server resource usage of the model engines.

[0063] 3. This framework can load balance all requests calling the model to each model engine, achieving high availability of online models. When the request concurrency is high, the framework will automatically increase the number of model instances, and when the request concurrency is low, the framework will automatically reduce the number of model instances. Therefore, this framework can meet high concurrency requirements while saving resources by reducing the number of instances when concurrency is low.

[0064] 4. The framework stores model deployment information on ZooKeeper. Therefore, when a model engine exits abnormally, the framework obtains the model deployment information stored on the abnormal model engine. Then, based on the server resource usage of the normal model engines, it selects the model engine with the most sufficient resources and migrates the models deployed on the abnormal model engine to other normal model engines. In addition, it forwards the model call requests being processed on the abnormal model engine to other normal model engines, thereby ensuring the fault tolerance of the system.

[0065] 5. The framework also implements hot updates for model versions. It stores model deployment information in ZooKeeper, allowing the framework to read this information and record the model engine information for each model. When a user wants to update a model version, the framework sequentially updates the model files on each model engine. While one model engine is updating, models on other model engines can still be used. If the update is successful, it updates the model files on the next model engine; if the update fails, it automatically rolls back to the previous version. This ensures hot updates of model versions without affecting business logic.

[0066] The distributed online deployment method and system for Python models provided by this invention can ensure high concurrency, high availability, and high fault tolerance of online models. Algorithm engineers only need to upload the model files to the system, and the system will automatically deploy the Python model online and provide online calling documentation. This can reduce labor costs and speed up the efficiency of model application.

[0067] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for distributed online deployment of Python models, the method comprising: The method comprises: Converting the logic of offline data processing and feature engineering before Python model training into online Python code through data processing and feature engineering code converter; Loading different kinds of Python models through model engine, including: Model file initialization: loading Python model into memory to generate model prediction object according to user uploaded Python model file; Model input parameter and model prediction result identification: parsing Python model file content into Python object, and identifying model input parameter and model prediction result by reading Python object variable; Model prediction: converting received data into input parameter format of Python model, calling model prediction object generated in memory according to input parameter, and obtaining and returning model prediction result when request data is distributed to model engine; Using distributed deployment framework to deploy Python model online, including: Storing user uploaded Python model file in distributed file storage system, so that each model engine can load Python model file; According to the size of concurrent call quantity and the number of requests that each model engine can handle, calculating the number of model engines currently required; When deploying a new Python model, setting the number of instances y of the model, and selecting y model engines with the most sufficient resources to deploy the model according to the resource usage of the model engine server.

2. The Python model distributed online deployment method according to claim 1, characterized in that, The code converter is used for users to configure the logic of offline data processing and feature engineering before model training through the interface, and store the user configuration in JSON format; and read user configuration and generate online Python code with the same logic according to the logic of offline data processing and feature engineering.

3. The method of claim 1, wherein, The code converter uses Jython-based online Python code calculation method to convert Python code into bytecode and then run it in JVM.

4. The Python model distributed online deployment method according to claim 3, characterized in that, Users encapsulate Python model into the format defined by model template, then compress it into ZIP package, upload Python model file package, and model engine decompresses model file package for subsequent processing.

5. The method of claim 1, wherein, The model engine has built-in Python algorithm package to support the loading of Python model, and if the algorithm package is not available in the model engine, the user needs to upload the algorithm package together with the model file.

6. The method of claim 1, wherein, The distributed deployment framework balances all model calling requests to each model engine, increases the number of model instances when the request concurrency is large, and reduces the number of model instances when the request concurrency is small.

7. The method of claim 1, wherein, The distributed deployment framework stores model deployment information on Zookeeper, when a model engine exits abnormally, acquires the model deployment information stored on the abnormal model engine, and then selects the model engine with the most sufficient resources according to the server resource usage of the normal model engine, and migrates the model deployed on the abnormal model engine to other normal model engines; the model calling request being processed on the abnormal model engine is forwarded to other normal model engines.

8. The method of claim 1, wherein, The distributed deployment framework stores model deployment information on Zookeeper, when a model engine exits abnormally, acquires the model deployment information stored on the abnormal model engine, and then selects the model engine with the most sufficient resources according to the server resource usage of the normal model engine, and migrates the model deployed on the abnormal model engine to other normal model engines; the model calling request being processed on the abnormal model engine is forwarded to other normal model engines. 9.A distributed online deployment system for Python models, characterized in that, The system is implemented based on a distributed deployment framework, and the system comprises a code converter and a model engine. The code converter is configured to convert the logic of offline data processing and feature engineering before Python model training into online Python code. The model engine is configured to load different kinds of Python models, and the model engine comprises: Model file initialization: according to the Python model file uploaded by a user, the Python model is loaded into memory to generate a model prediction object; Model input parameter and model prediction result identification: the content of the Python model file is parsed into a Python object, and the model input parameter and the model prediction result are identified by reading the variable of the Python object; Model prediction: when request data is distributed to the model engine, the received data is converted into the input parameter format of the Python model, the model prediction object generated in the memory is called according to the input parameter, and the model prediction result is obtained and returned; The distributed deployment framework implements distributed online deployment of Python models, and the distributed deployment framework comprises: A distributed file storage system is used to store the Python model file uploaded by the user, so that each model engine can load the Python model file; According to the size of the concurrent call quantity and the number of requests that can be processed by each model engine, the number of model engines required at present is calculated; When a new Python model is deployed, the number of instances y of the model is set, and the y model engines with the most sufficient resources are selected according to the server resource usage to deploy the model.

Images