Multi-machine deployment scheduling method and device based on model service automation

Abstract

This invention proposes a multi-machine deployment and scheduling method based on automated model service, comprising: obtaining a JSON description file representing the model service workflow through a client and issuing a service initialization request; the scheduling server receiving the request and allocating inference server resources; the scheduling server allocating inference server resources that meet the needs of each client based on a dynamic weighted least connections algorithm; returning the request call IP and port of the inference server corresponding to the inference server resources to the client; the client performing inference on the requested service instance and returning the inference result. This invention improves the stability of the inference service by dynamically allocating inference server resources to clients through multi-machine deployment and load balancing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence applications. Background Art

[0002] With the continuous development of artificial intelligence theory and technology, the ability of artificial intelligence scene integration has been continuously improved. A single neural network model is no longer sufficient to support a large number of complex and changing business scenarios, especially in open environments such as the power sector. Complex and changing scenarios and tasks often have problems such as increased categories, changes in attributes, and diverse goals, which require multiple models to form different workflows for processing. The back-and-forth migration between different scenarios and tasks creates difficulties in model management and workflow management, and also limits the flexibility and scalability of technology reuse. Therefore, how to quickly and efficiently combine and orchestrate models and data processing components to automatically build agile and flexible model service workflows has great practical value and research significance.

[0003] Currently, mainstream model service libraries such as Tensorflow Serving and TorchServe only support their own Tensorflow and Pytorch frameworks, and cannot provide model service and management functions for other frameworks; BentoML and MLFlow support rapid deployment of large-scale deep learning models in multiple frameworks, and can customize the loading strategy of model versions and manage model iterations through configuration files, but do not support the implementation of model orchestration workflows, and cannot achieve flexible combinations of deep learning models, pre-processing components, and post-processing components in complex open scenarios; Seldon is a model orchestration framework, but only after building a model API server and containerizing it with docker can Seldon schedule model containers on a Kubernetes cluster, and does not provide one-stop deployment services for native models.

[0004] As mentioned above, most of the current mainstream open source machine learning online service frameworks only provide single model reasoning services, and do not provide configurable models and multiple pre-processing and post-processing component orchestration functions. Therefore, when the model is deployed, it is necessary to design and implement supporting model combination reasoning solutions for different business scenarios. Although this design can temporarily meet business needs, it has the following two obvious disadvantages.

[0005] 1) The coding workload is heavy. Each business scenario requires writing a model combination reasoning solution, including preprocessing of input data, multi-model combination reasoning, intermediate feature processing, and a complete set of post-processing modules. If there are few models, few dependent interfaces, and a single scenario, the above solution is acceptable; but in actual application scenarios, the tasks are complicated and constantly changing, which brings a very heavy workload.

[0006] 2) The system has poor maintainability and scalability. Although fixed model workflows can solve tasks in specific application scenarios, they cannot achieve flexible combinations of multiple algorithms and multiple processing components. Especially in a complex open environment, if the model is directly placed in an application or service, due to the complexity and variability of tasks, the required models and data processing components are numerous, which will cause serious coupling between model reasoning and task logic processing, increase maintenance costs after the service is launched, reduce model reuse efficiency, and poor system flexibility and scalability. Summary of the invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the purpose of the present invention is to propose a multi-machine deployment scheduling method based on model service automation, which is used to dynamically allocate inference server resources to clients and improve the stability of inference services.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a multi-machine deployment scheduling method based on model service automation, comprising:

[0010] The client obtains a JSON description file representing the model service workflow and issues a service initialization request, and the scheduling server receives the request and allocates inference server resources;

[0011] The scheduling server allocates inference server resources that meet the needs of each client based on the dynamic weighted least connection algorithm;

[0012] The request calling IP and port of the inference server corresponding to the inference server resource are returned to the client, and the client performs inference on the request calling service instance and returns the inference result.

[0013] In addition, the multi-machine deployment scheduling method based on model service automation according to the above embodiment of the present invention may also have the following additional technical features:

[0014] Further, in one embodiment of the present invention, the inference of the request calling service instance by the client and returning the inference result includes:

[0015] Get a JSON description file that effectively represents the model service workflow;

[0016] Processing the JSON description file to obtain information about the graph structure of the model service workflow;

[0017] Complete the initialization of the graph structure and model instantiation according to the information of the graph structure;

[0018] The execution order of each node in the graph structure of layered sequence initialization and model instantiation is completed, forming a service instance for the combined execution of multiple models, data processing components, and post-processing components;

[0019] Obtain request data from the client, call the service instance to perform reasoning on the request data, and return the reasoning result.

[0020] Furthermore, in one embodiment of the present invention, the information of the graph structure includes node information and edge information.

[0021] Further, in one embodiment of the present invention, completing the initialization of the graph structure and model instantiation according to the information of the graph structure includes:

[0022] 1) Initialize the graph structure according to the node information and edge information, graph G = (V(G), E(G));

[0023] 2) Select Vi from the graph nodes V(G) in sequence;

[0024] 3) If the Vi is a model node, execute step 4), otherwise go to step 7);

[0025] 4) If the model node has been registered in the model repository, go to step 6), otherwise go to step 5);

[0026] 5) Register the model node in the model repository;

[0027] 6) Obtain model parameters and instantiate model nodes;

[0028] 7) Determine whether the nodes in V(G) have been traversed. If so, the process ends, otherwise go to step 2).

[0029] Further, in one embodiment of the present invention, the hierarchical sequence completes the execution order of each node in the graph structure of initialization and model instantiation, forming a service instance executed by a combination of multiple models and data processing components and post-processing components, including:

[0030] 1) Initialize the layer order execution list L;

[0031] 2) Generate a topologically sorted list T(G) of the directed acyclic graph G;

[0032] 3) Select nodes Ti from T(G) in sequence;

[0033] 4) If the node Ti is a model node, go to step 5), otherwise go to step 6);

[0034] 5) Obtain the current instantiation model and previous node information of the node Ti, and insert them into the hierarchical execution list L;

[0035] 6) If the node Ti is a data processing node or a post-processing node, go to step 7), otherwise report an error and go to step 8);

[0036] 7) Instantiate the processing node or post-processing node of the data and insert it into the layer sequence execution list;

[0037] 8) Determine whether the nodes in T(G) have been traversed. If so, the service instance is constructed. Otherwise, go to step 3).

[0038] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a multi-machine deployment scheduling device based on model service automation, including:

[0039] An initialization module is used to obtain a JSON description file representing a model service workflow through a client and issue a service initialization request, and the scheduling server receives the request and allocates inference server resources;

[0040] A scheduling module, used to allocate inference server resources that can meet the needs of each client through a scheduling server based on a dynamic weighted least connection algorithm;

[0041] The inference module is used to return the request calling IP and port of the inference server corresponding to the inference server resource to the client, and perform inference on the request calling service instance through the client to return the inference result.

[0042] Furthermore, in one embodiment of the present invention, the service calling module is also used to:

[0043] Get a JSON description file that effectively represents the model service workflow;

[0044] Processing the JSON description file to obtain information about the graph structure of the model service workflow;

[0045] Complete the initialization of the graph structure and model instantiation according to the information of the graph structure;

[0046] The execution order of each node in the graph structure of layered sequence initialization and model instantiation is completed, forming a service instance for the combined execution of multiple models, data processing components, and post-processing components;

[0047] Obtain request data from the client, call the service instance to perform reasoning on the request data, and return the reasoning result.

[0048] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the multi-machine deployment scheduling method based on model service automation as described above.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the multi-machine deployment scheduling method based on model service automation as described above is implemented.

[0050] The multi-machine deployment scheduling method based on model service automation in the embodiment of the present invention, on the one hand, proposes an automated orchestration solution that supports multiple models and multiple components, encapsulates multiple models and data processing components and post-processing components, and combines them into a directed acyclic graph (DAG) in a serial or parallel manner, so as to realize rapid orchestration and reorganization of models in complex task scenarios;

[0051] On the other hand, a multi-machine deployment scheduling subsystem was designed. Based on the organizational structure of client-scheduling server-inference server, it dynamically allocates inference server resources to the client through multi-machine deployment and load balancing to improve the stability of the inference service. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A flowchart of a multi-machine deployment scheduling method based on model service automation provided in an embodiment of the present invention.

[0054] Figure 2 A multi-machine deployment scheduling architecture diagram provided for an embodiment of the present invention.

[0055] Figure 3 An automatic model arrangement architecture diagram provided for an embodiment of the present invention.

[0056] Figure 4 A processing flow chart of an orchestration engine module provided by an embodiment of the present invention.

[0057] Figure 5 A service generation process flow chart provided in an embodiment of the present invention.

[0058] Figure 6 A diagram of the overall system architecture provided for an embodiment of the present invention.

[0059] Figure 7A flow chart of a multi-machine deployment and scheduling device based on model service automation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0061] The following describes a multi-machine deployment scheduling method based on model service automation according to an embodiment of the present invention with reference to the accompanying drawings.

[0062] Figure 1 A flowchart of a multi-machine deployment scheduling method based on model service automation provided in an embodiment of the present invention.

[0063] like Figure 1 As shown, the multi-machine deployment scheduling method based on model service automation includes the following steps:

[0064] S101: The client obtains a JSON description file representing a model service workflow and issues a service initialization request, and the scheduling server receives the request and allocates inference server resources;

[0065] S102: Allocate inference server resources that can meet the needs of each client through the scheduling server based on a dynamic weighted least connection algorithm;

[0066] S103: Return the request calling IP and port of the inference server corresponding to the inference server resource to the client, and perform inference on the request calling service instance through the client to return the inference result.

[0067] like Figure 2 The figure shows the architecture diagram of multi-machine deployment scheduling.

[0068] Further, in one embodiment of the present invention, the inference of the request calling service instance by the client and returning the inference result includes:

[0069] Get a JSON description file that effectively represents the model service workflow;

[0070] Processing the JSON description file to obtain information about the graph structure of the model service workflow;

[0071] Complete the initialization of the graph structure and model instantiation according to the information of the graph structure;

[0072] The execution order of each node in the graph structure of layered sequence initialization and model instantiation is completed, forming a service instance for the combined execution of multiple models, data processing components, and post-processing components;

[0073] Obtain request data from the client, call the service instance to perform reasoning on the request data, and return the reasoning result.

[0074] The automated model orchestration proposed in this solution includes file acquisition, file processing, orchestration engine, service generation and service invocation. The automated model orchestration architecture diagram is as follows: Figure 3 shown.

[0075] The file acquisition part is used to obtain a JSON description file that can effectively represent the model service workflow from the outside. The description file can be generated by directly uploading a JSON file or by visual drag and drop editing. The requirement for the description file is that the parameter fields must comply with the model service workflow configuration specification.

[0076] The file processing part is used to process and parse the JSON description file obtained by the file acquisition module. The specific functions are as follows:

[0077] Parse the description file and return structured JSON format data.

[0078] Get the model service workflow diagram structure information and store it.

[0079] Furthermore, in one embodiment of the present invention, the information of the graph structure includes node information and edge information.

[0080] Further, in one embodiment of the present invention, completing the initialization of the graph structure and model instantiation according to the information of the graph structure includes:

[0081] 1) Initialize the graph structure according to the node information and edge information, graph G = (V(G), E(G));

[0082] 2) Select Vi from the graph nodes V(G) in sequence;

[0083] 3) If the Vi is a model node, execute step 4), otherwise go to step 7);

[0084] 4) If the model node has been registered in the model repository, go to step 6), otherwise go to step 5);

[0085] 5) Register the model node in the model repository;

[0086] 6) Obtain model parameters and instantiate model nodes;

[0087] 7) Determine whether the nodes in V(G) have been traversed. If so, the process ends, otherwise go to step 2).

[0088] The processing flow of the orchestration engine is as follows: Figure 4 shown.

[0089] Further, in one embodiment of the present invention, the hierarchical sequence completes the execution order of each node in the graph structure of initialization and model instantiation, forming a service instance executed by a combination of multiple models and data processing components and post-processing components, including:

[0090] 1) Initialize the layer order execution list L;

[0091] 2) Generate a topologically sorted list T(G) of the directed acyclic graph G;

[0092] 3) Select nodes Ti from T(G) in sequence;

[0093] 4) If the node Ti is a model node, go to step 5), otherwise go to step 6);

[0094] 5) Obtain the current instantiation model and previous node information of the node Ti, and insert them into the hierarchical execution list L;

[0095] 6) If the node Ti is a data processing node or a post-processing node, go to step 7), otherwise report an error and go to step 8);

[0096] 7) Instantiate the processing node or post-processing node of the data and insert it into the layer sequence execution list;

[0097] 8) Determine whether the nodes in T(G) have been traversed. If so, the service instance is constructed. Otherwise, go to step 3).

[0098] The processing flow of the service generation part is as follows Figure 5 shown.

[0099] The service call part uses the service instance constructed by the service generation module to listen to the client call request, call the service instance for reasoning on the incoming data, and return the reasoning result.

[0100] The above is a complete solution of the present invention, and the overall system architecture is shown in Figure 6.

[0101] The multi-machine deployment scheduling method based on model service automation in the embodiment of the present invention, on the one hand, proposes an automated orchestration solution that supports multiple models and multiple components, encapsulates multiple models and data processing components and post-processing components, and combines them into a directed acyclic graph (DAG) in a serial or parallel manner, so as to realize rapid orchestration and reorganization of models in complex task scenarios;

[0102] On the other hand, a multi-machine deployment scheduling subsystem was designed. Based on the organizational structure of client-scheduling server-inference server, it dynamically allocates inference server resources to the client through multi-machine deployment and load balancing to improve the stability of the inference service.

[0103] Figure 7 A structural diagram of a multi-machine deployment and scheduling device based on model service automation provided in an embodiment of the present invention.

[0104] like Figure 7 As shown, the multi-machine deployment scheduling device based on model service automation includes: an initialization module 100, a scheduling module 200, and an inference module 300, wherein:

[0105] An initialization module is used to obtain a JSON description file representing a model service workflow through a client and issue a service initialization request, and the scheduling server receives the request and allocates inference server resources;

[0106] A scheduling module, used to allocate inference server resources that can meet the needs of each client through a scheduling server based on a dynamic weighted least connection algorithm;

[0107] The inference module is used to return the request calling IP and port of the inference server corresponding to the inference server resource to the client, and perform inference on the request calling service instance through the client to return the inference result.

[0108] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the multi-machine deployment scheduling method based on model service automation as described above.

[0109] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the multi-machine deployment scheduling method based on model service automation as described above is implemented.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0111] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0112] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A multi-machine deployment scheduling method based on model service automation, characterized in that: The following steps are involved: The client obtains a JSON description file representing the model service workflow and issues a service initialization request, and the scheduling server receives the request and allocates inference server resources; The scheduling server allocates inference server resources that meet the needs of each client based on the dynamic weighted least connection algorithm; Returning the request calling IP and port of the inference server corresponding to the inference server resource to the client, performing inference on the request calling service instance through the client, and returning the inference result; The inference of the request calling service instance by the client and returning the inference result includes: Get a JSON description file that effectively represents the model service workflow; Processing the JSON description file to obtain information about the graph structure of the model service workflow; Complete the initialization of the graph structure and model instantiation according to the information of the graph structure; The execution order of each node in the graph structure of the hierarchical completion initialization and model instantiation is formed to form a service instance of the combined execution of multiple models and data processing components and post-processing components. In this case, multiple models and data processing components and post-processing components are encapsulated and combined into a directed acyclic graph in a serial or parallel manner to achieve rapid arrangement and reorganization of models in complex task scenarios. Obtain request data from the client, call the service instance to perform reasoning on the request data, and return the reasoning result.

2. The method according to claim 1, characterized in that: The information of the graph structure includes node information and edge information.

3. The method according to claim 2, characterized in that The initialization of the graph structure and model instantiation according to the information of the graph structure includes: 1) Initialize the graph structure according to the node information and edge information, graph G = (V(G), E(G)); 2) Select Vi from the graph nodes V(G) in sequence; 3) If the Vi is a model node, execute step 4), otherwise go to step 7); 4) If the model node has been registered in the model repository, go to step 6), otherwise go to step 5); 5) Register the model node in the model repository; 6) Obtain model parameters and instantiate model nodes; 7) Determine whether the nodes in V(G) have been traversed. If so, the process ends, otherwise go to step 2).

4. The method according to claim 1, characterized in that: The hierarchical sequence completes the execution order of each node in the graph structure of initialization and model instantiation, forming a service instance for the combined execution of multiple models, data processing components, and post-processing components, including: 1) Initialize the layer order execution list L; 2) Generate a topologically sorted list T(G) of the directed acyclic graph G; 3) Select nodes Ti from T(G) in sequence; 4) If the node Ti is a model node, go to step 5), otherwise go to step 6); 5) Obtain the current instantiation model and previous node information of the node Ti, and insert them into the hierarchical execution list L; 6) If the node Ti is a data processing node or a post-processing node, go to step 7), otherwise report an error and go to step 8); 7) Instantiate the processing node or post-processing node of the data and insert it into the layer sequence execution list; 8) Determine whether the nodes in T(G) have been traversed. If so, the service instance is constructed. Otherwise, go to step 3).

5. A multi-machine deployment scheduling device based on model service automation, characterized in that: include: An initialization module is used to obtain a JSON description file representing a model service workflow through a client and issue a service initialization request, and the scheduling server receives the request and allocates inference server resources; A scheduling module, used to allocate inference server resources that can meet the needs of each client through a scheduling server based on a dynamic weighted least connection algorithm; An inference module, used to return the request calling IP and port of the inference server corresponding to the inference server resource to the client, perform inference on the request calling service instance through the client, and return the inference result; The reasoning module is further used for: Get a JSON description file that effectively represents the model service workflow; Processing the JSON description file to obtain information about the graph structure of the model service workflow; Complete the initialization of the graph structure and model instantiation according to the information of the graph structure; The execution order of each node in the graph structure of the hierarchical completion initialization and model instantiation is formed to form a service instance of the combined execution of multiple models and data processing components and post-processing components. In this case, multiple models and data processing components and post-processing components are encapsulated and combined into a directed acyclic graph in a serial or parallel manner to achieve rapid arrangement and reorganization of models in complex task scenarios. Obtain request data from the client, call the service instance to perform reasoning on the request data, and return the reasoning result.

6. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-machine deployment and scheduling method based on model service automation as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-machine deployment scheduling method based on model service automation as described in any one of claims 1 to 4 is implemented.

Images