Target service scheduling method, device and system and electronic equipment

Abstract

The invention provides a target service scheduling method, device and system and electronic equipment. According to the method, under the condition that the reasoning request which is sent by the client and carries the input data of the target service is obtained, the target reasoning graph comprising the multiple model components is determined, and the scheduling sequence of the model components in the target reasoning graph is further determined; processing the input data of the target service in sequence according to the scheduling sequence of each model component to obtain a data processing result; the input data of the target service is sequentially processed according to the scheduling sequence of the multiple model components in the target reasoning graph corresponding to the target service, so that the data processing result of the target service is determined, and multi-model target service scheduling is achieved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a target service scheduling method, device, system and electronic device. Background Art

[0002] The current method for multi-model service scheduling can deploy multiple model components in one scheduling unit to share computing resources and improve the elastic scaling capabilities of multi-objective services. However, this method does not have data exchange and dependencies between multiple model components deployed in one scheduling unit. In fact, it is still a single-objective service scheduling. It cannot achieve target service scheduling when the model structure is relatively complex and there is data interaction and dependency between model components. Summary of the invention

[0003] In view of this, the present application provides a target service scheduling method, device, system and electronic device to realize multi-model reasoning service scheduling.

[0004] The technical solutions provided by this application are as follows:

[0005] According to an embodiment of the first aspect of the present application, a target service scheduling method is provided, the method comprising:

[0006] Obtaining an inference request sent by a client, wherein the inference request carries input data of a target service;

[0007] Determine a target reasoning graph corresponding to the target service; the target reasoning graph includes multiple model components, the model component includes at least one of a model, a converter and a combiner, the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of multiple model components;

[0008] Determining a scheduling order for each model component in the target reasoning graph;

[0009] The input data of the target service is processed in sequence according to the scheduling order of each model component to obtain a data processing result.

[0010] Optionally, the target reasoning graph includes input-output relationships between multiple model components, and determining the scheduling order of each model component in the target reasoning graph includes:

[0011] Generate a model file based on the input-output relationship of multiple model components included in the acquired target reasoning graph, wherein the model file includes input-output information between the model components;

[0012] Reasoning graph information is generated based on the model file, and the reasoning graph information includes a scheduling order of each model component in the target reasoning graph. The scheduling order of each model component in the reasoning graph information is obtained based on input and output information of each model component in the target reasoning graph included in the model file.

[0013] Optionally, the method for acquiring the target reasoning graph includes:

[0014] Obtaining a constructed first directed acyclic graph DAG model, where the first DAG model includes multiple reference components;

[0015] Pruning the first DAG model to remove reference components used for reading, writing and model evaluation from the multiple reference components, to obtain a pruned second DAG model;

[0016] For each reference component of the second DAG model, according to the attribute of the reference component, replace the reference component in the second DAG model with a model component of the attribute;

[0017] The input-output relationship between multiple model components is determined according to the input-output relationship between the reference components included in the second DAG model, and the target reasoning graph is generated according to the input-output relationship between the multiple model components.

[0018] Optionally, for each reference component of the second DAG model, replacing the reference component in the second DAG model with a model component having the attribute according to the attribute of the reference component includes:

[0019] If the attribute of the reference component is an algorithm component, the reference component is replaced by the model; the algorithm component is used to process the data input into the component according to the algorithm included in the algorithm component;

[0020] If the attribute of the reference component is a data processing component, the reference component is replaced by the converter; the data processing component is used to convert the format of the data input into the component;

[0021] If the attribute of the reference component is a data merging component, the reference component is replaced by the combiner; the data merging component is used to merge multiple data input into the component.

[0022] Optionally, the method further includes:

[0023] Obtaining a data return request sent by a client, wherein the data return request carries a return data type; the return data type includes one or more of the following: the number of inference request receptions, the inference request response time, and the inference request response success rate;

[0024] Returning the target data corresponding to the return data type to the client;

[0025] Among them, the number of inference requests received is obtained by recording the number of inference requests sent by the client, the inference request response time is obtained by recording the time from obtaining the inference request sent by the client to obtaining the data processing result, and the inference request response success rate is obtained by the ratio of the number of inference requests that obtain the data processing result to the total number of inference requests sent by the client.

[0026] Optionally, the processing the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result includes:

[0027] The scheduling component forwards the inference request to the model component at the first position in the scheduling order according to the scheduling order of each model component included in the inference graph information, so that the model component at the first position in the scheduling order processes the input data carried by the inference request to obtain the first output data, and returns the first output data to the scheduling component;

[0028] The scheduling component forwards the inference request to the model component at the next position in the scheduling order, and uses the first output data as input data, so that the model component at the next position in the scheduling order processes the first output data to obtain second output data, and returns the second output data to the scheduling component;

[0029] The scheduling component uses the second output data as the first output data, returns the inference request to the model component at the next position in the scheduling order, uses the first output data as input data, so that the model component at the next position in the scheduling order processes the first output data to obtain the second output data, until the inference request is forwarded to the model component at the end of the scheduling order, so that the model component at the end of the scheduling order processes the first output data to obtain the data processing result;

[0030] Among them, the scheduling component is deployed in a container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to the multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

[0031] Optionally, each model component further corresponds to a language model encapsulator, and the language model encapsulator is used to convert the first type of data into a second type of data, where the second type refers to a data type supported by the model component; the method further includes:

[0032] Before inputting any data into any model component, if it is detected that the data type of the data is different from the second type, the data is input into the language model encapsulator corresponding to the model component so that the language model encapsulator converts the data type of the data into the second type.

[0033] According to an embodiment of the second aspect of the present application, a target service scheduling device is provided, the device comprising:

[0034] An acquisition unit, configured to acquire an inference request sent by a client, wherein the inference request carries input data of a target service;

[0035] a determination unit, configured to determine a target reasoning graph corresponding to the target service; the target reasoning graph includes a plurality of model components, the model components include at least one of a model, a converter, and a combiner, the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of a plurality of model components;

[0036] Determining a scheduling order for each model component in the target reasoning graph;

[0037] The processing unit is used to process the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result.

[0038] According to an embodiment of the third aspect of the present application, a target service scheduling system is provided, the system comprising:

[0039] A scheduling component, used to execute the method described in the first aspect;

[0040] A model component, comprising at least one of a model, a converter and a combiner, wherein the model is used to perform a prediction task, the converter is used to pre-process received data, and the combiner is used to integrate output data of multiple model components;

[0041] Among them, the scheduling component is deployed in a container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to the multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

[0042] According to an embodiment of the fourth aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0043] It can be seen from the above technical scheme that when obtaining an inference request sent by a client carrying input data of a target service, the present application determines a target inference graph including multiple model components, further determines the scheduling order of each model component in the target inference graph, and processes the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result; the present application processes the input data of the target service in sequence according to the scheduling order of multiple model components in the target inference graph corresponding to the target service to determine the data processing result of the target service, thereby realizing multi-model target service scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 A flowchart of a target service scheduling method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the recommendation system structure provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the road speed prediction system structure provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of converting a reference component into a model component provided in an embodiment of the present application;

[0049] Figure 5 Another reference component provided in an embodiment of the present application is converted into a schematic diagram of a model component;

[0050] Figure 6 A schematic diagram of the operation flow of the target service scheduling method provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the scheduling component structure provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of a language model encapsulator provided in an embodiment of the present application;

[0053] Fig. 9 A structural diagram of a target service scheduling system proposed in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0055] Fig.11 A structural diagram of a target service scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0057] Please refer to Figure 1 , Figure 1 A flowchart of a target service scheduling method provided in an embodiment of the present application.

[0058] First pass Figure 2 The background of the target scheduling method proposed in this application is introduced.

[0059] The following describes two typical scenarios.

[0060] Please refer to Figure 2 , Figure 2 A schematic diagram of the recommendation system structure provided in an embodiment of the present application.

[0061] like Figure 2 As shown in the figure, in this recommendation system, the dotted box below is a directed acyclic graph (DAG) model. The client inputs user features into the model through a standardized interface, so that the model can determine the initial recommendation results from the candidate product library based on the user features through the recall layer, and then obtain the final product recommendation results based on auxiliary information (such as user behavior feedback, product attributes, product strategy, etc.) through the sorting layer and return them to the client.

[0062] Please refer to Figure 3 , Figure 3 A schematic diagram of the road speed prediction system structure provided in an embodiment of the present application.

[0063] like Figure 3 As shown, in the road speed prediction system, the dotted box below is the DAG overall model. The road characteristics of the past hour are input into the model through the client, so that the model pre-processes the road characteristics based on auxiliary information (such as some information used to describe the scene: urban road network data, weather data, holidays, entertainment activity information, and historical traffic road information, etc.), and predicts through the core model. After data post-processing such as data conversion, the obtained data is returned to the road speed prediction result for the next 5 minutes.

[0064] It can be seen that in both of the above scenarios, the DAG model is published as a multi-model service to meet the needs of large-scale requests and data flows.

[0065] like Figure 1 As shown, the method may include the following steps:

[0066] Step 101: Obtain an inference request sent by a client.

[0067] In this embodiment, the inference request may carry input data of the target service. The client sends the inference request to predict related functions based on the input data through the target service, obtain a data processing result, and return the data processing result to the client.

[0068] At this point, the description of step 101 ends, and step 102 is executed next.

[0069] Step 102: determine the target reasoning graph corresponding to the target service.

[0070] In this embodiment, each target service corresponds to a target reasoning graph, and the target reasoning graph includes multiple model components. The model component may include at least one of a model, a converter, and a combiner.

[0071] Among them, the model is used to perform prediction tasks, the converter is used to preprocess the received data, and the combiner is used to integrate the output data of multiple model components.

[0072] Specifically, model Model: can be various types of machine learning models, such as classifiers, regression models, or deep learning networks.

[0073] Transformer: used to process and transform data, such as data cleaning, feature extraction, or format conversion, to ensure that the input meets the requirements of the model or the output result format meets user needs.

[0074] Combiner: It is used to combine the outputs of all upstream nodes and return them after performing specific processing.

[0075] In this embodiment, the method for obtaining the target reasoning graph may include:

[0076] Obtaining a constructed first directed acyclic graph DAG model, where the first DAG model includes multiple reference components;

[0077] Pruning the first DAG model to remove reference components used for reading, writing and model evaluation from the multiple reference components, to obtain a pruned second DAG model;

[0078] For each reference component of the second DAG model, according to the attribute of the reference component, replace the reference component in the second DAG model with a model component of the attribute;

[0079] The input-output relationship between the multiple model components is determined according to the input-output relationship between the reference components included in the second DAG model, and the target reasoning graph is generated according to the input-output relationship between the multiple model components.

[0080] In this embodiment, the first DAG model may be a DAG model constructed by a user and needs to be published as a multi-model service. The first DAG model generally includes multiple reference components.

[0081] In this embodiment, after obtaining the first DAG model, since the DAG structure diagram of the first DAG model constructed by the user is usually large and complex, it is first necessary to prune the first DAG model constructed by the user, remove the read-write components and the model evaluation components, and obtain the second DAG model.

[0082] For example, components such as data reading components, data storage components, and hyperparameter optimization components are removed, and only the core computing parts involved in model reasoning are retained. The process of pruning the first DAG model is a common method in related technologies and will not be repeated here.

[0083] Furthermore, for each reference component in the second DAG model, the reference component is replaced with a model component with the same attribute according to the attribute of the reference component, that is, for reference components at different positions in the second DAG model, these reference components are further abstracted into the following three types: model, converter and combiner. The process of replacing the reference component with the model component will be described in detail below and will not be repeated here.

[0084] After all reference components in the second DAG model are replaced with model components, the input-output relationship between multiple model components can also be determined based on the input-output relationship between the reference components included in the second DAG model, and a target reasoning graph can be generated based on the input-output relationship between the multiple model components.

[0085] It can be seen that the obtained target reasoning graph includes multiple models and records the input-output relationship between multiple model components.

[0086] The process of replacing the reference component with the model component is described in detail below.

[0087] As an embodiment, for each reference component of the second DAG model, replacing the reference component in the second DAG model with a model component of the attribute according to the attribute of the reference component includes:

[0088] If the attribute of the reference component is an algorithm component, the reference component is replaced by the model; the algorithm component is used to process the data input into the component according to the algorithm included in the algorithm component;

[0089] If the attribute of the reference component is a data processing component, the reference component is replaced with a converter; the data processing component is used to convert the format of the data input into the component;

[0090] If the attribute of the reference component is a data merging component, the reference component is replaced with a combiner; the data merging component is used to merge multiple data input into the component.

[0091] In this embodiment, the reference component in the second DAG model may be replaced by a model component with the same attribute, that is, the same function.

[0092] For example, the attribute of the reference component is an algorithm component. Since the algorithm component is used to process the data input into the component according to the algorithm included in the algorithm component, it can be replaced by a model component "model" that is also used to perform prediction tasks.

[0093] For another example, the attribute of the reference component is a data processing component. Since the data processing component is used to convert the format of the data input into the component, it can be replaced by a model component "converter" which is also used to pre-process the received data.

[0094] For another example, the attribute of the reference component is a data merging component. Since the data merging component is used to merge multiple data input into the component, it can be replaced by a model component "combiner" which is also used to integrate the output data of multiple model components.

[0095] The following two specific examples introduce the process of converting reference components into model components:

[0096] Please refer to Figure 4 , Figure 4 A reference component provided in an embodiment of the present application is converted into a model component schematic.

[0097] like Figure 4 As shown, Figure 4 In the DAG model training diagram on the left, Iris-Dataset is the iris data set. The data set is input into the random forest algorithm for prediction, and the prediction results are further input into the model prediction evaluation component for evaluation.

[0098] When processing the above DAG model to obtain the target reasoning graph, the DAG model can be pruned first, and the data input component (i.e., Iris-Dataset in the figure) and the model prediction evaluation component can be removed from the DAG model. Since the remaining random forest is an algorithm component, it will be replaced by the "model", and the single-component scheduling graph on the left (including only one "model") will be obtained.

[0099] Figure 4In the DAG model training diagram on the right side of the figure, Iris-Dataset is the iris data set. The data set is input into the data type conversion component for data type conversion, and the data conversion result is further output.

[0100] When processing the above DAG model to obtain the target reasoning graph, the DAG model can be pruned first, and the data input component (i.e., Iris-Dataset in the figure) and the data output component can be removed from the DAG model. Since the remaining data type conversion component is a data processing component, it will be replaced with a "converter" to obtain the single-component scheduling graph on the right (including only one "converter").

[0101] This concludes Figure 4 Description.

[0102] Please refer to Figure 5 , Figure 5 Another reference component provided in an embodiment of the present application is converted into a model component schematic diagram.

[0103] like Figure 5 As shown, the process of converting the complex DAG model training graph given on the left into the target reasoning graph may include:

[0104] First, "traffic data" and "weather data" are pruned as data input components.

[0105] The data type conversion is abstracted as converter 1, and the feature derivation is abstracted as converter 2; the data splitting is abstracted as converter 3, and the regression model and the CNN model are abstracted as model 1 and model 2 respectively; the feature fusion is abstracted as combiner 1; and the data post-processing is abstracted as combiner 2.

[0106] In this embodiment, through the above-mentioned conversion mechanism, the dependency relationship (ie, input-output relationship) between the reference components can be parsed, and the dependency relationship can be converted into the input-output relationship between the model components, and the complex DAG can be reconstructed into a target reasoning graph.

[0107] This concludes Figure 5 Description.

[0108] At this point, the description of step 102 ends, and step 103 is executed next.

[0109] Step 103, determining the scheduling order of each model component in the target reasoning graph.

[0110] In this embodiment, after the target reasoning graph is determined, a model file may be generated according to the input-output relationship of multiple model components included in the target reasoning graph, and the model file includes input-output information between the model components.

[0111] Further based on the model file, the reasoning graph information is generated, the reasoning graph information includes the scheduling order of each model component in the target reasoning graph, and the scheduling order of each model component in the reasoning graph information is obtained based on the input and output information of each model component in the target reasoning graph included in the model file.

[0112] As an embodiment, the model file may be a yaml file, and the reasoning graph information is generated through the yaml file, and the corresponding target reasoning graph is deployed as a target service in the kubernetes cluster according to the scheduling order of each model component included in the reasoning graph information.

[0113] Specifically, taking the model file as a yaml file as an example, in this embodiment, it can be deployed based on the K8S (kubernetes) cloud environment. K8S can recognize the custom yaml file type to deploy the entire target application.

[0114] As an example, Figure 5 The yaml file corresponding to the target reasoning graph shown is shown in the following table:

[0115]

[0116]

[0117] Table 1

[0118] As shown in Table 1, the yaml file records the scheduling order and input and output of each model component.

[0119] The input and output in the yaml file represent the overall input and output of the target reasoning graph, respectively, and the steps represent the scheduling order of each model component in the target reasoning graph.

[0120] The meanings of the fields in each step in steps are as follows: name indicates the model component name; inputs indicates which model components' outputs make up the input data of the model component. The current model component will not be executed until all the model components corresponding to its input data are executed. If the input of the current model component is one of the outputs of other model components, outputs is required to indicate which output of which model component is used as the input of the current model component.

[0121] As an example, take the following content in Table 1 as an example:

[0122] "-name:transformer3

[0123] inputs:

[0124] transformer2

[0125] -name:model1

[0126] inputs:

[0127] -transformer3.outputs.OUTPUT1”

[0128] The above content means: for converter 3, its input is the output of converter 2. Since converter 2 has only one output, there is no need to further describe converter 2; for model 1, its input is the output of converter 3. Since converter 3 has two outputs, it is further defined that the input of model 1 is the first output (OUTPUT1) of converter 3. The other contents of the above yaml file are similar and will not be repeated here.

[0129] It can be seen that the model file generated according to the target reasoning graph actually includes the input and output information between each model component. Further, the reasoning graph information can be determined according to the model file, that is, the scheduling order of each model component can be obtained according to the input and output information.

[0130] For example, according to the above yaml file, the scheduling order of each model component can be determined as follows: Figure 5 The scheduling order from top to bottom.

[0131] At this point, the description of step 103 ends, and step 104 is executed next.

[0132] Step 104, processing the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result.

[0133] In this embodiment, after the model file is generated, the corresponding pod can be generated after deployment according to the model file. The containers contained in the pod are all model components in the target reasoning graph and a scheduling component engine responsible for data scheduling. Among them, pod is the basic deployment unit of Kubernetes (K8S, an open source container orchestration platform for automated deployment, management and expansion of containerized applications), and a pod is a collection of one or more containers.

[0134] In this embodiment, the input data of the target service is processed in sequence according to the scheduling order of each model component, and the method for obtaining the data processing result may specifically include:

[0135] The scheduling component forwards the inference request to the model component at the top of the scheduling order according to the scheduling order of each model component included in the inference graph information, so that the model component at the top of the scheduling order processes the input data carried by the inference request to obtain the first output data, and returns the first output data to the scheduling component;

[0136] The scheduling component forwards the inference request to the model component in the next position in the scheduling order, and uses the first output data as input data, so that the model component in the next position in the scheduling order processes the first output data to obtain second output data, and returns the second output data to the scheduling component;

[0137] The scheduling component uses the second output data as the first output data, returns and forwards the inference request to the model component in the next position in the scheduling order, uses the first output data as input data, so that the model component in the next position in the scheduling order processes the first output data to obtain the second output data, until the inference request is forwarded to the model component in the last position in the scheduling order, so that the model component in the last position in the scheduling order processes the first output data to obtain the data processing result;

[0138] Among them, the scheduling component is deployed in the container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

[0139] In this embodiment, each target service corresponds to a pod. Each pod includes a scheduling component and multiple model components. The scheduling component is used to forward the input data in the reasoning request to each model component in sequence for processing according to the scheduling order of each model component in the reasoning graph information to obtain the data processing result.

[0140] Below through Figure 6 The operation process of the target service scheduling method proposed in this application is described.

[0141] Please refer to Figure 6 , Figure 6 A schematic diagram of the operation flow of the target service scheduling method provided in an embodiment of the present application.

[0142] like Figure 6 As shown, in this embodiment, the external request is first routed by the gateway to the scheduling component engine.

[0143] The scheduling component sends the request to subsequent model components (such as transformer, model, combiner, etc.) in the scheduling order in the reasoning graph information according to the model reasoning graph structure.

[0144] After receiving the request, each model component processes it and returns the result to the scheduling component;

[0145] The scheduling component decides which component to pass the data to next based on the stored inference graph information until the processing is completed.

[0146] Finally, the dispatch component collects the output of the last component and returns it to the requesting client.

[0147] In this embodiment, during the entire complex multi-objective service reasoning process, the scheduling component is responsible for managing the scheduling order of data between components and its request and response flows based on the reasoning graph information. Each model component does not need to know which model component its input and output come from or are sent to. It only needs to receive the data input by the scheduling component, process the input data and return the processed data to the scheduling component.

[0148] This concludes the description of step 104.

[0149] As an embodiment, the method proposed in the embodiment of the present application may also include:

[0150] Get the data return request sent by the client, which carries the return data type; the return data type includes one or more of the following: the number of inference requests received, the inference request response time, and the inference request response success rate;

[0151] Return the target data corresponding to the return data type to the client;

[0152] Among them, the number of inference requests received is obtained by recording the number of inference requests sent by the client, the inference request response time is obtained by recording the time from obtaining the inference request sent by the client to obtaining the data processing result, and the inference request response success rate is obtained by the ratio of the number of inference requests that obtain the data processing result to the total number of inference requests sent by the client.

[0153] In this embodiment, each time an inference request sent by a client is received, the inference request sent by the client may be counted once to count the number of received inference requests.

[0154] The time when the inference request sent by the client is obtained and the time when the data processing result is obtained may also be recorded, and the difference between the two times may be used to determine the inference request response time.

[0155] In addition, the success rate of inference request responses may be determined by the ratio of the number of inference requests that obtain data processing results to the total number of inference requests sent by the client.

[0156] It should be noted that the target data recorded above are all statistics for the same target service. Since multiple target services may exist at the same time, the target data corresponding to each target service may be counted separately.

[0157] In this embodiment, the scheduling component is also used to record the input and output of any model in a lightweight manner to facilitate error troubleshooting and debugging of intermediate results, and to control the traffic in the inference graph to achieve effects such as model weighting and model comparison. For example, in the process of processing the input data of the target service in sequence according to the scheduling order of each model component, the time it takes for each model component to process the data can also be recorded, such as recording the time from when the model component receives the input data to when it outputs the processed data, to determine the calling status of each model component.

[0158] Please refer to Figure 7 , Figure 7 A schematic diagram of the scheduling component structure provided in an embodiment of the present application.

[0159] like Figure 7 As shown in the figure, the inference graph controller is the core of the entire scheduling component. It is responsible for parsing the inference graph structure in the model file and converting it into the corresponding inference graph for management. It is responsible for the scheduling and data transmission of each component in the inference graph, and dynamically routes it to the corresponding component container according to information such as the component name. During the model reasoning process, multiple models without dependencies can be executed in parallel through an asynchronous communication mechanism, thereby improving the speed of real-time model reasoning. At the same time, the scheduling component also includes some commonly used modules such as data cache module, log collector module, traffic controller module, service monitor module, etc., which will not be repeated here.

[0160] As an embodiment, each model component further corresponds to a language model encapsulator, and the language model encapsulator is used to convert the first type of data into the second type of data, and the second type refers to the data type supported by the model component; the method further includes:

[0161] Before inputting any data into any model component, if it is detected that the data type of the data is different from the second type, the data is input into the language model encapsulator corresponding to the model component so that the language model encapsulator converts the data type of the data into the second type.

[0162] In this embodiment, in the model reasoning service invention, in order to support the normal execution and mixed scheduling of multiple types of language models, corresponding wrappers can also be designed for various languages ​​to package the user's model and coordinate the normal flow of data in cross-language situations.

[0163] Regarding the processing of input and output data types, the embodiments of the present application can design relevant commonly used codecs such as pandas, numpy, spark dataframe, java dataframe, string, etc., to more clearly convert the input data included in the request information directly into the types supported by the model component. The user only needs to input data when calling the target service, and the language wrapper will automatically convert the data type. The user does not need to convert the input data format into the data type supported by the model component before inputting it.

[0164] As an embodiment, as shown in Table 2, Table 2 is an example of converting input data into a pandas dataframe format:

[0165]

[0166] Table 2

[0167] Please refer to Figure 8 , Figure 8 A schematic diagram of a language model encapsulator provided in an embodiment of the present application.

[0168] like Figure 8 As shown, it includes a python language model wrapper, including commonly used codecs such as pandas, numpy, string, etc. that can be designed in the embodiment of the present application, and a scala language model wrapper, including commonly used codecs such as sparkdataframe, string, etc.

[0169] There may also be different issues regarding the data types supported by each model component. In the embodiments of the present application, a model saving and model loading format that is common in its language is specially set for each language model encapsulator. As long as the model saved in this way can be correctly loaded by the encapsulator, it provides a variety of customized target service capabilities. For this type of customized target service, the codecs of various language model encapsulators can be used to automatically convert the data types of input and output data. Users do not need to worry about how the data format is handled across languages, and only need to input the data into the interface corresponding to the target service.

[0170] Similarly, after the inference model completes reasoning, that is, after obtaining the final data processing result, the data processing result needs to be returned to the client. At this time, the data type corresponding to the data processing result can also be converted into a data type that can be recognized by the client or specified by the client through the language model encapsulator. This application does not impose any restrictions on this.

[0171] This concludes Figure 1 Description of the flow chart of the target service scheduling method.

[0172] When obtaining an inference request sent by a client and carrying input data of a target service, the present application determines a target inference graph including multiple model components, further determines a scheduling order of each model component in the target inference graph, and processes the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result; the present application processes the input data of the target service in sequence according to the scheduling order of multiple model components in the target inference graph corresponding to the target service to determine the data processing result of the target service, thereby realizing multi-model target service scheduling.

[0173] This application reconstructs a complex multi-objective service reasoning graph structure by splitting the complex DAG graph structure, and designs a scheduling component responsible for controlling data flow in the complex reasoning graph, and then designs a one-click method for publishing corresponding multi-model services from the DAG graph provided by the user.

[0174] Through this method, users can directly train DAG Figure 1 One-click publishing is the corresponding multi-model inference service. During this process, users do not need to write code related to the target service, which greatly reduces the technical threshold for publishing the target service.

[0175] At the same time, the method proposed in this application supports more complex reasoning graph structures. As long as the reasoning graph is a DAG structure, it can be converted into a corresponding multi-model reasoning service, and the user does not need to care about the complex conversion logic and dependencies of this part; the newly added scheduling component in this application controls the calling order of multiple models in the multi-model reasoning service through the reasoning graph, and can lightly record the input and output of any model, which is more conducive to error troubleshooting and debugging of intermediate results. In addition, it can also control the traffic in the reasoning graph.

[0176] In addition, the present application designs a method that supports publishing an inference graph composed of multiple language type models as a target service by containerizing the target service runtime environment and providing corresponding wrappers for various language type models.

[0177] Please refer to Fig. 9 , Fig. 9 A structural diagram of a target service scheduling system proposed in an embodiment of the present application.

[0178] like Fig. 9 As shown, the system includes:

[0179] Scheduling component for execution Figure 1 Methods in

[0180] A model component, including at least one of a model, a converter, and a combiner, wherein the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of multiple model components;

[0181] Among them, the scheduling component is deployed in the container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

[0182] Among them, the scheduling component and the model component have been described in detail above and will not be repeated here.

[0183] This concludes Fig. 9 Description.

[0184] Please refer to Fig.10 , Fig.10 It is a schematic structural diagram of an electronic device proposed in an embodiment of the present application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a terminal interaction device at the logical level. Of course, in addition to the software implementation method, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0185] Please refer to Fig.11 , Fig.11 This is a structural diagram of a target service scheduling device proposed in an embodiment of the present application, which is applied to the scheduling component engine deployed in any pod in Kubernetes. Fig.11 As shown, the device may include an acquisition unit 1101, a processing unit 1102, and an output unit 1103. Specifically, the device includes:

[0186] The acquisition unit 1101 is used to acquire an inference request sent by a client, where the inference request carries input data of a target service;

[0187] A determination unit 1102 is used to determine a target reasoning graph corresponding to a target service; the target reasoning graph includes multiple model components, the model component includes at least one of a model, a converter, and a combiner, the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of multiple model components;

[0188] Determine the scheduling order of each model component in the target reasoning graph;

[0189] The processing unit 1103 is used to process the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result.

[0190] Optionally, the target reasoning graph includes input-output relationships between multiple model components, and the determining unit 1102 is specifically configured to:

[0191] Generate a model file based on the input-output relationship of multiple model components included in the acquired target reasoning graph, where the model file includes input-output information between the model components;

[0192] The reasoning graph information is generated based on the model file, and the reasoning graph information includes the scheduling order of each model component in the target reasoning graph. The scheduling order of each model component in the reasoning graph information is obtained based on the input and output information of each model component in the target reasoning graph included in the model file.

[0193] Optionally, the determining unit is further configured to:

[0194] Obtaining a constructed first directed acyclic graph DAG model, where the first DAG model includes multiple reference components;

[0195] Pruning the first DAG model to remove reference components used for reading, writing and model evaluation from the multiple reference components, to obtain a pruned second DAG model;

[0196] For each reference component of the second DAG model, according to the attribute of the reference component, replace the reference component in the second DAG model with a model component of the attribute;

[0197] The input-output relationship between the multiple model components is determined according to the input-output relationship between the reference components included in the second DAG model, and the target reasoning graph is generated according to the input-output relationship between the multiple model components.

[0198] Optionally, the determining unit 1102 is further configured to:

[0199] If the attribute of the reference component is an algorithm component, the reference component is replaced by the model; the algorithm component is used to process the data input into the component according to the algorithm included in the algorithm component;

[0200] If the attribute of the reference component is a data processing component, the reference component is replaced with a converter; the data processing component is used to convert the format of the data input into the component;

[0201] If the attribute of the reference component is a data merging component, the reference component is replaced with a combiner; the data merging component is used to merge multiple data input into the component.

[0202] Optionally, the processing unit 1103 is further configured to:

[0203] Get the data return request sent by the client, which carries the return data type; the return data type includes one or more of the following: the number of inference requests received, the inference request response time, and the inference request response success rate;

[0204] Return the target data corresponding to the return data type to the client;

[0205] Among them, the number of inference requests received is obtained by recording the number of inference requests sent by the client, the inference request response time is obtained by recording the time from obtaining the inference request sent by the client to obtaining the data processing result, and the inference request response success rate is obtained by the ratio of the number of inference requests that obtain the data processing result to the total number of inference requests sent by the client.

[0206] Optionally, the processing unit 1103 is specifically configured to:

[0207] The scheduling component forwards the inference request to the model component at the top of the scheduling order according to the scheduling order of each model component included in the inference graph information, so that the model component at the top of the scheduling order processes the input data carried by the inference request to obtain the first output data, and returns the first output data to the scheduling component;

[0208] The scheduling component forwards the inference request to the model component in the next position in the scheduling order, and uses the first output data as input data, so that the model component in the next position in the scheduling order processes the first output data to obtain second output data, and returns the second output data to the scheduling component;

[0209] The scheduling component uses the second output data as the first output data, returns and forwards the inference request to the model component in the next position in the scheduling order, uses the first output data as input data, so that the model component in the next position in the scheduling order processes the first output data to obtain the second output data, until the inference request is forwarded to the model component in the last position in the scheduling order, so that the model component in the last position in the scheduling order processes the first output data to obtain the data processing result;

[0210] Among them, the scheduling component is deployed in the container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

[0211] Optionally, each model component further corresponds to a language model encapsulator, and the language model encapsulator is used to convert the first type of data into the second type of data, where the second type refers to the data type supported by the model component; the processing unit 1103 is also used to:

[0212] Before inputting any data into any model component, if it is detected that the data type of the data is different from the second type, the data is input into the language model encapsulator corresponding to the model component so that the language model encapsulator converts the data type of the data into the second type.

[0213] So far, completed Fig.11 Description of the target service scheduling device.

[0214] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed, the method disclosed in the above example of the present application can be implemented.

[0215] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the computer-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0216] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A target service scheduling method, characterized in that: The method includes: Obtaining an inference request sent by a client, wherein the inference request carries input data of a target service; Determine a target reasoning graph corresponding to the target service; the target reasoning graph includes multiple model components, the model component includes at least one of a model, a converter and a combiner, the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of multiple model components; Determining a scheduling order for each model component in the target reasoning graph; The input data of the target service is processed in sequence according to the scheduling order of each model component to obtain a data processing result.

2. The method according to claim 1, characterized in that The target reasoning graph includes input-output relationships between multiple model components, and determining the scheduling order of each model component in the target reasoning graph includes: Generate a model file based on the input-output relationship of multiple model components included in the acquired target reasoning graph, wherein the model file includes input-output information between the model components; Reasoning graph information is generated based on the model file, and the reasoning graph information includes a scheduling order of each model component in the target reasoning graph. The scheduling order of each model component in the reasoning graph information is obtained based on input and output information of each model component in the target reasoning graph included in the model file.

3. The method according to claim 1, characterized in that The method for obtaining the target reasoning graph includes: Obtaining a constructed first directed acyclic graph DAG model, where the first DAG model includes multiple reference components; Pruning the first DAG model to remove reference components used for reading, writing and model evaluation from the multiple reference components, to obtain a pruned second DAG model; For each reference component of the second DAG model, according to the attribute of the reference component, replace the reference component in the second DAG model with a model component of the attribute; The input-output relationship between multiple model components is determined according to the input-output relationship between the reference components included in the second DAG model, and the target reasoning graph is generated according to the input-output relationship between the multiple model components.

4. The method according to claim 3, characterized in that For each reference component of the second DAG model, replacing the reference component in the second DAG model with a model component having the attribute according to the attribute of the reference component includes: If the attribute of the reference component is an algorithm component, the reference component is replaced by the model; the algorithm component is used to process the data input into the component according to the algorithm included in the algorithm component; If the attribute of the reference component is a data processing component, the reference component is replaced by the converter; the data processing component is used to convert the format of the data input into the component; If the attribute of the reference component is a data merging component, the reference component is replaced by the combiner; the data merging component is used to merge multiple data input into the component.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining a data return request sent by a client, wherein the data return request carries a return data type; the return data type includes one or more of the following: the number of inference request receptions, the inference request response time, and the inference request response success rate; Returning the target data corresponding to the return data type to the client; Among them, the number of inference requests received is obtained by recording the number of inference requests sent by the client, the inference request response time is obtained by recording the time from obtaining the inference request sent by the client to obtaining the data processing result, and the inference request response success rate is obtained by the ratio of the number of inference requests that obtain the data processing result to the total number of inference requests sent by the client.

6. The method according to claim 1, characterized in that The step of processing the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result includes: The scheduling component forwards the inference request to the model component at the first position in the scheduling order according to the scheduling order of each model component included in the inference graph information, so that the model component at the first position in the scheduling order processes the input data carried by the inference request to obtain the first output data, and returns the first output data to the scheduling component; The scheduling component forwards the inference request to the model component at the next position in the scheduling order, and uses the first output data as input data, so that the model component at the next position in the scheduling order processes the first output data to obtain second output data, and returns the second output data to the scheduling component; The scheduling component uses the second output data as the first output data, returns the inference request to the model component at the next position in the scheduling order, uses the first output data as input data, so that the model component at the next position in the scheduling order processes the first output data to obtain the second output data, until the inference request is forwarded to the model component at the end of the scheduling order, so that the model component at the end of the scheduling order processes the first output data to obtain the data processing result; Among them, the scheduling component is deployed in a container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to the multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

7. The method according to claim 1, characterized in that Each model component also corresponds to a language model encapsulator, and the language model encapsulator is used to convert the first type of data into the second type of data, where the second type refers to the data type supported by the model component; the method further includes: Before inputting any data into any model component, if it is detected that the data type of the data is different from the second type, the data is input into the language model encapsulator corresponding to the model component so that the language model encapsulator converts the data type of the data into the second type.

8. A target service scheduling device, characterized in that: The device includes: An acquisition unit, configured to acquire an inference request sent by a client, wherein the inference request carries input data of a target service; a determination unit, configured to determine a target reasoning graph corresponding to the target service; the target reasoning graph includes a plurality of model components, the model components include at least one of a model, a converter, and a combiner, the model is used to perform a prediction task, the converter is used to preprocess received data, and the combiner is used to integrate output data of a plurality of model components; Determining a scheduling order for each model component in the target reasoning graph; The processing unit is used to process the input data of the target service in sequence according to the scheduling order of each model component to obtain a data processing result.

9. A target service scheduling system, characterized in that: The system includes: A scheduling component for executing the method according to any one of claims 1 to 7; A model component, comprising at least one of a model, a converter and a combiner, wherein the model is used to perform a prediction task, the converter is used to pre-process received data, and the combiner is used to integrate output data of multiple model components; Among them, the scheduling component is deployed in a container, different model components are deployed in different containers, the container corresponding to the scheduling component and the containers corresponding to the multiple model components are located in the same pod, each target service corresponds to a pod, and different target services correspond to different pods.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images