An inference request scheduling method and system, an electronic device, and a storage medium

Abstract

The application relates to the field of data processing, in particular to an inference request scheduling method and system, an electronic device and a storage medium, which comprises the following steps: collecting inference task requests; extracting text feature vectors of the inference task requests; obtaining feature similarities between the inference task requests according to the text feature vectors; predicting the length of output texts generated after processing the inference task requests as predicted lengths by using the text feature vectors; grouping the inference task requests according to the predicted lengths and the feature similarities; and scheduling the inference task requests according to the grouping. Compared with the prior art, the application can effectively make the predicted length difference of the inference task requests in each group smaller and the feature similarity larger by grouping the inference task requests according to the predicted lengths and the feature similarities, thereby effectively improving the processing efficiency of the inference task requests in the group.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and more particularly, to an inference request scheduling method and system, an electronic device, and a storage medium. BACKGROUND

[0002] With the continuous development of language processing models, more and more people begin to use language processing models to process problems. As the number of people using language processing models increases and the frequency of people asking questions increases, the language processing model system needs to respond to a large number of question requests at the same time. Therefore, the throughput of the system needs to be improved.

[0003] In order to improve the throughput of the system and reduce the computational cost of a single inference task processing, in the existing inference task scheduling technology of the language processing model, a batch processing scheduling strategy is usually used to combine multiple inference requests to a system GPU (Graphics Processing Unit) of the language processing model for processing. The inference requests are usually batched according to the text length of the inference requests. Although this can improve the processing efficiency of the inference requests to a certain extent, the improvement is limited.

[0004] Therefore, there is an urgent need for a scheduling method that can more effectively improve the processing efficiency of inference requests. SUMMARY

[0005] The present application provides an inference request scheduling method, system, electronic device, and storage medium for more effectively improving the processing efficiency of inference requests of a language processing model.

[0006] According to a first aspect of the present application, an inference request scheduling method is provided, the method comprising:

[0007] collecting inference task requests;

[0008] extracting text feature vectors of the inference task requests;

[0009] obtaining feature similarities between the inference task requests according to the text feature vectors;

[0010] using the text feature vectors to predict the length of the output text generated after processing the inference task requests as a predicted length;

[0011] grouping the inference task requests according to the predicted length and the feature similarities;

[0012] scheduling and processing the inference task requests according to the grouping.

[0013] Optionally, the obtaining of the feature similarity between each of the inference task requests comprises:

[0014] processing the text feature vectors by using the pre-trained similarity prediction model to obtain the feature similarity between each of the inference task requests;

[0015] The similarity prediction model comprises a double tower network layer and a similarity regression calculation network layer connected in sequence.

[0016] Optionally, the training of the similarity prediction model comprises:

[0017] collecting a plurality of first historical inference task requests and first historical grouping information of the first historical inference task requests;

[0018] extracting features of the first historical inference task requests to obtain first historical text feature vectors of each of the first historical inference task requests;

[0019] adding first historical grouping labels to the corresponding first historical text feature vectors according to the first historical grouping information, and constructing a first training set according to the first historical text feature vectors after the first historical grouping labels are added;

[0020] training the similarity prediction model according to the first training set until a preset first loss function converges, to obtain the pre-trained similarity prediction model.

[0021] Optionally, the first loss function is:

[0022]

[0023] wherein, denotes the first loss function, denotes the similarity between the i th first historical text feature vector and the j th first historical text feature vector in the first training set, denotes the first historical grouping label of the i th first historical text feature vector and the j th first historical text feature, or wherein, denotes that the first historical inference task requests corresponding to the i th and j th first historical text feature vectors were historically scheduled in the same group, denotes that the first historical inference task requests corresponding to the i th and j th first historical text feature vectors were historically scheduled in different groups.

[0024] Optionally, the grouping of the inference task requests according to the predicted length and the feature similarity comprises:

[0025] processing the prediction length and the feature similarity through a pre-trained grouping prediction model to obtain a prediction grouping label of each of the inference task requests;

[0026] grouping the inference task requests according to the prediction grouping label of each of the inference task requests.

[0027] Optionally, the training of the grouping prediction model comprises:

[0028] collecting a plurality of second historical inference task requests and second historical grouping information of the second historical inference task requests;

[0029] performing feature extraction on the second historical inference task requests to obtain second historical text feature vectors of the second historical inference task requests;

[0030] obtaining historical feature similarities between each of the second historical inference task requests according to the second historical text feature vectors;

[0031] predicting lengths of second historical output texts generated after processing the second historical inference task requests as historical prediction lengths using the second historical text feature vectors;

[0032] adding second historical grouping labels to corresponding historical prediction lengths and historical feature similarities according to the second historical grouping information, and constructing a second training set according to the historical prediction lengths and the historical feature similarities after adding the second historical grouping labels;

[0033] training the grouping prediction model according to the second training set until a preset second loss function converges, to obtain the pre-trained grouping prediction model.

[0034] Optionally, the second loss function is constructed according to a weighting of each of the historical prediction length differences in the second training set, wherein the weight of each historical prediction length difference is calculated according to the corresponding second historical grouping label and the historical feature similarity.

[0035] Optionally, the second loss function is:

[0036]

[0037] In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. In the formula, L represents the second loss function, L represents the i-th historical prediction length in the second training set, L represents the j-th historical prediction length in the second training set, and represents a normalization parameter. For grouping weight, denoted as:

[0038]

[0039] In the formula, denotes the historical feature similarity between the second historical inference task request corresponding to the i th and j th historical prediction length in the second training set; denotes the second historical grouping label of the i th and j th historical prediction length in the second training set, Or wherein, denotes that the second historical inference task request corresponding to the i th and j th historical prediction length is historically scheduled in the same group, denotes that the second historical inference task request corresponding to the i th and j th historical prediction length is historically scheduled in different groups.

[0040] Optionally, the step of predicting the length of the output text generated after processing the inference task request as a predicted length by using the text feature vector comprises:

[0041] processing the text feature vector by using the pre-trained output length prediction model to predict the length of the output text generated after processing the inference task request as a predicted length;

[0042] The output length prediction model comprises an encoding network layer, a pooling network layer and a regression network layer connected in sequence.

[0043] Optionally, the step of scheduling and processing the inference task request according to the grouping comprises:

[0044] For each group, a first scheduling index of the corresponding group is obtained according to the maximum value of the predicted length of the inference task request;

[0045] According to the first scheduling index, a resource processing space of the corresponding group is determined;

[0046] The inference task request is scheduled to the corresponding resource processing space according to the grouping.

[0047] Optionally, the step of determining a resource processing space of the corresponding group according to the first scheduling index comprises:

[0048] A filling index of each group is determined by using the first scheduling index;

[0049] The first scheduling index is adjusted according to the filling index to obtain a second scheduling index of each group;

[0050] The second scheduling indicator of the group and the number of inference task requests in the group are used to calculate the required space size of the corresponding group, and the resource processing space is allocated to the corresponding group according to the space size.

[0051] Optionally, the text feature vector of the inference task request is extracted, including:

[0052] The word pieces of the inference task request are extracted, and the number of word pieces is determined;

[0053] The encoding vector of the word piece is obtained, and the embedding vector of the inference task request is obtained according to the encoding vector of each word piece;

[0054] The classification information of the word piece is extracted, and the topic vector of the inference task request is obtained according to the classification information of each word piece;

[0055] The text feature vector of the inference task request is obtained according to the number of word pieces, the embedding vector and the topic vector of the inference task request.

[0056] According to a second aspect of the present application, a kind of inference request scheduling system is provided, and the system includes:

[0057] Task acquisition module, for collecting inference task request;

[0058] Feature extraction module, for extracting the text feature vector of the inference task request;

[0059] Similarity prediction module, for obtaining the feature similarity between each inference task request according to the text feature vector;

[0060] Length prediction module, for using the text feature vector, the length of the output text generated after processing the inference task request is predicted as predicted length;

[0061] Task grouping module, for grouping the inference task request according to the predicted length and the feature similarity;

[0062] Task scheduling module, for scheduling processing the inference task request according to the group.

[0063] According to a third aspect of the present application, an electronic device is provided, including:

[0064] Memory, for storing one or more computer programs;

[0065] Processor, when the one or more computer programs are executed by the processor, realize the inference task scheduling method of the first aspect.

[0066] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for causing a processor to implement the inference task scheduling method of the first aspect when executed.

[0067] Based on any one of the above aspects, the inference task scheduling method, system, electronic device and computer storage medium provided by the embodiments of the present application can obtain the feature similarity between inference task requests and the predicted length of the output text generated by the inference task request, and group the inference task requests based on the feature similarity and the predicted length, so that the predicted length difference between the inference task requests in each group is smaller and the correlation is higher, thereby effectively improving the processing efficiency of each group when scheduling each group. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0069] Figure 1 An illustrative application scenario diagram of the inference request scheduling method provided by the present embodiment.

[0070] Figure 2 A step flowchart of the inference request scheduling method provided by the present embodiment.

[0071] Figure 3 A step flowchart of the similarity prediction model training provided by the present embodiment.

[0072] Figure 4 A step flowchart of the grouping prediction model training provided by the present embodiment.

[0073] Figure 5 A step flowchart of the resource processing space determination provided by the present embodiment.

[0074] Figure 6 A functional module diagram of the inference request scheduling system provided by the present embodiment.

[0075] Figure 7 A device structure diagram of the electronic device provided by the present embodiment. DETAILED DESCRIPTION

[0076] The drawings in the present application are only used for illustrative description, and cannot be understood as a limitation of the present application. In order to better illustrate the following embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual product size; it is understandable for those skilled in the art that some well-known structures in the drawings and their descriptions may be omitted.

[0077] In order to enable those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0078] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0079] With the continuous development of language processing models, more and more people begin to use language processing models to process problems, and with the increase in the number of people using language processing models and the increase in the frequency of asking questions, the language processing model system needs to respond to a large number of question requests at the same time, and therefore the throughput of the system needs to be improved.

[0080] In order to improve the throughput of the system and reduce the computational cost of single inference task processing, in the existing inference task scheduling technology of the language processing model, a batch processing scheduling strategy is usually adopted, that is, multiple inference requests are combined to a system GPU of the language processing model for processing. The existing batch processing is usually based on the text length of the inference request.

[0081] However, when processing the inference request, the factors affecting the processing efficiency are not only the text length of the input inference request, but also the text length required to be output by the inference request and the relevance between the inference requests in the same batch. Therefore, although the inference requests can be batched according to the text length of the input inference request, the processing efficiency of the inference request can be improved to a certain extent, but the degree of improvement is limited.

[0082] Therefore, there is an urgent need for a scheduling method that can more effectively improve the processing efficiency of inference requests.

[0083] The embodiment provides a technical solution that can solve the above problems. The specific embodiments of the application are described in detail below with reference to the drawings.

[0084] Exemplarily, an application scenario of a task processing method provided by the embodiment of the application is shown. As shown in the figure, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100. Figure 1

[0085] It can be understood that the server 100 can be a stand-alone electronic device, or a cluster composed of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a vehicle-mounted terminal, etc., but is not limited thereto.

[0086] In an implementable manner, the server 100 and the terminal 200 can respectively execute the task processing method provided by the embodiment of the application, or alternatively, the task processing method provided by the embodiment of the application is partially executed in the server 100 and partially executed in the terminal 200.

[0087] As shown in the figure, the embodiment provides an inference request scheduling method, which can include the following steps: Figure 2

[0088] S1: Collecting an inference task request;

[0089] In the embodiment, the inference task request input by a user can be collected based on a preset period, and the inference task request collected in each period is processed by scheduling. The inference task request can be a text request.

[0090] In an alternative implementation, information of an input user and time of a request and the like data can also be collected, and the inference task request and the information of the input user and the time of the request are constructed as standardized data, so as to facilitate statistical analysis on related data of the inference task request.

[0091] S2: Extracting a text feature vector of the inference task request;

[0092] In the embodiment, the extraction of the text feature vector can include:

[0093] S21: Extracting a word piece of the inference task request, and determining a quantity of the word piece;

[0094] ​​In the embodiment, a preset word segmenter can be used to segment the text of the reasoning task request to obtain each word unit of the reasoning task request, and the number of word units segmented from the reasoning task request is counted. The word unit can be a word, a sub-word, a character, or a punctuation mark of the reasoning task request.

[0095] S22: An encoding vector of each word unit is obtained, and an embedding vector of the reasoning task request is obtained according to the encoding vectors of the word units.

[0096] In the embodiment, an encoder can be used to process each word unit of the reasoning task request to obtain an encoding vector of each word unit, and then the encoding vectors of the word units are processed through a pre-trained text processing model to infer an embedding vector of the reasoning task request.

[0097] S23: Classification information of each word unit is extracted, and a topic vector of the reasoning task request is obtained according to the classification information of each word unit.

[0098] In the embodiment, the classification information of each word unit, such as a classification label, can be used to infer a topic vector of the reasoning task request; the topic vector can reflect the text style and main content information of the reasoning task request.

[0099] S24: A text feature vector of the reasoning task request is obtained according to the number of word units, the embedding vector and the topic vector of the reasoning task request.

[0100] In the embodiment, the number of word units can be converted into a vector representation, and the vector representation of the number of word units, the embedding vector and the topic vector of the reasoning task request are spliced to obtain the text feature vector.

[0101] The text feature vector is obtained by using the number of word units, the embedding vector and the topic vector of the reasoning task request, so that the text feature vector contains information of multiple angles of the reasoning task request. On the one hand, the text feature vector can be better used to determine the correlation between each reasoning task request, and on the other hand, the text feature vector can be better used to predict the text length of the result obtained after processing the reasoning task request.

[0102] In an optional implementation, the text feature vector can further include a user preference vector. The user preference vector can be extracted by analyzing historical reasoning task requests input by a corresponding user and output results after processing the historical reasoning task requests, and reflecting the user style and demand preference vector as the user preference vector.

[0103] S3: Obtain feature similarities between each of the reasoning task requests according to the text feature vectors;

[0104] It can be understood that the higher the correlation in the reasoning task requests in the same group, the more relevant information can be used between the reasoning task requests when processing the reasoning task requests in the group, thereby improving the efficiency of processing the reasoning requests. Therefore, by calculating the feature similarities between the reasoning task requests, the reasoning task requests with high feature similarities are divided into the same group, which can effectively improve the efficiency of processing the reasoning task requests in the same group.

[0105] In the embodiment, the feature similarities between the reasoning task requests can be obtained by:

[0106] The text feature vectors are processed by the pre-trained similarity prediction model to obtain the feature similarities between each of the reasoning task requests;

[0107] In an optional implementation, the similarity prediction model can include a double tower network layer and a similarity regression calculation network layer connected in sequence. The double tower network layer includes two parallel neural network branches, and the text feature vectors of two reasoning task requests for which the feature similarities are to be calculated are input as a group of to-be-processed data to the double tower network layer for processing, wherein each neural network branch processes one text feature vector in the to-be-processed data. It can be understood that the double tower network layer can input multiple groups of to-be-processed data at the same time for parallel processing.

[0108] The similarity regression calculation network layer includes a feature concatenation layer and a first MLP (Multilayer Perceptron, multi-layer perceptron) regression network layer. The feature concatenation layer is used to concatenate the text feature vectors processed by the two neural network branches to obtain a concatenated vector. The first MLP regression network layer is used to learn features of the concatenated vector, predict the similarity between the two text feature vectors therein, and output the similarity through a Sigmoid function as the feature similarity between the corresponding reasoning task requests. It can be understood that, corresponding to the double tower network layer, if multiple groups of to-be-processed data are input to the double tower network layer for parallel processing, the feature concatenation layer concatenates the corresponding text feature vectors output in parallel by the two neural network branches in the double tower network layer to obtain multiple concatenated vectors, and then inputs the obtained multiple concatenated vectors to the first MLP regression network layer for parallel feature learning, and outputs the feature similarities corresponding to each group of to-be-processed data.

[0109] In an example, the text feature vectors of the two inference task requests requiring feature similarity calculation can be processed into a neural network branch of a double tower network layer of the similarity prediction model respectively, and then the text feature vectors processed by the neural network branch are spliced by a feature splicing layer to obtain corresponding spliced features, and finally the first MLP regression network layer is used for prediction to obtain the feature similarity between the inference task requests.

[0110] In the embodiment, as shown in Figure 3 The training of the similarity prediction model can include:

[0111] S31: Collect a plurality of first historical inference task requests and first historical grouping information of the first historical inference task requests;

[0112] In the embodiment, the first historical grouping information includes whether the collected historical inference task requests are grouped into the same group in the historical calling process. By collecting the first historical grouping information of the first historical inference task requests, the similarity prediction model can learn in combination with the historical grouping situation, so as to facilitate grouping of inference task requests with higher similarity into the same group in the subsequent process.

[0113] S32: Feature extraction is performed on the first historical inference task requests to obtain first historical text feature vectors of the first historical inference task requests;

[0114] It can be understood that in the embodiment, the step of obtaining the first historical text feature vectors of the first historical inference task requests in the step S32 is similar to the step of extracting the text feature vectors of the inference task requests in the step S2, and the extraction of the first historical text feature vectors in the step S32 can be realized by referring to the description of the step S2, which will not be described further herein.

[0115] S33: A first historical grouping label is added to the corresponding first historical text feature vectors according to the first historical grouping information, and a first training set is constructed according to the first historical text feature vectors after adding the first historical grouping label;

[0116] In the embodiment, the first historical grouping label can be added to two first historical text feature vectors requiring feature similarity prediction, and is used to represent whether the two first historical text feature vectors are grouped into the same group for scheduling in the historical scheduling process.

[0117] For example, for the i th first historical text feature vector and the j th first historical text feature vector in the first training set, the first historical grouping label can be represented as wherein, or , indicates that the first historical reasoning task request corresponding to the i-th and j-th first historical text feature vectors is historically scheduled in the same group, indicates that the first historical reasoning task request corresponding to the i-th and j-th first historical text feature vectors is historically scheduled in different groups.

[0118] S34: training the similarity prediction model according to the first training set until a preset first loss function converges, to obtain a pre-trained similarity prediction model.

[0119] In this embodiment, the first loss function can be represented as:

[0120]

[0121] In the formula, indicates the first loss function, indicates the similarity between the i-th first historical text feature vector and the j-th first historical text feature vector in the first training set, indicates the first historical grouping label of the i-th first historical text feature vector and the j-th first historical text feature; as described above, or wherein, indicates that the first historical reasoning task request corresponding to the i-th and j-th first historical text feature vectors is historically scheduled in the same group, indicates that the first historical reasoning task request corresponding to the i-th and j-th first historical text feature vectors is historically scheduled in different groups.

[0122] S4: using the text feature vector to predict the length of the output text generated after processing the reasoning task request as a predicted length;

[0123] In this embodiment, the predicted length can be obtained as:

[0124] By processing the text feature vector through the pre-trained output length prediction model, the length of the output text generated after processing the reasoning task request is predicted as a predicted length.

[0125] In an optional implementation, the output length prediction model can include an encoding network layer, a pooling network layer and a regression network layer connected in sequence. The encoding network layer can be set as a neural network including a Transformer encoder, for performing encoding processing on the text feature vector; the pooling network layer is configured to perform pooling processing on the text feature vector after the encoding processing; and the regression network layer is configured to perform learning prediction based on the text feature vector after the pooling processing, to obtain the predicted length.

[0126] In an implementation, the training of the output length prediction model can include:

[0127] S41: Collecting a plurality of third historical reasoning task requests and historical output lengths corresponding to the third historical reasoning task requests;

[0128] S42: Performing feature extraction on the third historical reasoning task requests to obtain third historical text feature vectors of the third historical reasoning task requests;

[0129] It can be understood that in the present embodiment, the first historical text feature vector can be obtained by referring to the above step S32, and will not be described further herein.

[0130] S43: Adding a length label to the corresponding third historical text feature vector according to the historical output length, and constructing a third training set according to the third historical text feature vector after adding the length label;

[0131] S44: Training the output length prediction model according to the third training set until a preset third loss function converges, to obtain a pre-trained output length prediction model.

[0132] In an implementation, the third loss function can be obtained based on the mean square error between the output result of the output length prediction model and the corresponding length label, and can be expressed as:

[0133]

[0134] In the formula, the is the third loss function, N is the number of third historical text feature vectors in the third training set, represents a predicted output result of the i-th third historical text feature vector in the third training set in the output length prediction model, represents a length label corresponding to the i-th first historical text feature vector.

[0135] S5: Grouping the reasoning task requests according to the predicted length and the feature similarity;

[0136] In the embodiment, the grouping step can be:

[0137] The prediction length and the feature similarity are processed by the pre-trained grouping prediction model to obtain a prediction grouping label of each inference task request; and the inference task requests are grouped according to the prediction grouping label of each inference task request.

[0138] In an implementation, for each of the collected inference task requests, a corresponding prediction grouping label is added between two inference task requests to indicate whether the two inference task requests can be divided into the same group, and the grouping of the inference task requests is determined according to the prediction grouping label of each inference task request.

[0139] In an implementation, as shown in Figure 4 The training of the grouping prediction model can include:

[0140] S51: Collecting a plurality of second historical inference task requests and second historical grouping information of the second historical inference task requests;

[0141] It can be understood that the collection of the second historical inference task requests and the acquisition of the second historical grouping information in the step S51 can be implemented by referring to the step S31, and will not be further described here.

[0142] S52: Extracting features of the second historical inference task requests to obtain second historical text feature vectors of the second historical inference task requests;

[0143] It can be understood that the extraction of the second historical text feature vectors in the step S52 can be implemented by referring to the step S32, and will not be further described here.

[0144] S53: Acquiring a historical feature similarity between each of the second historical inference task requests according to the second historical text feature vectors;

[0145] In an optional implementation, the historical feature similarity can be obtained by processing the second historical text feature vectors by the similarity prediction model, and the acquisition step of the historical feature similarity can be implemented by referring to the training step of the similarity prediction model or the acquisition step of the feature similarity, and will not be further described here.

[0146] S54: Predicting a length of a second historical output text generated after processing the second historical inference task requests as a historical prediction length by using the second historical text feature vectors;

[0147] In an alternative embodiment, the historical prediction length can be obtained by processing the second historical text feature vector by the output length prediction model described above. The historical prediction length can be obtained by referring to the training step of the output length prediction model described above, and thus will not be described further.

[0148] S55: adding a second historical grouping label to the corresponding historical prediction length and historical feature similarity according to the second historical grouping information, and constructing a second training set according to the historical prediction length and historical feature similarity after adding the second historical grouping label;

[0149] It can be understood that the second historical grouping label is the same as the first historical grouping label. In this embodiment, the second historical grouping label can be added to two second historical text feature vectors that need to be predicted for feature similarity, to indicate whether the two second historical text feature vectors are grouped into the same group for scheduling in the historical scheduling process.

[0150] S56: training the grouping prediction model according to the second training set until a preset second loss function converges, to obtain a pre-trained grouping prediction model.

[0151] In an alternative embodiment, the second loss function is constructed according to the weights of the historical prediction length difference values in the second training set, wherein the weight of each historical prediction length difference value is calculated according to the corresponding second historical grouping label and the historical feature similarity. The second loss function can be represented as:

[0152]

[0153] In the formula, denotes the second loss function, denotes the i-th historical prediction length in the second training set, denotes the j-th historical prediction length in the second training set; denotes a normalization parameter, is a Sigmoid function; is a grouping weight, and is represented as:

[0154]

[0155] In the formula, denotes the historical feature similarity between the i-th and j-th historical prediction length in the second training set; denotes the second historical grouping label of the i-th and j-th historical prediction length in the second training set, or wherein, indicates that the second historical inference task request corresponding to the i-th and j-th historical prediction length is scheduled in the same group historically, indicates that the second historical inference task request corresponding to the i-th and j-th historical prediction length is scheduled in different groups historically.

[0156] It can be understood that, in the process of training the grouping prediction model, the similarity prediction model can be used to obtain the historical feature similarity, and the output length prediction model can be used to obtain the historical feature similarity. In a preferred embodiment, the similarity prediction model, the output length prediction model and the grouping prediction model can be jointly trained end-to-end, and the joint loss function of the joint training can be:

[0157]

[0158] wherein the , and are weights corresponding to the similarity prediction model, the output length prediction model and the grouping prediction model, respectively.

[0159] S6: scheduling the inference task request according to the grouping.

[0160] In this embodiment, the step S6 can include:

[0161] S61: for each group, obtaining a first scheduling index of the corresponding group according to the maximum value of the predicted length of the inference task request;

[0162] S62: determining a resource processing space of the corresponding group according to the first scheduling index;

[0163] S63: scheduling the inference task request to the corresponding resource processing space according to the grouping.

[0164] In this embodiment, the resource processing space can be a video memory space in a GPU for processing the inference task request, and the first scheduling index refers to the maximum video memory space occupied when processing the inference task request in the group. By using the maximum value of the predicted length of the inference task request in the group, the resource processing space can be allocated to the corresponding group, which can ensure that the resource processing space can perform parallel processing on all inference task requests in the corresponding group, thereby effectively improving the processing efficiency of the inference task request.

[0165] Further, as shown in Figure 5 , the step S62 can include:

[0166] A1: determining a padding indicator of each group by using the first scheduling indicator;

[0167] A2: adjusting the first scheduling indicator according to the padding indicator to obtain a second scheduling indicator of each group;

[0168] A3: calculating a space size required by a corresponding group by using the second scheduling indicator of the group and the number of inference task requests in the group, and allocating a resource processing space for the corresponding group according to the space size.

[0169] In order to be able to cope with the error when processing the inference task request, further guarantee that all inference task requests in the group can be effectively processed, when allocating a resource processing space for the group, a certain amount of margin needs to be considered.

[0170] At the same time, when the inference task requests in the group are processed in the resource processing space, a certain amount of padding data needs to be added to each inference task request in the group, so that all inference task requests in the group are unified to the same length, and then the inference task requests can be more conveniently processed in parallel in the resource processing space. When the inference task request is completed, the padding data added in the scheduling processing of the inference task request is removed.

[0171] Therefore, in the embodiment, the space size required by the group can be obtained by multiplying the second scheduling indicator and the number of inference task requests in the group, and then the resource processing space can be allocated for the group according to the required space size, so that all inference task requests in the group can be processed in parallel in the resource processing space, and the amount of padding data required to be added in the group is minimized, reducing the waste of computing resources.

[0172] In an optional embodiment, the first scheduling indicator can be multiplied by a preset multiple as the padding indicator; the preset multiple can be set to 5%-10%.

[0173] In an optional embodiment, the method further comprises obtaining the source of the inference task request according to the information of the input user in the standardized data, and feeding back the processing result according to the source of the inference task request after scheduling and processing the inference task request according to the group.

[0174] In the embodiment, an inference request scheduling system is also provided, as shown in Figure 6 The system can comprise:

[0175] The task acquisition module 11 is configured to collect inference task requests.

[0176] In the embodiment, the task obtaining module 11 can be configured to perform Figure 2 The step S1 is shown, and the specific description of the task obtaining module 11 can refer to the description of the step S1.

[0177] The feature extraction module 12 is configured to extract a text feature vector of the reasoning task request.

[0178] In the embodiment, the feature extraction module 12 can be configured to perform Figure 2 The step S2 is shown, and the specific description of the feature extraction module 12 can refer to the description of the step S2.

[0179] The similarity prediction module 13 is configured to obtain a feature similarity between each of the reasoning task requests according to the text feature vector.

[0180] In the embodiment, the similarity prediction module 13 can be configured to perform Figure 2 The step S3 is shown, and the specific description of the similarity prediction module 13 can refer to the description of the step S3.

[0181] The length prediction module 14 is configured to predict a length of an output text generated after processing the reasoning task request as a predicted length, by using the text feature vector.

[0182] In the embodiment, the length prediction module 14 can be configured to perform Figure 2 The step S4 is shown, and the specific description of the length prediction module 14 can refer to the description of the step S4.

[0183] The task grouping module 15 is configured to group the reasoning task requests according to the predicted length and the feature similarity.

[0184] In the embodiment, the task grouping module 15 can be configured to perform Figure 2 The step S5 is shown, and the specific description of the task grouping module 15 can refer to the description of the step S5.

[0185] The task scheduling module 16 is configured to schedule the reasoning task requests for processing according to the grouping.

[0186] In the embodiment, the task scheduling module 16 can be configured to perform Figure 2 The step S6 is shown, and the specific description of the task scheduling module 16 can refer to the description of the step S6.

[0187] The embodiment of the application further provides an electronic device, and the structure thereof is as shown in Figure 7The electronic device can be the electronic device of the present embodiment Figure 1 The server 100 or the terminal 200 shown.

[0188] The electronic device includes a memory 21, a processor 22, a communication module 23, and an input / output interface 24, and optionally, the memory 21, the processor 22, the communication module 23, and the input / output interface 24 can be connected and communicated through a bus 25.

[0189] The memory 21 is configured to store one or more computer programs and transmit the codes of the computer programs to the processor 22; when the one or more computer programs are executed by the processor 22, the inference request scheduling method in the present embodiment is implemented.

[0190] Optionally, the electronic device can be connected to a network through the communication module 23 to communicate with other devices such as terminals or servers through the network to realize data interaction. The electronic device can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers, or other types of computers. The electronic device can also be various forms of mobile terminals, such as smartphones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0191] Optionally, the electronic device can connect the required input / output devices such as keyboards, display devices, etc. through the input / output interface 24, and the electronic device itself can have a display device and can also be externally connected to other display devices through the input / output interface 24. Optionally, the storage device such as hard disk, etc. can be connected through the input / output interface 24, so that the data in the electronic device can be stored in the storage device, or the data in the storage device can be read, and the data in the storage device can also be stored in the memory 21. It can be understood that the input / output interface 24 can be a wired interface or a wireless interface. According to different actual application scenarios, the devices connected to the input / output interface 24 can be a component of the electronic device, or an external device connected to the electronic device when needed.

[0192] Optionally, the memory 21 can be a volatile memory and / or a non-volatile memory, the volatile memory can be a random access memory, and the non-volatile memory can be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0193] Optionally, the computer program stored in the processor 22 can be divided into one or more modules stored in the memory 21 and executed by the processor 22 to complete the method provided by the embodiment. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0194] Optionally, the processor 22 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 22 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and any appropriate controller, microcontroller, processor, etc. The processor 22 executes various methods and processes of the embodiment, for example, an inference request scheduling method of the embodiment.

[0195] Optionally, the bus 25 can include a channel for transmitting information. According to different functions, the bus 25 can be divided into an address bus, a data bus, a control bus, etc.

[0196] In an optional implementation, the embodiment further provides a computer storage medium having a computer program stored thereon, which enables the computer to execute the method of the method embodiment when executed by the computer. Part or all of the computer program can be loaded and / or installed on the memory 21 of the electronic device. When the computer program is executed by the processor 22, one or more steps of the inference request scheduling method of the embodiment can be executed.

[0197] Optionally, the computer readable storage medium can be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0198] Obviously, the above embodiments of the present application are only examples for clearly illustrating the technical solutions of the present application, and are not intended to limit the specific embodiments of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for scheduling inference requests, characterized in that, The method comprises: collecting inference task requests; extracting text feature vectors of the inference task requests; obtaining feature similarities between the inference task requests according to the text feature vectors; predicting lengths of output texts generated after processing the inference task requests as predicted lengths by using the text feature vectors; grouping the inference task requests according to the predicted lengths and the feature similarities; obtaining first scheduling indicators of each group according to maximum values of the predicted lengths of the inference task requests; determining padding indicators of each group by using the first scheduling indicators; adjusting the first scheduling indicators according to the padding indicators to obtain second scheduling indicators of each group; calculating space sizes required by each group by using the second scheduling indicators of the group and numbers of inference task requests in the group, and allocating resource processing spaces for each group according to the space sizes; scheduling the inference task requests to the resource processing spaces according to the groups.

2. The method of claim 1, wherein, The method comprises: processing the text feature vectors by using a pre-trained similarity prediction model to obtain the feature similarities between the inference task requests; the similarity prediction model comprises a double-tower network layer and a similarity regression calculation network layer connected in sequence.

3. The method of claim 2, wherein, The training of the similarity prediction model comprises: collecting a plurality of first historical inference task requests and first historical grouping information of the first historical inference task requests; extracting features of the first historical inference task requests to obtain first historical text feature vectors of the first historical inference task requests; adding first historical grouping labels to the first historical text feature vectors according to the first historical grouping information, and constructing a first training set according to the first historical text feature vectors after adding the first historical grouping labels; training the similarity prediction model according to the first training set until a preset first loss function converges, to obtain a pre-trained similarity prediction model.

4. The method of claim 3, wherein, The first loss function is: In the formula, denotes a first loss function, denotes a similarity between an i-th first historical text feature vector and a j-th first historical text feature vector in the first training set, denotes a first historical grouping label of the i-th first historical text feature vector and the j-th first historical text feature, or wherein, denotes that the i-th and j-th first historical text feature vectors correspond to the same group in historical scheduling of the first historical inference task request, denotes that the i-th and j-th first historical text feature vectors correspond to different groups in historical scheduling of the first historical inference task request.

5. The method of claim 1, wherein, The method comprises: processing the predicted lengths and the feature similarities by using a pre-trained grouping prediction model to obtain predicted grouping labels of the inference task requests; grouping the inference task requests according to the predicted grouping labels of the inference task requests.

6. The method of claim 5, wherein, The training of the grouping prediction model comprises: collecting a plurality of second historical inference task requests and second historical grouping information of the second historical inference task requests; extracting features of the second historical inference task requests to obtain second historical text feature vectors of the second historical inference task requests; obtaining historical feature similarities between the second historical inference task requests according to the second historical text feature vectors; predict, by using the second historical text feature vector, a length of a second historical output text generated after processing a second historical reasoning task request as a historical predicted length; add a second historical grouping label to the corresponding historical predicted length and historical feature similarity according to the second historical grouping information, and construct a second training set according to the historical predicted length and historical feature similarity after adding the second historical grouping label; train the grouping prediction model according to the second training set until a preset second loss function converges, to obtain a pre-trained grouping prediction model.

7. The method of claim 6, wherein, The second loss function is constructed according to a weight of each historical predicted length difference value in the second training set, wherein the weight of each historical predicted length difference value is calculated according to the corresponding second historical grouping label and the historical feature similarity.

8. The method of claim 6, wherein, The second loss function is: wherein denotes a second loss function, denotes the i-th historical prediction length in the second training set, denotes the j-th historical prediction length in the second training set; denotes a normalization parameter, is a Sigmoid function; is a grouping weight, denoted as: In the formula, denotes the historical feature similarity between the second historical inference task requests corresponding to the i-th and j-th historical prediction lengths in the second training set; denotes the second historical grouping label of the i-th and j-th historical prediction lengths in the second training set, or wherein, denotes that the second historical inference task requests corresponding to the i-th and j-th historical prediction lengths are historically scheduled in the same group, denotes that the second historical inference task requests corresponding to the i-th and j-th historical prediction lengths are historically scheduled in different groups.

9. The method of claim 1, wherein, The prediction of the length of the output text generated after processing the reasoning task request by using the text feature vector as the predicted length includes: processing the text feature vector by using the pre-trained output length prediction model to predict the length of the output text generated after processing the reasoning task request as the predicted length; The output length prediction model includes an encoding network layer, a pooling network layer and a regression network layer connected in sequence.

10. The method of claim 1-9, wherein, The extraction of the text feature vector of the reasoning task request includes: extracting word pieces of the reasoning task request and determining a quantity of the word pieces; obtaining an embedding vector of the reasoning task request according to the embedding vector of each word piece; extracting classification information of the word pieces and obtaining a topic vector of the reasoning task request according to the classification information of each word piece; obtaining the text feature vector of the reasoning task request according to the quantity of the word pieces, the embedding vector and the topic vector of the reasoning task request.

11. An inference request scheduling system, comprising: The system includes: a task acquisition module configured to collect reasoning task requests; a feature extraction module configured to extract text feature vectors of the reasoning task requests; a similarity prediction module configured to obtain feature similarities between the reasoning task requests according to the text feature vectors; a length prediction module configured to predict lengths of output texts generated after processing the reasoning task requests by using the text feature vectors as predicted lengths; a task grouping module configured to group the reasoning task requests according to the predicted lengths and the feature similarities; a task scheduling module configured to, for each group, obtain a first scheduling index of the corresponding group according to a maximum value of the predicted lengths of the reasoning task requests in the group, determine a padding index of each group by using the first scheduling index, adjust the first scheduling index according to the padding index to obtain a second scheduling index of each group, calculate a required space size of the corresponding group by using the second scheduling index of the group and a quantity of the reasoning task requests in the group, and allocate a resource processing space for the corresponding group according to the space size; and schedule the reasoning task requests to the resource processing spaces according to the groups.

12. The reasoning request scheduling system of claim 11, wherein, The feature similarity between each of the inference task requests is obtained according to the text feature vectors, including: processing the text feature vectors by a pre-trained similarity prediction model to obtain the feature similarity between each of the inference task requests; the similarity prediction model includes a double tower network layer and a similarity regression calculation network layer connected in sequence.

13. The system of claim 12, wherein the inference request scheduling system is further configured to: The training of the similarity prediction model includes: collecting a plurality of first historical inference task requests and first historical grouping information of the first historical inference task requests; performing feature extraction on the first historical inference task requests to obtain first historical text feature vectors of each of the first historical inference task requests; adding first historical grouping labels to the corresponding first historical text feature vectors according to the first historical grouping information, and constructing a first training set according to the first historical text feature vectors after adding the first historical grouping labels; training the similarity prediction model according to the first training set until a preset first loss function converges, to obtain a pre-trained similarity prediction model.

14. An electronic device, comprising: including: a memory for storing one or more computer programs; a processor, when the one or more computer programs are executed by the processor, implements an inference request scheduling method according to any one of claims 1-10.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the processor execute and implement an inference request scheduling method according to any one of claims 1-10.

Images