Automatic distribution method and apparatus for request information, and non-volatile storage medium

Abstract

Disclosed in the present application are an automatic distribution method and apparatus for request information, and non-volatile computer-readable storage medium. The method comprises: receiving massive request information, wherein each piece of request information is used for requesting the processing of a task corresponding to the request information; for each piece of request information, acquiring related information of the request information, and on the basis of the related information of the request information, determining by means of a classification model the number of tokens of output data corresponding to the request information; by means of the classification model and on the basis of the number of tokens of each piece of request information and the number of tokens of the output data, determining a request mode corresponding to the request information; and distributing each piece of request information to a corresponding cluster group on the basis of the request mode corresponding to the request information, wherein the cluster group is obtained by means of deploying, on the basis of a clustering result of request modes of historical massive request information, a plurality of large language models having different configuration modes.

Description

Automatic distribution method and device of request information, and nonvolatile storage medium

[0001] Related applications

[0002] The present application claims priority from the Chinese patent application No. 202410480091.7, filed on April 19, 2024, and entitled "Automatic distribution method and device of request information, and nonvolatile storage medium", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, and in particular to an automatic distribution method and device of request information, and a nonvolatile computer readable storage medium. BACKGROUND

[0004] With the development of deep learning and artificial intelligence technology, the application field of large model technology is becoming more and more extensive, and the parameter scale of large model is gradually becoming larger. At the same time, the memory, video memory, CPU and GPU and other computing resources occupied by large model during running are unprecedentedly huge. Therefore, how to improve the inference efficiency of large model while ensuring that the use effect is basically unchanged is the key to determine the landing of large model. In the related technology, the inference efficiency of large model is improved by the method of model parallel, which only focuses on how to perform model parallel, and does not consider the difference between the input token length and the output token length of large model. However, the throughput of large model is constrained by the input / output (IO) of video memory, and its inference throughput depends on how much batch processing can be adapted in GPU memory. If the length of input and output is not considered, it will cause serious waste of resources.

[0005] At present, there is no effective solution to the above problems. SUMMARY

[0006] The embodiments of the present application provide an automatic distribution method and device of request information, and a nonvolatile computer readable storage medium, to at least solve the technical problem of wasting resources caused by not considering the input token length and output token length of large model when improving the inference efficiency of large model.

[0007] The application provides an automatic distribution method of request information in a first aspect, comprising: receiving a plurality of pieces of request information, wherein each piece of request information is used for requesting to process a task corresponding to the request information; for each piece of request information, obtaining related information of the request information, determining a token quantity of output data corresponding to the request information according to the related information of the request information by using a classification model; automatically determining a request mode corresponding to each piece of request information according to a token quantity of the request information and a token quantity of the output data by using the classification model; and distributing each piece of request information to a corresponding cluster group according to the request mode corresponding to each piece of request information, wherein the cluster group is obtained by deploying a plurality of large language models with different configuration modes according to an automatic clustering result of request modes of a plurality of pieces of historical request information.

[0008] In some embodiments, the classification model is trained by using historical information of different target objects as training data, and the historical information of each target object at least includes: historical request information sent by the target object, a conversation frequency corresponding to each piece of historical request information, output data corresponding to each piece of historical request information, and a target identifier used for indicating that the classification model outputs a plurality of pieces of output data for one piece of request information.

[0009] In some embodiments, the request modes include: a first request mode in which the token quantity of the request information belongs to a first numerical interval and the token quantity of the output data belongs to the first numerical interval, a second request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a second numerical interval, a third request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a third numerical interval, a fourth request mode in which the token quantity of the request information belongs to a second numerical interval and the token quantity of the output data belongs to the first numerical interval, a fifth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the second numerical interval, a sixth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the third numerical interval, a seventh request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the first numerical interval, an eighth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the second numerical interval, and a ninth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the third numerical interval, wherein the third numerical interval indicates a token quantity greater than the token quantity indicated by the second numerical interval, and the second numerical interval indicates a token quantity greater than the token quantity indicated by the first numerical interval.

[0010] In some embodiments, the cluster groups are deployed by: for each piece of historical request information, determining a request mode corresponding to the historical request information according to the token quantity of the historical request information and the token quantity of the output data corresponding to the historical request information; determining a clustering result according to the request modes of the plurality of pieces of historical request information, wherein the clustering result includes: the number of cluster groups, and the proportions of different cluster groups; and deploying the plurality of large language models of different configuration modes as the plurality of cluster groups according to the clustering result.

[0011] In some embodiments, determining the clustering result according to the request modes of the plurality of pieces of historical request information includes: determining the number of types of the request modes corresponding to the plurality of pieces of historical request information as the number of cluster groups; determining the number of historical request information corresponding to each request mode; and determining the ratio of the plurality of historical request information quantities as the proportions of different cluster groups.

[0012] In some embodiments, the distributing each piece of request information to a corresponding cluster group according to the request mode corresponding to the request information comprises: for each piece of request information, determining a sum of the token quantity of the request information and the token quantity of the output data as a first target token quantity corresponding to the request information; obtaining a corresponding second target token quantity of each cluster group, wherein each second target token quantity is a maximum token quantity processed by each cluster group; and determining the cluster group to which each piece of request information is to be distributed according to the first target token quantity and the second target token quantity.

[0013] In some embodiments, the determining the cluster group to which each piece of request information is to be distributed according to the first target token quantity and the second target token quantity comprises: determining a difference value between each first target token quantity and each second target token quantity to obtain a plurality of difference values; determining a minimum positive difference value in the plurality of difference values corresponding to each first target token quantity as a target difference value corresponding to the first target token quantity; and determining the cluster group indicated by the second target token quantity corresponding to the target difference value as the cluster group to which the request information corresponding to the first target token quantity is to be distributed.

[0014] The application also provides, in a second aspect, an automatic distribution apparatus of request information, comprising: a receiving module configured to receive a plurality of pieces of request information, wherein each piece of request information is used to request processing of a task corresponding to the request information; a first determining module configured to, for each piece of request information, obtain relevant information of the request information, and determine a unit text token quantity of output data corresponding to the request information according to the relevant information of the request information by using a classification model; a second determining module configured to automatically determine a request mode corresponding to each piece of request information according to a token quantity of the request information and a token quantity of the output data by using the classification model; and a distribution module configured to distribute each piece of request information to a corresponding cluster group according to the request mode corresponding to the request information, wherein the cluster group is obtained by deploying a plurality of large language models of different configuration modes according to an automatic clustering result of request modes of a plurality of historical pieces of request information.

[0015] The application also provides, in a third aspect, a nonvolatile computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable to execute the automatic distribution method of request information of the first aspect when executed by a processor.

[0016] The application also provides, in a fourth aspect, an electronic device comprising a memory and a processor, wherein the memory has a computer program stored therein, and the processor is executable to execute the automatic distribution method of request information of the first aspect when executing the computer program.

[0017] In the embodiment of the present application, a plurality of request information is received, wherein each piece of request information is used to request processing of a task corresponding to the request information; for each piece of request information, the related information of the request information is obtained, and a classification model is used to determine the number of unit texts tokens of output data corresponding to the request information according to the related information of the request information; the token number of the request information and the token number of the output data are automatically determined by the classification model according to the request mode corresponding to each piece of request information; each piece of request information is distributed to the corresponding cluster group according to the request mode corresponding to each piece of request information; the request information is clustered by the token length of the input request information and the token length of the predicted output data, and the request information is distributed to which group according to the clustering result, so that the input token length and the output token length are increased as the consideration factors when distributing the request information, thereby achieving the technical effect of optimal allocation of the entire cluster under the condition of meeting the business and unchanged effect, and further solving the technical problems of wasting resources caused by not considering the input token length and the output token length of the large model when improving the inference efficiency of the large model. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technical solutions, the drawings needed to be used in the embodiments or related technical solution descriptions will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] FIG. 1 is a hardware structure block diagram of a computer device for implementing an automatic distribution method of request information according to an embodiment of the present application.

[0020] FIG. 2 is a step flow chart of an automatic distribution method of request information according to an embodiment of the present application.

[0021] FIG. 3 is a schematic diagram of a continuous batch processing according to an embodiment of the present application.

[0022] FIG. 4 is a structure diagram of an automatic distribution device of request information according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable personnel in the technical field to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0026] Unit text (token): refers to a piece of text or symbol divided into individual units, usually separated by spaces, punctuation marks or other separators. In natural language processing and computer programming, text is often divided into tokens for processing and analysis.

[0027] Large model: a model with a large number of parameters and high demand for computing resources. These models usually have higher complexity and can handle larger scale data and more complex tasks. For example, large neural network models in deep learning.

[0028] Small model: a model with a small number of parameters and low demand for computing resources. These models usually have lower complexity and are suitable for processing smaller scale data and simple tasks. For example, some models based on traditional machine learning methods such as logistic regression, decision tree, etc.

[0029] Large language model: a language model with larger model size and number of parameters compared to conventional language models.

[0030] In the related art, when improving the inference efficiency of the model, only how to perform model parallelism is focused on, and in the scheme of how to perform model parallelism, the input token length and the output token length requested by the user are often greatly different, therefore, there is a problem of wasting resources caused by not considering the difference between the input token length and the output token length.

[0031] According to the embodiments of the present application, a method for automatically distributing request information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed by a set of computer executable instructions, such as in a computer system. In addition, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0032] The method provided by the embodiments of the present application can be executed in a mobile terminal, a computer device or a similar computing device. FIG. 1 shows a hardware structure block diagram of a computer device for implementing the method for automatically distributing request information. As shown in FIG. 1, the computer device 10 can include one or more (as shown in FIG. 1, 102a, 102b, …, 102n) processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA)), a memory 104 for storing data, and a transmission device 106 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be a port of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 1 is only schematic, and it does not limit the structure of the above-mentioned electronic device. For example, the computer device 10 can also include more or fewer components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the computer device 10. As referred to in the embodiments of the present application, the data processing circuit serves as a processor control (for example, selection of a variable resistance terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the automatic distribution method of request information in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the automatic distribution method of request information as described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the computer device 10 through a network. The network in the above embodiments includes but is not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The network in the above embodiments can specifically include a wireless network provided by a communication provider of the computer device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0036] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer device 10.

[0037] The embodiments of the present application provide an automatic distribution method of request information capable of running in the above running environment. As shown in FIG. 2, FIG. 2 is a step flowchart of an automatic distribution method of request information, which includes the following steps S202 to S208.

[0038] Step S202: receiving a plurality of request information, wherein each request information is used to request a task corresponding to the processing request information.

[0039] In step S202, the request information input by the user is received, wherein the machine learning model processing the request information can obtain the specific content of the request information through the request information, and determine the specific business corresponding to the request information. For example, the request information is a sentence: "what is a?", and the specific content received by the machine learning model processing the request information is the sentence, and the business corresponding to the request information can be determined as a query business.

[0040] Step S204: For each piece of request information, obtain the related information of the request information, and determine the token quantity of the output data corresponding to the request information according to the related information of the request information by using the classification model.

[0041] In step S204, the machine learning model processing the request information calls the classification model to perform clustering processing on the received multiple pieces of request information after receiving the request information. Specifically, the related information of each piece of request information is input into the pre-trained classification model, and the classification model can determine the token quantity of the output data corresponding to the request information.

[0042] In some embodiments, the step of determining the token quantity of the output data corresponding to the request information according to the related information of the request information by using the classification model can include: determining the conversation times corresponding to the request information, and determining the token quantity of the request information; determining the target object sending the request information, and obtaining the historical request information sent by the target object; inputting the conversation times corresponding to the request information, the token quantity of the request information, and the historical request information sent by the target object as the related information of the request information into the classification model to obtain the token quantity of the output data output by the classification model; wherein the classification model is trained by using the historical information of different target objects as training data, and the historical information of each target object at least includes: the historical request information sent by the target object, the conversation times corresponding to each piece of historical request information, the output data corresponding to each piece of historical request information, and a target identifier for indicating that the classification model outputs multiple pieces of output data for one piece of request information.

[0043] The number of input tokens is easy to count, but the number of output tokens can only be known after the large model processes the input information and determines the output data. Therefore, in this embodiment, a pre-trained classification model is used to predict the number of tokens of the output data corresponding to the input data. In this embodiment, the classification model is trained using historical information of multiple target objects as training data, where the target object is the user / account sending the request information, and the historical information of the target object includes: the request information sent by each account / user before the current time (historical request information), the number of conversations of each historical request information, the output data corresponding to each historical request information, and the command prompt (Prompt) for guiding the model to output multiple outputs (i.e. target identifier). The target identifier Prompt is included in the input data. If a target identifier Prompt exists in a piece of input request information, it indicates that multiple output data are output for the input request information. After the classification model is trained in the above manner, when determining the number of tokens of the output data corresponding to the request information, the following steps are performed: first, obtain the relevant information of the request information. Since the classification model is trained using historical information of multiple target objects as training data, when obtaining the relevant information of the request information, first determine the number of tokens of the request information, the number of conversations corresponding to the request information, and the target object (user / account) sending the request information, and obtain the historical request information sent by the target object (user / account) to obtain the number of tokens of the output data corresponding to the request information output by the classification model as input data of the classification model. The number of conversations corresponding to the request information refers to the number of times the target object (user / account) sends the request information when communicating with the machine learning model.

[0044] Step S206: automatically determine the request mode corresponding to each request information by the classification model according to the number of tokens of the request information and the number of tokens of the output data.

[0045] In step S206, for each request information, the classification model determines the number of tokens of the output data corresponding to the request information, and further determines the request mode corresponding to the request information according to the number of tokens of the request information and the number of tokens of the output data corresponding thereto.

[0046] In some embodiments, the request mode corresponding to the request information is determined according to the token quantity of the request information and the token quantity of the output data, wherein the request mode comprises: a first request mode in which the token quantity of the request information belongs to a first numerical interval and the token quantity of the output data belongs to the first numerical interval, a second request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a second numerical interval, a third request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a third numerical interval, a fourth request mode in which the token quantity of the request information belongs to a second numerical interval and the token quantity of the output data belongs to the first numerical interval, a fifth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the second numerical interval, a sixth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the third numerical interval, a seventh request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the first numerical interval, an eighth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the second numerical interval, and a ninth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the third numerical interval; wherein the token quantity indicated by the third numerical interval is greater than the token quantity indicated by the second numerical interval, and the token quantity indicated by the second numerical interval is greater than the token quantity indicated by the first numerical interval.

[0047] The request mode of each request information is determined by the number of tokens of the request information and the number of tokens of the output data corresponding to the request information. In the embodiment, the request modes are divided into: short input short output request mode (i.e., first request mode), short input medium output request mode (i.e., second request mode), short input long output request mode (i.e., third request mode), medium input short output request mode (i.e., fourth request mode), medium input medium output request mode (i.e., fifth request mode), medium input long output request mode (i.e., sixth request mode), long input short output request mode (i.e., seventh request mode), long input medium output request mode (i.e., eighth request mode), and long input long output request mode (i.e., ninth request mode). Whether the input is short input, medium input or long input is determined by the number of tokens of the request information as input. For example, a first numerical interval corresponding to short input, a second numerical interval corresponding to medium input, and a third numerical interval corresponding to long input are defined in advance, and the type of input is determined by judging which numerical interval the number of tokens of the request information belongs to. When different numerical intervals are set, it should be ensured that the number of tokens indicated by the (first) numerical interval corresponding to short input is less than the number of tokens indicated by the (second) numerical interval corresponding to medium input, and the number of tokens indicated by the (second) numerical interval corresponding to medium input is less than the number of tokens indicated by the (third) numerical interval corresponding to long input.

[0048] Step S208: distributing each request information to the corresponding cluster group according to the request mode corresponding to each request information, wherein the cluster group is obtained by deploying large language models of different configuration modes according to the automatic clustering result of the request mode of historical request information.

[0049] In step S208, after determining the request mode of each request information by using the classification model, each request information is distributed to the corresponding cluster group. When the machine learning model for processing request information is applied to distribute the request information, the cluster group to which the request information is distributed is fixed and unchangeable, and is generated by deploying large language models of different configuration modes according to the clustering result of the request mode of the training data when the machine learning model for processing request information is trained by using a large amount of training data. The training data is a large amount of historical request information sent by multiple different users, for example, 1 TB (byte) of historical request information as training data. The configuration modes include learning rate, number of layers, number of hidden units, etc. When the same large language model is set to different numbers of hidden units, the two large language models generated after configuration are two large language models of different configuration modes. The training data of the machine learning model for processing request information is a large amount of historical request information sent by multiple different users, for example, 1 TB (byte) of historical request information as training data.

[0050] According to some embodiments of the present application, the cluster groups are deployed in the following manner: for each piece of historical request information, a request pattern corresponding to the historical request information is determined according to the number of tokens of the historical request information and the number of tokens of the output data corresponding to the historical request information; a clustering result is determined according to the request patterns of the plurality of pieces of historical request information, wherein the clustering result includes: the number of cluster groups, and the proportions of different cluster groups; and a plurality of large language models of different configuration modes are deployed as the plurality of cluster groups according to the clustering result.

[0051] In the embodiment, a large amount of historical request information sent by a plurality of different users, for example, 1 TB (byte) of historical request information as training data, is used in the process of training the machine learning model for processing request information to deploy a generated cluster group. Specifically, when the cluster group is deployed, first, the request mode corresponding to each historical request information is determined, and when the request mode is clustered, the number of request modes corresponding to the plurality of historical request information is recorded, and the number of request information corresponding to each request mode is recorded; the number of request mode categories is used as the number of cluster groups, and the ratio of the number of request information corresponding to different request modes is used as the proportion of each cluster group, and the number of cluster groups and the proportion of each cluster group together constitute the clustering result. The large language model of different configuration modes is deployed according to the above clustering result, wherein the number of cluster groups is equal to the number of configuration modes, and the proportion of each cluster group is the proportion of the large language model of different configuration modes. For example, the request mode of 10,000 historical request information is divided into: short input short output request mode (i.e. first request mode), medium input medium output request mode (i.e. fifth request mode), long input medium output request mode (i.e. eighth request mode), and long input long output request mode (i.e. ninth request mode), then the number of cluster groups is 4, and the large language model has 4 configurations, which are large language model supporting first request mode, large language model supporting fifth request mode, large language model supporting eighth request mode, and large language model supporting ninth request mode; among the 10,000 historical request information, 2000 historical request information has a request mode of short input short output request mode (i.e. first request mode), 1000 historical request information has a request mode of medium input medium output request mode (i.e. fifth request mode), 4000 historical request information has a request mode of long input medium output request mode (i.e. eighth request mode), and 3000 historical request information has a request mode of long input long output request mode (i.e. ninth request mode), then the proportion of the cluster group is equal to 2000:1000:4000:3000=2:1:4:3, so in the deployed cluster group, the number of large language models supporting first request mode: the number of large language models supporting fifth request mode: the number of large language models supporting eighth request mode: the number of large language models supporting ninth request mode = 2:1:4:3. The deployment mode (number and proportion) of the cluster group is fixed after the machine learning model for processing request information is trained, and each cluster group is identified with a corresponding number. When the machine learning model for processing request information is distributed, the machine learning model for processing request information selects the cluster group corresponding to each request information in the fixed cluster group, and then distributes it to the corresponding cluster group; or outputs the number of the cluster group corresponding to each request information to the user, and the user distributes the request information to the corresponding cluster group according to the number.

[0052] In some embodiments, the step of determining the clustering result according to the request mode of the plurality of request information can include: determining the number of categories of the request mode corresponding to the plurality of request information as the number of cluster groups; determining the number of request information corresponding to each request mode; and determining the ratio of the plurality of request information quantities as the proportion of different cluster groups.

[0053] Specifically, when clustering according to the request mode of the request information, first, determine the number of categories of the request mode corresponding to the plurality of received request information, and take the number of categories of the request mode as the number of cluster groups when dividing the cluster into cluster groups; for example, after determining that among the ten received request information, three request information correspond to a first request mode, two request information correspond to a second request mode, and five request information correspond to a third request mode, the number of categories of the request mode corresponding to the request information is three, and the cluster is divided into three cluster groups. The proportion of the number of machine learning models included in the plurality of cluster groups is determined according to the number of request information included in each request mode; for example, in the above example, three request information correspond to the first request mode, two request information correspond to the second request mode, and five request information correspond to the third request mode; the proportion of the number of machine learning models included in the (first) cluster group for processing request information of the first request mode, the number of machine learning models included in the (second) cluster group for processing request information of the second request mode, and the number of machine learning models included in the (third) cluster group for processing request information of the third request mode is 3:2:5; that is, when the cluster is divided into three cluster groups, the proportion of the number of machine learning models included in the three cluster groups is 3:2:5.

[0054] It should be noted that after dividing the cluster into multiple cluster groups, the cluster's resources need to be allocated to each cluster group. At this point, all resources provided by the cluster need to be quantified. Since the number of user requests (QPS) in most online service environments fluctuates greatly over time, the average and maximum amount of resources used after the model is deployed are set through methods such as memory paging pre-allocation and continuous batch processing to perform resource quantification. Figure 3 is a schematic diagram of continuous batch processing. As shown in Figure 3, the maximum number of tokens that can be processed by continuous batch processing is specified to be 8 (T1-T8). Request information is processed in parallel, that is, four sequences can simultaneously process request information S1 with an input token number of 3, request information S2 with an input token number of 2, request information S3 with an input token number of 3, and request information S4 with an input token number of 4. Since continuous batch processing requires that the batch file be filled as fully as possible, after outputting the output result S1' with a token number of 2 for S1 and adding the terminator End after the output result to indicate that the processing of request information S1 is complete, if there are any remaining resources in the sequence, the remaining resources are used to process other request information. For example, in Figure 3, after request information S1 is processed, the remaining resources are used to process request information S6; after request information S3 is processed, the processing result S3' is output and the terminator End is added, the remaining resources are used to process request information S5 and output the processing result S5'; after request information S4 is processed, the processing result S4' is output and the terminator End is added, the remaining resources are used to process request information S7. However, request information S2, the corresponding processing result S2', and the terminator End have already filled 8 tokens (T1-T8), which means that the resources of the sequence have been used up. Therefore, the sequence is no longer used to process other information besides S2.

[0055] According to some embodiments of the present application, the step of distributing the request information to the corresponding cluster group according to the token number of the request information and the token number of the output data may include: for each request information, determining the sum of the token number of the request information and the token number of the output data as the first target token number corresponding to the request information; obtaining the corresponding second target token number of each cluster group, wherein each second target token number is the maximum token number processed by each cluster group; and determining the cluster group to which each request information is to be distributed based on the first target token number and the second target token number.

[0056] In the embodiment, before the request information is distributed to the corresponding cluster group, it is firstly determined to which cluster group the request information should be distributed, specifically, the sum of the token number of the request information and the token number of the output data corresponding to the request information (i.e. the first target token number) is determined; then the maximum token number (i.e. the second target token number) that each cluster group can process is determined; by comparing the sum of the token number of the request information and the token number of the output data corresponding to the request information (i.e. the first target token number) with the maximum token number (i.e. the second target token number) that each cluster group can process, it is determined to which cluster group the request information is distributed. The maximum token number (i.e. the second target token number) that each cluster group can process is determined by the machine models included in each cluster group. For a cluster group, the maximum token number that each machine learning model in the cluster group can process is obtained, and the sum of the maximum token numbers that the plurality of machine learning models can process is determined as the maximum token number that the cluster group can process. The maximum token number of each machine learning model is adjustable.

[0057] According to some embodiments of the present application, the step of determining the cluster group to which each piece of request information is to be distributed according to the first target token number and the second target token number can include: determining the difference between each first target token number and each second target token number to obtain a plurality of difference values; determining the minimum positive difference value in the plurality of difference values corresponding to each first target token number as the target difference value corresponding to the first target token number; and determining the cluster group indicated by the second target token number corresponding to the target difference value as the cluster group to which the request information corresponding to the first target token number is to be distributed.

[0058] As mentioned in the above embodiment, there are cluster groups capable of processing different token numbers in the cluster group, and each request can be allocated to the cluster group closest to the sum of the token number of the request information and the token number of the output data (i.e. the first target token number) and greater than the sum, so as to optimize the performance of the entire cluster. Therefore, in the embodiment, after the sum of the token number of the request information and the token number of the output data corresponding to the request information (i.e. the first target token number) is determined, when the request information is distributed, the difference between the first target token number and the maximum token number (i.e. the second target token number) that each cluster group can process is determined, and the minimum positive difference value (i.e. the minimum positive difference value) that is greater than zero and has the minimum value is selected from the obtained plurality of difference values; the first target token number and the second target token number that generate the difference value are determined, and the request information corresponding to the first target token number is distributed to the cluster group corresponding to the second target token number.

[0059] Through the above steps, the required computing resources such as GPU of the large model can be quantified through continuous batch processing, model parallelism and the like; the requests in the cluster are grouped through clustering; the large model clusters of different configurations are automatically deployed according to the grouping; and the small model (i.e., the classification model) is introduced in load balancing, so that each request is distributed to the optimal cluster group, so that the batch processing file (Batch) that can be carried by the cluster is larger, and the core resource utilization is higher.

[0060] FIG. 4 is a structural diagram of an automatic distribution device of request information according to an embodiment of the present application. As shown in FIG. 4, the automatic distribution device of request information comprises: a receiving module 40 configured to receive a plurality of pieces of request information, wherein each piece of request information is used to request processing of a task corresponding to the request information; a first determining module 42 configured to, for each piece of request information, acquire related information of the request information, and determine, through a classification model, a token quantity of unit text of output data corresponding to the request information according to the related information of the request information; a second determining module 44 configured to automatically determine, through the classification model, a request mode corresponding to each piece of request information according to a token quantity of the request information and a token quantity of the output data; and a distribution module 46 configured to distribute each piece of request information to a corresponding cluster group according to the request mode corresponding to each piece of request information, wherein the cluster group is obtained by deploying a plurality of large language models of different configuration modes according to an automatic clustering result of request modes of a plurality of pieces of historical request information.

[0061] It should be noted that a specific implementation of the embodiment shown in FIG. 4 can refer to the related description of the embodiment shown in FIG. 2, which will not be described here again.

[0062] The embodiment of the present application further provides a non-volatile computer readable storage medium, which stores a computer program. The computer program is run by a processor to execute the above automatic distribution method of request information.

[0063] The non-volatile storage medium is used to store a program for performing the following functions: receiving a plurality of pieces of request information, wherein each piece of request information is used to request processing of a task corresponding to the request information; for each piece of request information, acquiring related information of the request information, and determining, through a classification model, a token quantity of unit text of output data corresponding to the request information according to the related information of the request information; automatically determining, through the classification model, a request mode corresponding to each piece of request information according to a token quantity of the request information and a token quantity of the output data; and distributing each piece of request information to a corresponding cluster group according to the request mode corresponding to each piece of request information, wherein the cluster group is obtained by deploying a plurality of large language models of different configuration modes according to an automatic clustering result of request modes of a plurality of pieces of historical request information.

[0064] The embodiment of the application further provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to perform the automatic distribution method of the request information.

[0065] Specifically, the processor in the electronic device executes the computer program to perform the following steps: receiving a plurality of request information, wherein each piece of request information is used to request a task corresponding to the processing request information; for each piece of request information, obtaining the related information of the request information, determining the number of unit texts tokens of the output data corresponding to the request information according to the related information of the request information by using a classification model; automatically determining the request mode corresponding to each piece of request information according to the token number of the request information and the token number of the output data by using the classification model; and distributing each piece of request information to a corresponding cluster group according to the request mode corresponding to each piece of request information, wherein the cluster group is obtained by deploying a plurality of large language models with different configuration modes according to the automatic clustering result of the request modes of a plurality of historical request information.

[0066] It should be noted that each module in the automatic distribution device of the request information can be a program module (for example, a set of program instructions for implementing a certain specific function) or a hardware module. For the latter, it can be in the following form, but is not limited to this: the form of each module is a processor, or the functions of each module are implemented by a processor.

[0067] The serial numbers of the above embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0068] In the above embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0069] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the device embodiment described above is only schematic. For example, the division of the units can be different, and each unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other form.

[0070] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0071] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software functional unit.

[0072] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application or the part that essentially contributes to the related art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various storage program codes.

[0073] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0074] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0075] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An automatic distribution method of request information, comprising: receiving a plurality of pieces of request information, wherein each piece of the request information is used to request processing of a task corresponding to the request information; for each piece of the request information, obtaining related information of the request information, determining a token quantity of output data corresponding to the request information according to the related information of the request information by using a classification model; automatically determining a request mode corresponding to each piece of the request information according to the token quantity of the request information and the token quantity of the output data by using the classification model; distributing each piece of the request information to a corresponding cluster group according to the request mode corresponding to each piece of the request information, wherein the cluster group is obtained by deploying a plurality of large language models with different configuration modes according to an automatic clustering result of request modes of a plurality of pieces of historical request information.

2. The method of claim 1, wherein the determining of the token quantity of the output data corresponding to the request information by using the classification model according to the related information of the request information comprises: determining a conversation quantity corresponding to the request information, and determining the token quantity of the request information; determining a target object sending the request information, and obtaining historical request information sent by the target object; inputting the conversation quantity corresponding to the request information, the token quantity of the request information, and the historical request information sent by the target object as the related information of the request information into the classification model to obtain the token quantity of the output data output by the classification model, wherein the classification model is trained by using historical information of different target objects as training data, and the historical information of each target object at least includes historical request information sent by the target object, a conversation quantity corresponding to each piece of the historical request information, output data corresponding to each piece of the historical request information, and a target identifier used to indicate that the classification model outputs a plurality of pieces of the output data for one piece of the request information.

3. The method of claim 1, wherein the request pattern comprises: a first request mode in which the token quantity of the request information belongs to a first numerical interval and the token quantity of the output data belongs to the first numerical interval, a second request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a second numerical interval, a third request mode in which the token quantity of the request information belongs to the first numerical interval and the token quantity of the output data belongs to a third numerical interval, a fourth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the first numerical interval, a fifth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the second numerical interval, a sixth request mode in which the token quantity of the request information belongs to the second numerical interval and the token quantity of the output data belongs to the third numerical interval, a seventh request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the first numerical interval, an eighth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the second numerical interval, and a ninth request mode in which the token quantity of the request information belongs to the third numerical interval and the token quantity of the output data belongs to the third numerical interval; wherein the third numerical interval indicates a token quantity greater than that indicated by the second numerical interval, and the second numerical interval indicates a token quantity greater than that indicated by the first numerical interval.

4. The method of claim 1, wherein the cluster groups are deployed by: for each of the historical request information, determining a request mode corresponding to the historical request information according to a token quantity of the historical request information and a token quantity of output data corresponding to the historical request information; determining a clustering result according to the request modes of the plurality of historical request information, the clustering result comprising: a number of cluster groups and proportions of different cluster groups; and deploying the large language models of different configuration modes as the plurality of cluster groups according to the clustering result.

5. The method of claim 4, wherein determining a clustering result according to the request modes of the plurality of historical request information comprises: determining a number of types of the request modes corresponding to the plurality of historical request information as the number of cluster groups; determining a number of historical request information corresponding to each of the request modes; determining proportions of different cluster groups as ratios of the numbers of historical request information.

6. The method of claim 1, wherein distributing each of the request information to a corresponding cluster group according to the request mode corresponding to the request information comprises: For each of the request information, a sum of a token number of the request information and a token number of the output data is determined as a first target token number corresponding to the request information; A corresponding second target token number of each of the cluster groups is obtained, wherein each of the second target token numbers is a maximum token number processed by each of the cluster groups; A cluster group to which each of the request information is to be distributed is determined according to the first target token number and the second target token number.

7. The method of claim 6, wherein the cluster group to which each of the request information is to be distributed is determined according to the first target token number and the second target token number, comprising: determining a difference value between each of the first target token numbers and each of the second target token numbers to obtain a plurality of difference values; determining a minimum positive difference value in the plurality of difference values corresponding to each of the first target token numbers as a target difference value corresponding to each of the first target token numbers; determining a cluster group indicated by the second target token number corresponding to the target difference value as the cluster group to which the request information corresponding to the first target token number is to be distributed.

8. An automatic distribution device of request information, comprising: a receiving module configured to receive a plurality of request information, wherein each of the request information is used to request processing of a task corresponding to the request information; a first determining module configured to, for each of the request information, obtain related information of the request information, and determine a unit text token number of output data corresponding to the request information according to the related information of the request information by using a classification model; a second determining module configured to automatically determine a request mode corresponding to each of the request information according to a token number of the request information and a token number of the output data by using the classification model; a distribution module configured to distribute each of the request information to a corresponding cluster group according to the request mode corresponding to each of the request information, wherein the cluster group is obtained by deploying a plurality of large language models of different configuration modes according to an automatic clustering result of request modes of a plurality of historical request information.

9. A non-volatile computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to perform the automatic distribution method of request information of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, wherein the memory has a computer program stored therein, and the processor executes the automatic distribution method of request information of any one of claims 1 to 7 when the computer program is executed.

Images