Method and apparatus for automatically distributing request information, and nonvolatile storage medium
By predicting the number of input and output tokens using a classification model, automatically determining the request pattern and distributing request information, the problem of resource waste in improving the inference efficiency of large models is solved, and more efficient resource utilization is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2024-04-19
- Publication Date
- 2026-05-19
AI Technical Summary
When improving the inference efficiency of large models, existing technologies do not take into account the difference in length between input and output tokens, resulting in wasted resources.
The system predicts the number of input and output tokens using a classification model, automatically determines the request pattern, distributes the request information to the corresponding cluster groups, and deploys a large language model based on the clustering results of historical request information.
Resource allocation was optimized, the inference efficiency of large models was improved, resource waste was reduced, and higher utilization of core resources was achieved.
Smart Images
Figure CN118394469B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a method and apparatus for automatically distributing request information, and a non-volatile storage medium. Background Technology
[0002] With the development of deep learning and artificial intelligence technologies, the application of large-scale model technology is becoming increasingly widespread, and the parameter scale of large models is gradually increasing. At the same time, the computational resources consumed by large models during runtime, including memory, video memory, central processing unit (CPU), and graphics processing unit (GPU), are also unprecedentedly large. Improving the inference efficiency of large models while maintaining essentially the same performance is crucial for their widespread adoption. Related technologies that improve the inference efficiency of large models through model parallelism only focus on how to perform model parallelism, without considering the difference between the length of the input unit text (token) and the length of the output token. However, the throughput of large models is constrained by the input-output (IO) operations of the video memory; its inference throughput depends on the batch processing capacity that the GPU memory can accommodate. Ignoring the lengths of the input and output units will result in a serious waste of resources.
[0003] There is currently no effective solution to the above problems. Summary of the Invention
[0004] This application provides an automatic distribution method and apparatus for request information, as well as a non-volatile storage medium, to at least solve the technical problem of wasted resources caused by not considering the length of the input token and the length of the output token of a large model when improving the inference efficiency of a large model.
[0005] According to one aspect of the embodiments of this application, an automatic distribution method for request information is provided, comprising: receiving multiple request messages, wherein each request message is used to request processing a task corresponding to the request message; for each request message, obtaining relevant information of the request message, and using a classification model to determine the number of unit text tokens of the output data corresponding to the request message based on the relevant information of the request message; automatically determining the request pattern corresponding to the request message based on the number of tokens of the request message and the number of tokens of the output data using the classification model; and distributing each request message to a corresponding cluster group, wherein the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages.
[0006] Optionally, a classification model is used to determine the number of unit text tokens in the output data corresponding to the request information based on the relevant information of the request information. This includes: determining the number of dialogues corresponding to the request information and determining the number of tokens in the request information; determining the target object that sent the request information and obtaining the historical request information sent by the target object; inputting the number of dialogues corresponding to the request information, the number of tokens in the request information, and the historical request information sent by the target object as relevant information of the request information into the classification model to obtain the number of tokens in the output data of the classification model. The classification model is trained using the historical information of different target objects as training data. The historical information of each target object includes at least: the historical request information sent by the target object, the number of dialogues corresponding to each historical request information, the output data corresponding to each historical request information, and a target identifier used to instruct the classification model to output multiple output data for a request information.
[0007] Optionally, the request mode corresponding to the request information is determined based on the number of tokens in the request information and the number of tokens in the output data. The request modes include: a first request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a first value range; a second request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a second value range; a third request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a first value range; a fourth request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a second value range; and a fifth request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a third value range. The fifth request pattern belongs to the second numerical range; the sixth request pattern belongs to the second numerical range and the number of tokens in the output data belongs to the third numerical range; the seventh request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the first numerical range; the eighth request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the second numerical range; and the ninth request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the third numerical range. In this pattern, the number of tokens indicated by the third numerical range is greater than the number of tokens indicated by the second numerical range, and the number of tokens indicated by the second numerical range is greater than the number of tokens indicated by the first numerical range.
[0008] Optionally, cluster groups are deployed as follows: for each historical request information, the request pattern corresponding to the historical request information is determined based on the number of tokens in the historical request information and the number of tokens in the output data corresponding to the historical request information; the clustering result is determined based on the request patterns of multiple historical request information, wherein the clustering result includes: the number of cluster groups and the proportion of different cluster groups; and large language models with multiple different configuration modes are deployed into multiple cluster groups based on the clustering result.
[0009] Optionally, the clustering result is determined based on the request patterns of multiple historical request information, including: determining the number of request patterns corresponding to multiple historical request information as the number of cluster groups; determining the number of historical request information contained in each request pattern; and determining the ratio of the number of multiple historical request information as the proportion of different cluster groups.
[0010] Optionally, the request information is distributed to the corresponding cluster group based on the number of tokens in the request information and the number of tokens in the output data, including: for each request information, determining the sum of the number of tokens in the request information and the number of tokens in the output data as the first target token number corresponding to the request information; obtaining the corresponding second target token number for each cluster group, wherein each second target token number is the maximum number of tokens that each cluster group can process; and determining the cluster group to which each request information is to be distributed based on the first target token number and the second target token number.
[0011] Optionally, determining the cluster group to which each request message is to be distributed based on the first target token number and the second target token number includes: determining the difference between each first target token number and each second target token number to obtain multiple differences; determining the smallest positive number among the multiple differences corresponding to each first target token number as the target difference corresponding to each first target token number; and determining the cluster group indicated by the second target token number corresponding to the target difference as the cluster group to which the request message corresponding to the first target token number is to be distributed.
[0012] According to another aspect of the embodiments of this application, an automatic distribution device for request information is also provided, comprising: a receiving module, configured to receive multiple request messages, wherein each request message is used to request processing a task corresponding to the request message; a first determining module, configured to, for each request message, obtain relevant information of the request message and use a classification model to determine the number of unit text tokens of the output data corresponding to the request message based on the relevant information of the request message; a second determining module, configured to automatically determine the request pattern corresponding to the request message based on the number of tokens in the request message and the number of tokens in the output data using a classification model; and a distribution module, configured to distribute each request message to a corresponding cluster group, wherein the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages.
[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, in which a computer program is stored, wherein the above-described automatic distribution method of request information is executed by running the computer program on the device where the non-volatile storage medium is located.
[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described automatic distribution method of request information through the computer program.
[0015] In this embodiment, multiple request messages are received, each request message is used to request the task corresponding to the request message. For each request message, relevant information of the request message is obtained, and a classification model is used to determine the number of unit text tokens of the output data corresponding to the request message based on the relevant information of the request message. The request mode corresponding to the request message is automatically determined by the classification model based on the number of tokens of the request message and the number of tokens of the output data. Each request message is distributed to the corresponding cluster group. The request messages are clustered by the length of the input request message token and the predicted length of the output data token. The clustering result determines which group the request message is distributed to. This achieves the purpose of adding the length of the input token and the length of the output token to the consideration when distributing request messages. This achieves the technical effect of optimal allocation for the entire cluster while meeting business needs and maintaining the same effect. In turn, it solves the technical problem of wasted resources caused by not considering the length of the input token and the length of the output token of the large model when improving the inference efficiency of the large model. Attached Figure Description
[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an automatic distribution method of request information according to an embodiment of this application;
[0018] Figure 2 This is a flowchart of the steps of an automatic distribution method for request information according to an embodiment of this application;
[0019] Figure 3 This is a schematic diagram of a continuous batch processing according to an embodiment of this application;
[0020] Figure 4 This is a structural diagram of an automatic request information distribution device according to an embodiment of this application. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:
[0024] A token is a piece of text or symbols that has been divided into individual units, usually separated by spaces, punctuation marks, or other delimiters. In natural language processing and computer programming, text is frequently segmented into tokens for processing and analysis.
[0025] Large models: Models with a large number of parameters and high computational resource requirements. These models typically have higher complexity and can handle larger-scale data and more complex tasks. For example, large neural network models in deep learning.
[0026] Small models: Models with fewer parameters and lower computational resource requirements. These models typically have lower complexity and are suitable for handling small datasets and simple tasks. Examples include models based on traditional machine learning methods such as logistic regression and decision trees.
[0027] Large-scale language models: Language models with a larger model size and number of parameters compared to conventional language models.
[0028] In related technologies, when improving the inference efficiency of models, most focus only on how to perform model parallelism. However, these parallelism solutions often fail to consider the significant difference between the length of the input token and the length of the output token requested by the user. Therefore, there is a problem of wasted resources due to this neglect of the difference between the input and output token lengths. To address this issue, this application provides a related solution, which is detailed below.
[0029] According to an embodiment of this application, an embodiment of an automatic distribution method for request information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0030] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing an automatic distribution method for request information is shown. Figure 1As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0031] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a form of processor control (e.g., selection of a variable resistor termination path connected to an interface).
[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the automatic distribution method of request information in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned automatic distribution method of request information. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0034] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.
[0035] This application provides an embodiment of an automatic distribution method for request information that can run in the above-described operating environment. Figure 2 This is a flowchart of the steps of the automatic distribution method for request information provided in the embodiments of this application, such as... Figure 2 As shown, the method includes the following steps:
[0036] Step S202: Receive multiple request messages, where each request message is used to request the task corresponding to the request message.
[0037] In step S202, the request information input by the user is received. The machine learning model that processes the request information can obtain the specific content of the request information and determine the specific business corresponding to the request information. For example, if the request information is a statement: "What is 'a'?", the specific content received by the machine learning model that processes the request information is the statement, and it can determine that the business corresponding to the request information is a query business.
[0038] Step S204: For each request message, obtain the relevant information of the request message, and use a classification model to determine the number of unit text tokens in the output data corresponding to the request message based on the relevant information of the request message.
[0039] In step S204, after receiving the request information, the machine learning model that processes the request information calls the classification model to perform clustering processing on the multiple received request information. Specifically, the relevant information of each request information is input into the pre-trained classification model, and the classification model can determine the number of tokens in the output data corresponding to the request information.
[0040] Optionally, the number of unit text tokens in the output data corresponding to the request information is determined by a classification model based on the relevant information of the request information. This includes: determining the number of dialogues corresponding to the request information and determining the number of tokens in the request information; determining the target object that sent the request information and obtaining the historical request information sent by the target object; inputting the number of dialogues corresponding to the request information, the number of tokens in the request information, and the historical request information sent by the target object as relevant information of the request information into the classification model to obtain the number of tokens in the output data of the classification model. The classification model is trained using the historical information of different target objects as training data. The historical information of each target object includes at least: the historical request information sent by the target object, the number of dialogues corresponding to each historical request information, the output data corresponding to each historical request information, and a target identifier used to instruct the classification model to output multiple output data for a request information.
[0041] The number of input tokens is easy to count, but the number of output tokens can only be known after the large model has processed the input information and output the exact output data. Therefore, in this embodiment, a pre-trained classification model is used to predict the number of tokens in the output data corresponding to the input data. In this embodiment, the classification model is trained using historical information of multiple target objects as training data. The target objects are the users / accounts that sent the request information. The historical information of the target objects includes: the request information sent by each account / user before the current time (historical request information), the number of conversations for each historical request information, the output data corresponding to each historical request information, and a command prompt (i.e., target identifier) used to guide the model to output multiple data. The target identifier Prompt is included in the input data. If a target identifier Prompt exists in an input request information, it indicates that multiple output data should be output for the input request information. After training using the above method, the classification model performs the following steps when determining the number of tokens in the output data corresponding to the request information: First, it obtains relevant information about the request information. Since the classification model is trained using historical information from multiple target objects as training data, it first determines the number of tokens in the request information, the number of dialogues corresponding to the request information, and the target object (user / account) that sent the request information. It then obtains the historical request information sent by this target object (user / account) so that the number of tokens in the request information, the number of dialogues, and the historical request information sent by the target object (user / account) can be used as input data for the classification model to obtain the number of tokens in the output data corresponding to the request information. The number of dialogues corresponding to the request information indicates which stage the target object (user / account) sent the request information when interacting with the machine learning model (sending the request information).
[0042] Step S206: The request mode corresponding to the request information is automatically determined by the classification model based on the number of tokens in the request information and the number of tokens in the output data.
[0043] In step S206, for each request message, after determining the number of tokens in the output data corresponding to the request message, the classification model further determines the request pattern corresponding to the request message based on the number of tokens in the request message and the number of tokens in the corresponding output data.
[0044] Optionally, the request mode corresponding to the request information is determined based on the number of tokens in the request information and the number of tokens in the output data. The request modes include: a first request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a first value range; a second request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a second value range; a third request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a first value range; a fourth request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a second value range; and a fifth request mode where the number of tokens in the request information and the number of tokens in the output data both belong to a third value range. The fifth request pattern belongs to the second numerical range; the sixth request pattern belongs to the second numerical range and the number of tokens in the output data belongs to the third numerical range; the seventh request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the first numerical range; the eighth request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the second numerical range; and the ninth request pattern belongs to the third numerical range and the number of tokens in the output data belongs to the third numerical range. In this pattern, the number of tokens indicated by the third numerical range is greater than the number of tokens indicated by the second numerical range, and the number of tokens indicated by the second numerical range is greater than the number of tokens indicated by the first numerical range.
[0045] The request pattern of each request message is determined by the number of tokens in that request message and the number of tokens in the output data corresponding to that request message. In this embodiment, the request patterns are divided into: short input short output request pattern (i.e., the first request pattern), short input medium output request pattern (i.e., the second request pattern), short input long output request pattern (i.e., the third request pattern), medium input short output request pattern (i.e., the fourth request pattern), medium input medium output request pattern (i.e., the fifth request pattern), medium input long output request pattern (i.e., the sixth request pattern), long input short output request pattern (i.e., the seventh request pattern), long input medium output request pattern (i.e., the eighth request pattern), and long input long output request pattern (i.e., the ninth request pattern). The type of input—whether it is short, medium, or long—is determined by the number of tokens in the request information. For example, a first numerical range corresponding to short input, a second numerical range corresponding to medium input, and a third numerical range corresponding to long input can be predefined. The type of input is determined by which numerical range the number of tokens in the request information belongs to. When setting different numerical ranges, it should be ensured that the number of tokens indicated by the (first) numerical range corresponding to short input is less than the number of tokens indicated by the (second) numerical range corresponding to medium input, and the number of tokens indicated by the (second) numerical range corresponding to medium input is less than the number of tokens indicated by the (third) numerical range corresponding to long-medium input.
[0046] Step S208: Distribute each request message to the corresponding cluster group. The cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of historical request information.
[0047] In step S208, after determining the request pattern of each request message using a classification model, each request message is distributed to its corresponding cluster group. When the machine learning model for processing request messages distributes the request messages, the cluster group to which the request message is distributed is fixed. This cluster group is generated by deploying large language models with different configuration modes based on the clustering results of the request patterns in the training data when training the machine learning model for processing request messages using massive amounts of training data. The training data consists of massive amounts of historical request messages sent by multiple different users; for example, 1TB (bytes) of historical request messages may be used as training data. Configuration modes include learning rate, number of layers, number of hidden units, etc. When the same large language model is configured with different numbers of hidden units, the two large language models generated after configuration are two large language models with different configuration modes. The training data for the machine learning model for processing request messages consists of massive amounts of historical request messages sent by multiple different users; for example, 1TB (bytes) of historical request messages may be used as training data.
[0048] According to an optional embodiment of this application, the cluster group is deployed in the following manner: for each historical request information, the request pattern corresponding to the historical request information is determined based on the number of tokens in the historical request information and the number of tokens in the output data corresponding to the historical request information; the clustering result is determined based on the request patterns of multiple historical request information, wherein the clustering result includes: the number of cluster groups and the proportion of different cluster groups; and large language models with multiple different configuration modes are deployed into multiple cluster groups based on the clustering result.
[0049] In this embodiment, massive amounts of historical request information sent by multiple different users, for example, 1TB (bytes) of historical request information, are used as training data. During the training of a machine learning model for processing request information using this training data, cluster groups are deployed and generated. Specifically, when deploying and generating cluster groups, the request pattern corresponding to each historical request is first determined. When clustering request patterns, the total number of request patterns corresponding to multiple historical request information is recorded, as well as the number of request information items contained in each request pattern. The number of request pattern types is used as the number of cluster groups, and the ratio of the number of request information items contained in different request patterns is used as the proportion of each cluster group. The number of cluster groups and the proportion of each cluster group together constitute the clustering result. Large language models with different configuration patterns are deployed according to the above clustering result, where the number of cluster groups is equal to the number of configuration pattern types, and the proportion of each cluster group is the proportion of large language models with different configuration patterns. For example, the request patterns of 10,000 historical request messages are divided into four types: short input short output request pattern (i.e., the first request pattern), medium input medium output request pattern (i.e., the fifth request pattern), long input medium output request pattern (i.e., the eighth request pattern), and long input long output request pattern (i.e., the ninth request pattern). Therefore, the number of cluster groups is 4, and the large language model configuration has 4 types: a large language model supporting the first request pattern, a large language model supporting the fifth request pattern, a large language model supporting the eighth request pattern, and a large language model supporting the ninth request pattern. Of the 10,000 historical request messages, 2,000 have the short input short output request pattern (i.e., the first request pattern), and 1... If 000 historical request messages follow a medium-input, medium-output request pattern (i.e., the fifth request pattern), 4000 historical request messages follow a long-input, medium-output request pattern (i.e., the eighth request pattern), and 3000 historical request messages follow a long-input, long-output request pattern (i.e., the ninth request pattern), then the ratio of the cluster groups is 2000:1000:4000:3000 = 2:1:4:3. Therefore, in the deployed cluster groups, the ratio of large language models supporting the first request pattern to those supporting the fifth request pattern is 2:1:4:3. The deployment method (number and proportion) of cluster groups remains fixed after the machine learning model used to process request information is trained. Each cluster group is identified with a corresponding number. When the machine learning model used to process request information distributes request information, it selects the cluster group corresponding to each request information from the fixed cluster groups and then distributes it to the corresponding cluster group; or it outputs the number of the cluster group corresponding to each request information to the user, and the user distributes the request information to the corresponding cluster group according to the number.
[0050] Optionally, the clustering result is determined based on the request patterns of multiple request messages, including: determining the number of request patterns corresponding to multiple request messages as the number of cluster groups; determining the number of request messages contained in each request pattern; and determining the ratio of the number of multiple request messages as the proportion of different cluster groups.
[0051] When clustering based on the request patterns of the request information, the first step is to determine how many request patterns the received request information corresponds to. The number of request pattern types is used as the number of cluster groups when dividing the cluster. For example, if it is determined that among the ten received request information, 3 request information corresponds to the first request pattern, 2 request information corresponds to the second request pattern, and 5 request information corresponds to the third request pattern, then the number of request pattern types corresponding to the request information is 3, and the cluster is divided into 3 cluster groups. The ratio of the number of machine learning models contained in multiple cluster groups is determined based on the number of request messages contained in each request mode. For example, in the example above, there are 3 request messages corresponding to the first request mode, 2 request messages corresponding to the second request mode, and 5 request messages corresponding to the third request mode. The ratio of the number of machine learning models contained in the (first) cluster group used to process request messages of the first request mode to the number of machine learning models contained in the (second) cluster group used to process request messages of the second request mode and the number of machine learning models contained in the (third) cluster group used to process request messages of the third request mode is 3:2:5. That is, when the cluster is divided into 3 cluster groups, the ratio of the machine learning models contained in these three cluster groups is 3:2:5.
[0052] It should be noted that after dividing the cluster into multiple cluster groups, the cluster resources need to be allocated to each cluster group. At this time, all resources provided by the cluster need to be quantified. Since the number of user requests (QPS) fluctuates greatly over time in most online service environments, methods such as memory paging pre-allocation and continuous batch processing are used to set the average and maximum amount of resources used after the model is deployed, thus quantifying the resources. Figure 3 This is a diagram illustrating continuous batch processing, such as... Figure 3As shown, the maximum number of tokens that can be processed in continuous batch processing is 8 (T1-T8). Request information is processed in parallel, meaning that four sequences can simultaneously process request information S1 with 3 input tokens, request information S2 with 2 input tokens, request information S3 with 3 input tokens, and request information S4 with 4 input tokens. Since continuous batch processing requires the batch file to be as full as possible, after outputting a result S1' with 2 tokens for S1 and adding an end marker to indicate that processing of request information S1 is complete, if there are remaining resources in this sequence, they are used to process other request information, such as... Figure 3 In this sequence, after request information S1 is processed, the remaining resources are used to process request information S6; after request information S3 is processed, the processing result S3' is output and the end symbol is filled in, and the remaining resources are used to process request information S5 and output the processing result S5'; after request information S4 is processed, the processing result S4' is output and the end symbol is filled in, and the remaining resources are used to process request information S7. Request information S2, its corresponding processing result S2', and the end symbol have already filled 8 tokens (T1-T8), meaning the resources for this sequence have been exhausted. Therefore, this sequence is not used to process any other information besides S2.
[0053] According to an optional embodiment of this application, distributing request information to corresponding cluster groups based on the number of tokens in the request information and the number of tokens in the output data includes: for each request information, determining the sum of the number of tokens in the request information and the number of tokens in the output data as the first target token number corresponding to the request information; obtaining the corresponding second target token number for each cluster group, wherein each second target token number is the maximum number of tokens processed by each cluster group; and determining the cluster group to which each request information is to be distributed based on the first target token number and the second target token number.
[0054] In this embodiment, before distributing the request information to the corresponding cluster group, it is first determined which cluster group the request information should be distributed to. Specifically, the sum of the token count of the request information and the token count of the output data corresponding to the request information (i.e., the first target token count) is determined. Next, the maximum number of tokens that each cluster group can process (i.e., the second target token count) is determined. The cluster group to which the request information should be distributed is determined by comparing the sum of the token count of the request information and the token count of the output data corresponding to the request information (i.e., the first target token count) with the maximum number of tokens that each cluster group can process (i.e., the second target token count). The maximum number of tokens that each cluster group can process (i.e., the second target token count) is determined by the machine learning models contained in each cluster group. For a cluster group, the maximum number of tokens that each machine learning model in the cluster group can process is obtained, and the sum of the maximum number of tokens that multiple machine learning models can process is determined as the maximum number of tokens that the cluster group can process. The maximum number of tokens for each machine learning model can be adjusted.
[0055] According to another optional embodiment of this application, determining the cluster group to which each request message is to be distributed based on the first target token number and the second target token number includes: determining the difference between each first target token number and each second target token number to obtain multiple differences; determining the smallest positive number among the multiple differences corresponding to each first target token number as the target difference corresponding to each first target token number; and determining the cluster group indicated by the second target token number corresponding to the target difference as the cluster group to which the request message corresponding to the first target token number is to be distributed.
[0056] As mentioned in the above embodiments, there are cluster groups capable of handling different numbers of tokens within the cluster. Optimal cluster performance is achieved when each request is assigned to the cluster group that is largest and closest to the sum of the number of tokens in the request information and the number of tokens in the output data (i.e., the first target token number). Therefore, in this embodiment, after determining the sum of the number of tokens in the request information and the number of tokens in the corresponding output data (i.e., the first target token number), when distributing request information, the difference between the first target token number and the maximum number of tokens that each cluster group can handle (i.e., the second target token number) is determined. From the multiple differences obtained, the difference that is both greater than zero and the smallest (i.e., the smallest positive number) is selected. The first target token number and the second target token number that generate this difference are determined, and the request information corresponding to the first target token number that generates this difference is distributed to the cluster group corresponding to the second target token number that generates this difference.
[0057] Through the above steps, it is possible to quantify the computing resources such as GPU memory required by large models through continuous batch processing and model parallelism; group requests in the cluster through clustering; automatically deploy multiple large model clusters with different configurations according to the grouping; and introduce small models (i.e., classification models) into load balancing to distribute each request to the optimal cluster group, so that the cluster can handle a larger batch (batch file) and higher core resource utilization.
[0058] Figure 4 This is a structural diagram of an automatic request information distribution device according to an embodiment of this application, such as... Figure 4 As shown, the automatic distribution device for request information includes: a receiving module 40, used to receive multiple request messages, wherein each request message is used to request the task corresponding to the request message; a first determining module 42, for each request message, obtaining relevant information of the request message, and determining the number of unit text tokens of the output data corresponding to the request message based on the relevant information of the request message through a classification model; a second determining module 44, used to automatically determine the request pattern corresponding to the request message based on the number of tokens in the request message and the number of tokens in the output data through a classification model; and a distribution module 46, used to distribute each request message to the corresponding cluster group, wherein the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages.
[0059] It should be noted that, Figure 4 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.
[0060] This application also provides a non-volatile storage medium storing a computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned automatic distribution method of request information by running the computer program.
[0061] The aforementioned non-volatile storage medium is used to store programs that perform the following functions: receiving multiple request messages, where each request message is used to request the task corresponding to the request message; for each request message, obtaining relevant information about the request message, and using a classification model to determine the number of unit text tokens in the output data corresponding to the request message based on the relevant information about the request message; automatically determining the request pattern corresponding to the request message based on the number of tokens in the request message and the number of tokens in the output data using the classification model; and distributing each request message to the corresponding cluster group, where the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages.
[0062] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to execute the above-mentioned automatic distribution method of request information through the computer program.
[0063] The processor in the aforementioned electronic device is used to run a program that performs the following functions: receiving multiple request messages, where each request message requests a task corresponding to the request message; for each request message, obtaining relevant information about the request message, and using a classification model to determine the number of unit text tokens in the output data corresponding to the request message based on the relevant information; automatically determining the request pattern corresponding to the request message based on the number of tokens in the request message and the number of tokens in the output data using the classification model; and distributing each request message to a corresponding cluster group, where the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages.
[0064] It should be noted that each module in the above-mentioned automatic distribution device for request information can be a program module (e.g., a set of program instructions that implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.
[0065] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0066] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0067] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0069] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0071] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for automatically distributing request information, characterized in that, include: Receive multiple request messages, wherein each request message is used to request the processing of the task corresponding to the request message; For each of the request messages, relevant information of the request message is obtained, and a classification model is used to determine the number of unit text tokens in the output data corresponding to the request message based on the relevant information of the request message. The classification model automatically determines the request pattern corresponding to the request information based on the number of tokens in the request information and the number of tokens in the output data. Each request message is distributed to a corresponding cluster group, wherein the cluster group is obtained by deploying large language models with multiple different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages; wherein distributing each request message to a corresponding cluster group includes: determining the difference between each first target token number and each second target token number to obtain multiple differences, wherein the first target token number is the sum of the token number of the request message and the token number of the output data, and each second target token number is the maximum number of tokens that each cluster group can process; determining the smallest positive number among the multiple differences corresponding to each first target token number as the target difference corresponding to each first target token number; and determining the cluster group indicated by the second target token number corresponding to the target difference as the cluster group to which the request message corresponding to the first target token number is to be distributed; After distributing each request message to the corresponding cluster group, the method further includes: if the cluster group has finished processing the request message, determining whether the cluster group has any remaining resources, and if the cluster group has any remaining resources, using the remaining resources to process other request messages.
2. The method according to claim 1, characterized in that, A classification model is used to determine the number of unit text tokens in the output data corresponding to the request information based on relevant information, including: Determine the number of conversations corresponding to the request information, and determine the number of tokens in the request information; Identify the target object to which the request information is sent, and obtain the historical request information sent by the target object; The number of dialogues corresponding to the request information, the number of tokens in the request information, and the historical request information sent by the target object are input into the classification model as relevant information of the request information to obtain the number of tokens in the output data of the classification model. The classification model is trained using the historical information of different target objects as training data. The historical information of each target object includes at least: the historical request information sent by the target object, the number of dialogues corresponding to each historical request information, the output data corresponding to each historical request information, and a target identifier for instructing the classification model to output multiple output data for one request information.
3. The method according to claim 1, characterized in that, The request pattern corresponding to the request information is determined based on the number of tokens in the request information and the number of tokens in the output data. The request pattern includes: a first request pattern where the number of tokens in the request information and the number of tokens in the output data both belong to the first value range; a second request pattern where the number of tokens in the request information and the number of tokens in the output data both belong to the first value range; a third request pattern where the number of tokens in the request information and the number of tokens in the output data both belong to the first value range; a fourth request pattern where the number of tokens in the request information and the number of tokens in the output data both belong to the second value range; and a fifth request pattern where the number of tokens in the request information and the number of tokens in the output data both belong to the second value range. The fifth request pattern of the second numerical range, the sixth request pattern of the request information having a token count belonging to the second numerical range and the output data having a token count belonging to the third numerical range, the seventh request pattern of the request information having a token count belonging to the third numerical range and the output data having a token count belonging to the first numerical range, the eighth request pattern of the request information having a token count belonging to the third numerical range and the output data having a token count belonging to the second numerical range, and the ninth request pattern of the request information having a token count belonging to the third numerical range and the output data having a token count belonging to the third numerical range, wherein the token count indicated by the third numerical range is greater than the token count indicated by the second numerical range, and the token count indicated by the second numerical range is greater than the token count indicated by the first numerical range.
4. The method according to claim 1, characterized in that, The cluster group is deployed in the following manner: For each historical request information, the request mode corresponding to the historical request information is determined based on the number of tokens in the historical request information and the number of tokens in the output data corresponding to the historical request information. The clustering result is determined based on the request pattern of multiple historical request information, wherein the clustering result includes: the number of cluster groups and the proportion of different cluster groups; Based on the clustering results, the large language models with multiple different configuration modes are deployed into multiple cluster groups.
5. The method according to claim 4, characterized in that, The clustering results are determined based on the request patterns of multiple historical request information entries, including: The number of request patterns corresponding to multiple historical request messages is determined as the number of cluster groups; Determine the amount of historical request information contained in each of the aforementioned request patterns; The ratio of the number of the various historical request information items is determined as the proportion of the different cluster groups.
6. The method according to claim 1, characterized in that, The request information is distributed to the corresponding cluster group based on the number of tokens in the request information and the number of tokens in the output data, including: For each of the aforementioned request messages, the sum of the number of tokens in the request message and the number of tokens in the output data is determined as the first target number of tokens corresponding to the request message; Obtain the corresponding number of second target tokens for each of the cluster groups; The cluster group to which each request message is to be distributed is determined based on the first target token count and the second target token count.
7. An automatic distribution device for request information, characterized in that, include: A receiving module is used to receive multiple request messages, wherein each request message is used to request the processing of the task corresponding to the request message; The first determining module, for each of the request messages, obtains the relevant information of the request message and uses a classification model to determine the number of unit text tokens of the output data corresponding to the request message based on the relevant information of the request message; The second determining module is used to automatically determine the request pattern corresponding to the request information based on the number of tokens in the request information and the number of tokens in the output data using the classification model. A distribution module is used to distribute each request message to a corresponding cluster group, wherein the cluster group is obtained by deploying multiple large language models with different configuration modes based on the automatic clustering results of the request patterns of multiple historical request messages; wherein distributing each request message to a corresponding cluster group includes: determining the difference between each first target token number and each second target token number to obtain multiple differences, wherein the first target token number is the sum of the token number of the request message and the token number of the output data, and each second target token number is the maximum number of tokens that each cluster group can process; determining the smallest positive number among the multiple differences corresponding to each first target token number as the target difference corresponding to each first target token number; and determining the cluster group indicated by the second target token number corresponding to the target difference as the cluster group to which the request message corresponding to the first target token number is to be distributed; The automatic distribution device for request information is further configured to, after distributing each request information to the corresponding cluster group, determine whether the cluster group has any remaining resources after the cluster group has finished processing the request information, and if the cluster group has any remaining resources, use the remaining resources to process other request information.
8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the automatic distribution method of request information as described in any one of claims 1 to 6 by running the computer program.
9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the automatic distribution method for request information as described in any one of claims 1 to 6 through the computer program.