A reasoning request processing method, device and equipment and readable storage medium

Abstract

The application discloses a kind of inference request processing method, device, equipment and readable storage medium in computer technology, the method comprises: after receiving target inference request sent by client, obtain inference request before target inference request in processing sequence;Extract the keyword of inference request, and find the inference data matched with keyword in database;Approximate inference length determined based on inference data;Using approximate inference length, determine the inference time of inference request;Based on inference time, feedback the waiting time of target inference request to client.The inference time in the application fully considers the difference of different inference requests, and combines historical inference request record and the information such as average inference length under different classification.Therefore, the inference time is more realistic, and the waiting time fed back to the client is more accurate, which can effectively improve user experience.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to a reasoning request processing method, apparatus, device, and readable storage medium. Background Technology

[0002] When performing inference in a language model, if there are many inference requests and insufficient computing power, the model cannot respond to inference requests immediately and must wait for previous inference requests to complete before processing newly received inference requests. Therefore, users may need to wait after sending an inference request.

[0003] Providing users with feedback on inference wait times can effectively improve user experience. However, current methods simply convert the average inference time of completed requests into the wait time for incomplete requests. Since each request has a different question and answer, the inference length will also vary. This simple estimation results in highly inaccurate time estimates, negatively impacting user experience.

[0004] In summary, how to effectively solve the problem of waiting time for accurate prediction inference requests is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a reasoning request processing method, apparatus, device, and readable storage medium that can accurately predict the waiting time of reasoning requests and effectively improve the user experience.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] A reasoning request processing method, comprising:

[0008] After receiving the target inference request sent by the client, obtain the inference request whose processing order precedes the target inference request;

[0009] Extract the keywords of the reasoning request and search the database for historical reasoning request records that match the keywords;

[0010] If the historical reasoning request record is found, the approximate reasoning length of the reasoning request is determined using the reasoning length in the historical reasoning request record;

[0011] If the historical reasoning request record is not found, the category to which the reasoning request belongs in the database is determined, and the approximate reasoning length is determined using the average reasoning length corresponding to the category.

[0012] The inference time of the inference request is determined using the approximate inference length.

[0013] Based on the inference time, the waiting time for the target inference request is fed back to the client.

[0014] Preferably, determining the approximate inference length of the inference request using the inference length in the historical inference request record includes:

[0015] Obtain the number of occurrences of the historical reasoning request record, and the reasoning length of the corresponding historical reasoning request for each occurrence;

[0016] Using the occurrence count and the inference length, determine the average inference length of the keyword-related inference request;

[0017] The average inference length is determined as the approximate inference length.

[0018] Preferably, determining the category to which the inference request belongs in the database, and determining the approximate inference length using the average inference length corresponding to the category, includes:

[0019] Determine the industry, discipline, and type to which the inference request belongs in the database;

[0020] Obtain the average inference length of the industry, the average inference length of the discipline, and the average inference length of the type from the database;

[0021] The approximate inference length is obtained by weighted summing of the average inference length of the industry, the average inference length of the discipline, and the average inference length of the type.

[0022] Preferably, determining the inference time of the inference request using the approximate inference length includes:

[0023] Obtain the current inference computing power;

[0024] The ratio of the approximate inference length to the inference computing power is determined as the inference time.

[0025] Preferably, acquiring the reasoning request whose processing order precedes the target reasoning request includes:

[0026] Determine whether a reasoning response can be immediately provided to the target reasoning request;

[0027] If so, the language class model is used to infer the target inference request, and the inference result is fed back to the client in real time;

[0028] If not, the target inference request is inserted into the waiting queue;

[0029] Read the reasoning request that precedes the target reasoning request from the waiting queue.

[0030] Preferably, before searching the database for historical reasoning request records matching the keyword, the method further includes:

[0031] Create the database;

[0032] Create statistical items for the inference service in the database and set the average inference length corresponding to each statistical item; wherein, the statistical items include the category classification of inference requests under different dimensions, and the dimensions include industry, discipline and type;

[0033] Construct multiple keywords and determine the initial inference length and inference time for each keyword inference request;

[0034] Establish a mapping relationship between keywords and the statistical items they belong to under different dimensions;

[0035] Set the inference weight percentage for different dimensions;

[0036] Accordingly, after feeding back the inference result of the target inference request to the client, the method further includes:

[0037] The database records the target inference request record corresponding to the target inference request; wherein the target inference request record includes the keyword of the target inference request, the actual inference length, and the actual inference time;

[0038] Based on the mapping relationship between keywords and their corresponding statistical items under different dimensions, the statistical item to which the target inference request record belongs is determined in the database;

[0039] Based on the actual inference length, calculate the average inference length corresponding to each statistical item to which the target inference request record belongs;

[0040] Based on the average inference length corresponding to the statistical item to which the target inference request record belongs, update the average inference length of the statistical item to which the target inference request record belongs in the database.

[0041] Preferably, the inference request includes multiple requests, and based on the inference time, the waiting time for the target inference request is fed back to the client, including:

[0042] The inference times corresponding to multiple inference requests are summed to obtain the cumulative inference time;

[0043] The cumulative inference time is determined as the waiting time;

[0044] The waiting time is then fed back to the client.

[0045] A reasoning request processing apparatus, comprising:

[0046] The request determination module is used to receive the target inference request sent by the client and then obtain the inference request whose processing order is prior to the target inference request.

[0047] The matching and search module is used to extract the keywords of the reasoning request and search the database for historical reasoning request records that match the keywords;

[0048] The inference length estimation module is used to determine the approximate inference length of the inference request by using the inference length in the historical inference request record if the historical inference request record is found; if the historical inference request record is not found, the module determines the category to which the inference request belongs in the database and determines the approximate inference length by using the average inference length corresponding to the category.

[0049] An inference time estimation module is used to determine the inference time of the inference request using the approximate inference length;

[0050] The waiting feedback module is used to provide feedback to the client on the waiting time for the target inference request based on the inference time.

[0051] An electronic device, comprising:

[0052] Memory, used to store computer programs;

[0053] A processor is configured to implement the steps of the above-described inference request processing method when executing the computer program.

[0054] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described inference request processing method.

[0055] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the above-described inference request processing method.

[0056] Applying the method provided in this application embodiment, after receiving a target inference request sent by a client, the method obtains inference requests that precede the target inference request in the processing order; extracts keywords from the inference request and searches for historical inference request records that match the keywords in the database; if a historical inference request record is found, the approximate inference length of the inference request is determined using the inference length in the historical inference request record; if no historical inference request record is found, the method determines the category to which the inference request belongs in the database and determines the approximate inference length using the average inference length corresponding to the category; the method determines the inference time of the inference request using the approximate inference length; and based on the inference time, the method feeds back the waiting time of the target inference request to the client.

[0057] In this application, upon receiving a target inference request from a client, inference requests that precede the target inference request in the processing order can be obtained. Then, keywords of the inference request are extracted. By matching these keywords in a database, it is determined whether there are any historical inference request records similar to the target inference request. If data is found, based on the principle that similar inference requests have similar inference lengths, the approximate inference length of the inference request can be determined based on the inference length in the historical inference request records. If no data is found, the approximate inference length of the inference request is determined based on the category to which the inference request belongs in the database and the average inference length corresponding to that category. After determining the approximate inference length, the inference time of the inference request can be predicted based on the inference length. Based on this inference time, a more accurate waiting time for the target inference request can be fed back to the client.

[0058] Technical Effects: The inference time in this application fully considers the differences between various inference requests and incorporates information such as historical inference request records and average inference lengths across different categories. Therefore, this inference time more closely reflects actual inference times, resulting in more accurate waiting times reported to the client and effectively improving user experience.

[0059] Accordingly, embodiments of this application also provide inference request processing apparatus, devices, readable storage media, and computer program products corresponding to the above-described inference request processing method, which have the above-described technical effects, and will not be repeated here. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating the implementation of a reasoning request processing method in an embodiment of this application.

[0062] Figure 2 This is a flowchart illustrating a specific implementation of a reasoning request processing method in this application.

[0063] Figure 3 This is a schematic diagram of the structure of a reasoning request processing device according to an embodiment of this application;

[0064] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0065] Figure 5This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] To facilitate understanding of the technical solutions provided in the embodiments of this application, the relevant technical terms are explained below.

[0068] AI: Artificial Intelligence;

[0069] LLM: LLM is an abbreviation for Large Language Model. A large language model is a model trained on massive amounts of text that can recognize human language, perform language-related tasks, and has a large number of parameters.

[0070] Model weight file: The weight file of the model after training. The inference service program uses this file to complete the algorithm inference business.

[0071] Token: In language model inference, a token represents the inference word and is usually composed of a number. Generally, the model generates one token per inference, and one token represents one throughput. These tokens can be matched with corresponding words in the vocabulary.

[0072] Vectors: In the field of artificial intelligence, vectors are a way to describe the features of objects; large models will convert user-input text and images into structured vectors so that machines can understand and process this data; these vectors represent the features of text and images;

[0073] Language model inference: Language model inference generally adopts an autoregressive approach, recursively generating inference results word by word. Each time, a word is generated based on all the words and characters in the preceding text, and the next word is generated based on the current word and all the words in the preceding text.

[0074] Reasoning service: A reasoning service is an application that performs reasoning based on a user's request and then returns the result to the user.

[0075] Inference length: In large language models, because autoregressive reasoning is used, it is impossible to predict the length of the next word or subsequent answers until the model stops inference. Therefore, the final answer length is often not fixed.

[0076] Inference throughput: Large language models use inference throughput to represent inference performance, which is related to the computing power of the AI ​​computing card. Under full load, the inference throughput of the AI ​​computing card is a fixed value.

[0077] Inference time: In large language models, since the inference length for each request is not fixed, the corresponding inference time is also not fixed; the formula for calculating inference time is: Inference time = Total inference length / Inference throughput;

[0078] Batch inference: When inferring large models, multiple requests can be combined into a batch, and the entire batch is combined into a dataset, which is then fed into the computing card for computation. This allows multiple requests to be inferred simultaneously in one inference, effectively improving throughput.

[0079] Sample word extraction: Sample based on the reasoning results and convert the vectors reasoned by the machine into tokens;

[0080] Model inference prefill phase: input understanding and memory initialization;

[0081] Model inference / decoding phase: recursive inference and decoding output;

[0082] eos_token_id: The model inference terminator stops when this value is reached; for example, in llama3, eos_token_id = 2, inference stops when token = 2.

[0083] Database: A collection of data that allows users to manage, update, and access data efficiently.

[0084] Please refer to Figure 1 , Figure 1 This is a flowchart of an inference request processing method according to an embodiment of this application. The method can be applied to the inference end of a language model (such as a server or inference platform), and includes the following steps:

[0085] S101. After receiving the target inference request sent by the client, obtain the inference request whose processing order is before the target inference request.

[0086] After enabling the language inference service, it can receive target inference requests sent by clients. These requests can carry a language type question, such as, "Will it rain today?"

[0087] In practical applications, when business is not busy, upon receiving a target inference request, inference can be performed directly, meaning the corresponding inference result (the answer to the inference request) can be output to the client immediately. However, when business is busy, upon receiving a target inference request, the corresponding inference result cannot be output to the client immediately, requiring the user to wait. To improve user experience, the waiting time needs to be reminded to the user. In this case, the waiting time can be calculated by retrieving inference requests that precede the target inference request in the processing order.

[0088] In one specific embodiment of this application, obtaining a reasoning request whose processing order precedes that of the target reasoning request includes:

[0089] Determine whether a reasoning response can be immediately provided to the target reasoning request;

[0090] If so, the language class model is used to infer the target inference request, and the inference results are fed back to the client in real time;

[0091] If not, insert the target inference request into the waiting queue;

[0092] Read the inference request that precedes the target inference request from the waiting queue.

[0093] For ease of description, the above steps will be combined below.

[0094] In this embodiment, after receiving the target inference request, it can be determined whether the target inference request can be immediately inferred by judging whether there are other pending requests.

[0095] If direct reasoning is possible, a language model can be used to reason about the target inference request, and the client can receive the reasoning results in real time. In other words, in this case, the client can see the requested answer without waiting.

[0096] If direct inference is not possible at the moment, the target inference request can be inserted into the waiting queue, and then the inference requests preceding the target inference request can be read from the waiting queue.

[0097] It should be noted that, depending on the specific circumstances, there may be one or more inference requests.

[0098] S102. Extract the keywords of the reasoning request and search the database for historical reasoning request records that match the keywords.

[0099] When it is not possible to immediately process the target inference request, the keywords of the inference request can be extracted, and then the historical inference request records that match the keywords can be searched in the database.

[0100] Specifically, keyword extraction can involve slicing the inference request and searching for matching keywords in the database based on the slice information. For example, multiple keywords can be pre-set in the database. After slicing the inference request, each slice is matched with the keywords in the database, and the matched keywords are used as the keywords for the inference request.

[0101] Historical inference request records are the relevant records corresponding to inference requests that have completed response processing. They can specifically record information such as the keywords of the inference request, the length of the inference, and the inference time.

[0102] S103. If a historical reasoning request record is found, the approximate reasoning length of the reasoning request is determined by using the reasoning length in the historical reasoning request record.

[0103] If a historical inference request record is found and contains relevant keywords, it indicates that the inference requests are quite similar. Therefore, the approximate inference length of the inference request can be determined directly using the inference length in the historical inference request record.

[0104] In practical applications, the approximate inference length of an inference request can be determined based on the specific details of the historical inference request records found.

[0105] For example, if only one historical reasoning request record is found, the reasoning length in that historical reasoning request record can be directly determined as the approximate reasoning length of the reasoning request.

[0106] When multiple historical inference request records are found, the inference lengths in these records can be statistically analyzed to determine the approximate inference length of the inference request. For example, the median or mode of multiple inference lengths in the historical inference request records can be taken as the approximate inference length.

[0107] In one specific embodiment of this application, determining the approximate inference length of an inference request using the inference length in the historical inference request record includes:

[0108] Get the number of times the historical reasoning request record appears, and the reasoning length of the corresponding historical reasoning request for each appearance;

[0109] The average inference length of keyword-related inference requests is determined by using the frequency of occurrence and inference length.

[0110] The average inference length is determined as the approximate inference length.

[0111] For ease of description, the above steps will be combined below.

[0112] Based on the correlation between inference time and inference length, in this embodiment, an approximate inference length for the target inference request can be determined using a time estimation algorithm related to inference time, when there is direct inference data (i.e., historical inference request records) in the database. The formula is expressed as follows:

[0113] Where M is the number of times the requested keyword appears in the database, and t i Let T be the inference length of the i-th record in the database. N This represents the average inference length for this request.

[0114] S104. If no historical reasoning request record is found, determine the category to which the reasoning request belongs in the database, and use the average reasoning length corresponding to the category to determine the approximate reasoning length.

[0115] When no direct inference data is found, the category to which the inference request belongs in the database can be determined, and then the approximate inference length can be determined based on the average inference length corresponding to the category.

[0116] If the database is classified in only one dimension, the average inference length of the category can be directly determined as the approximate inference length.

[0117] When the database is classified in multiple dimensions, the average inference length of the categories to which different dimensions belong can be integrated to determine the approximate inference length.

[0118] In one specific embodiment of this application, determining the category to which the inference request belongs in the database and determining the approximate inference length using the average inference length corresponding to the category includes:

[0119] Determine the industry, discipline, and type of the inference request within the database;

[0120] Obtain the average inference length for the industry, the average inference length for the discipline, and the average inference length for the type from the database;

[0121] We obtain an approximate inference length by weighting and summing the average inference lengths of the industry, discipline, and type.

[0122] For ease of description, the above steps will be combined below.

[0123] In this embodiment, when there is no reasoning data with the same keyword as the reasoning request in the database, the industry, discipline and type of the reasoning request in the database can be determined first. Then, the average reasoning length of the industry, the average reasoning length of the discipline and the average reasoning length of the type are read from the database. Then, these average reasoning lengths are weighted and superimposed using a pre-set weighting coefficient, and the result of the weighted superposition is determined as the approximate reasoning length of the reasoning request.

[0124] For example, the formula for calculating the approximate inference length based on industry, discipline, and type is as follows:

[0125] T N = (K*0.3+Q*0.3+V*0.4),

[0126] Where K is the average inference length of the industry to which the request belongs, Q is the average inference length of the discipline to which the request belongs, V is the average inference length of the request type, and finally T is calculated. N The average inference length for this request is represented by the coefficients in the formula, which are weighting coefficients. These values ​​can be set and adjusted according to different actual situations.

[0127] S105. Using the approximate inference length, determine the inference time of the inference request.

[0128] In language models, because autoregressive reasoning is used, the length of the next word or subsequent answers cannot be predicted until the model reaches a stop point. Therefore, the final answer length is often not fixed. Since the inference length for each request is not fixed, the corresponding inference time is also not fixed; the formula for calculating inference time is: Inference Time = Total Inference Length / Inference Throughput.

[0129] In one specific embodiment of this application, determining the inference time of an inference request using an approximate inference length includes:

[0130] Obtain the current inference computing power;

[0131] The ratio of approximate inference length to inference computing power is defined as inference time.

[0132] Specifically, inference throughput refers to the server's inference computing power. Therefore, the calculation of inference time can be specifically as follows:

[0133] Where S represents the server's inference computing power. T t This represents the final estimated inference time. This statistical method allows for a more accurate estimation of inference time.

[0134] S106. Based on the inference time, provide feedback to the client on the waiting time for the target inference request.

[0135] Since the execution order of the target inference request follows that of the inference request, the waiting time for the target inference request includes the inference time of that inference request.

[0136] Therefore, the client can be informed of the waiting time for the target inference request based on the inference time.

[0137] In one specific embodiment of this application, the inference request includes multiple requests, and based on the inference time, the waiting time for the target inference request is fed back to the client, including:

[0138] The cumulative inference time is obtained by summing up the inference times corresponding to multiple inference requests;

[0139] The cumulative reasoning time is defined as the waiting time;

[0140] The waiting time is fed back to the client.

[0141] For ease of description, the above steps will be combined below.

[0142] If there are multiple inference requests preceding the target inference request in the execution order, the inference time for each request can be determined separately. Then, the inference times of these requests are summed to obtain the cumulative inference time. This cumulative inference time is then used as the waiting time and fed back to the client.

[0143] In one specific embodiment of this application, the inference time corresponding to each inference request inserted into the waiting queue can be calculated and saved;

[0144] When a new inference request is inserted, the inference time corresponding to the pending inference requests before the new inference request can be directly read and summed.

[0145] The summation result is used as the waiting time for newly inserted inference requests and is fed back to the client.

[0146] The method for calculating the inference time corresponding to multiple inference requests can refer to the operation steps S102 to S105 above, and will not be repeated here.

[0147] Applying the method provided in this application embodiment, after receiving a target inference request sent by a client, the method obtains inference requests that precede the target inference request in the processing order; extracts keywords from the inference request and searches for historical inference request records that match the keywords in the database; if a historical inference request record is found, the approximate inference length of the inference request is determined using the inference length in the historical inference request record; if no historical inference request record is found, the method determines the category to which the inference request belongs in the database and determines the approximate inference length using the average inference length corresponding to the category; the method determines the inference time of the inference request using the approximate inference length; and based on the inference time, the method feeds back the waiting time of the target inference request to the client.

[0148] In this application, upon receiving a target inference request from a client, inference requests that precede the target inference request in the processing order can be obtained. Then, keywords of the inference request are extracted. By matching these keywords in a database, it is determined whether there are any historical inference request records similar to the target inference request. If data is found, based on the principle that similar inference requests have similar inference lengths, the approximate inference length of the inference request can be determined based on the inference length in the historical inference request records. If no data is found, the approximate inference length of the inference request is determined based on the category to which the inference request belongs in the database and the average inference length corresponding to that category. After determining the approximate inference length, the inference time of the inference request can be predicted based on the inference length. Based on this inference time, a more accurate waiting time for the target inference request can be fed back to the client.

[0149] Technical Effects: The inference time in this application fully considers the differences between various inference requests and incorporates information such as historical inference request records and average inference lengths across different categories. Therefore, this inference time more closely reflects actual inference times, resulting in more accurate waiting times reported to the client and effectively improving user experience.

[0150] It should be noted that, based on the above embodiments, the embodiments of this application also provide corresponding improvement schemes. In the preferred / improved embodiments, the same or corresponding steps as in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.

[0151] In one specific embodiment of this application, before searching the database for historical reasoning request records matching the keywords, the method further includes:

[0152] Create the database;

[0153] Create statistical items for the inference service in the database and set the average inference length corresponding to each statistical item; wherein, the statistical items include the category classification of inference requests under different dimensions, and the dimensions include industry, discipline and type;

[0154] Construct multiple keywords and determine the initial inference length and inference time for each keyword inference request;

[0155] Establish a mapping relationship between keywords and the statistical items they belong to under different dimensions;

[0156] Set the inference weight percentage for different dimensions;

[0157] Accordingly, after feeding back the inference result of the target inference request to the client, the method further includes:

[0158] The database records the target inference request record corresponding to the target inference request; wherein the target inference request record includes the keyword of the target inference request, the actual inference length, and the actual inference time;

[0159] Based on the mapping relationship between keywords and their corresponding statistical items under different dimensions, the statistical item to which the target inference request record belongs is determined in the database;

[0160] Based on the actual inference length, calculate the average inference length corresponding to each statistical item to which the target inference request record belongs;

[0161] Based on the average inference length corresponding to the statistical item to which the target inference request record belongs, update the average inference length of the statistical item to which the target inference request record belongs in the database.

[0162] Statistical items refer to the categories assigned to reasoning requests based on different dimensions. For example, based on the industry dimension, the corresponding statistical items include, but are not limited to, agriculture, forestry, animal husbandry, fishery, mining, manufacturing, electricity, gas and water production and supply, construction, transportation, warehousing and postal services, information transmission, computer services and software, wholesale and retail, accommodation and catering, finance, and real estate. Based on the discipline dimension, the corresponding statistical items include, but are not limited to, philosophy, economics, law, education, literature, history, science, engineering, agriculture, medicine, military science, management, art, and interdisciplinary fields. Based on the type dimension, the corresponding statistical items include, but are not limited to, factual questions, explanatory questions, analytical questions, evaluative questions, comparative questions, problem-solving questions, and open-ended questions.

[0163] In other words, in the initial stage, an original database is established, covering the initial inference length and inference time of statistical items corresponding to various industries, disciplines, and types. After the service is deployed, user requests and inference data are continuously collected and recorded. As the amount of data increases and the database becomes more and more complete, the subsequent estimated inference time will become more and more accurate.

[0164] Furthermore, based on the classification of language models, the database can function as an independent entity, continuously iterating and sharing data with other newly deployed inference services. As usage time increases, the larger the database becomes, and the more accurate the predicted time becomes.

[0165] To facilitate those skilled in the art to better understand and implement the inference request processing method provided in the embodiments of this application, the specific implementation of the inference request processing method will be described in detail below with reference to specific scenarios.

[0166] After deploying a large language model on a cluster, the core inference scheduling process is as follows:

[0167] 1. Obtain user input and, based on the computing load, allocate the request to a less loaded inference group;

[0168] 2. Upon receiving the request, the inference group uses a tokenizer to convert the user's input text into machine-readable vectors based on the current memory usage of the computing card. It then combines the requested vector data with the vector data requested by the ongoing inference task into a batch dataset, which is then fed into the computing card for overall computation.

[0169] 3. Based on the calculation results in step 2, perform sample word extraction and convert the calculated vector into a token;

[0170] 4. Determine the token from step 3. If the token value of any request is equal to the eos_token_id of the model, then determine that the inference for that request has ended, remove the data of that request from the batch, and notify the computing card to release the computing resources for that request, freeing up space for new requests.

[0171] 5. Based on the token from step 3, if the request has not ended, continue executing the batch inference calculation from step 2, and repeat the process until the eos_token_id is output, at which point the inference request ends and the inference resources are released.

[0172] When the number of users and requests exceeds the computing card's capacity, users need to be prompted with the waiting time, and the inference time for each request needs to be estimated. As mentioned above, this application can estimate the inference completion time by establishing a large database and query algorithms, providing users with an accurate waiting time and a better user experience.

[0173] The implementation of the technical solution provided in this application includes a database, a time estimation algorithm, and an inference program.

[0174] The database can cover the initial inference length and inference time of various industries, disciplines and types; after the service is deployed, it continuously collects and records user requests and inference data, and the database becomes more and more complete as the data increases.

[0175] The estimated time algorithm operates independently of the inference program. It slices the user's request, extracts keywords, searches the database for matching keywords, and obtains inference length and inference time data. If no keywords are matched, it searches the original data in the database based on the industry, discipline, and type of the request. Through features such as industry, discipline, and type, it estimates the approximate inference length coefficient, calculates the approximate inference length using a formula, and then calculates it against the board's computing power to obtain the approximate inference length and inference time data.

[0176] As the core program for inference, the inference program starts from receiving a user request. When there are idle inference resources, it occupies the inference resources and executes inference until the inference is completed. When the inference is completed, the inference program needs to record the requested character keywords, inference length, inference time and other data, and write the data to the database. If there are no idle inference resources, it adds the request to the waiting queue and calls the estimated time algorithm to obtain the waiting time from the database and return it to the user.

[0177] Please refer to Figure 2 The overall solution process is as follows:

[0178] 1. The inference program starts, loads the model file, initializes the model, and begins accepting user requests;

[0179] 2. After receiving a user request, the inference program determines whether there are inference resources. If there are inference resources, it executes the inference and records data such as request information, inference length, and inference time after the inference is completed, and writes it to the database.

[0180] 3. After receiving a user request, if the inference program determines that there are no inference resources, it calls the estimation algorithm to calculate the queuing time, estimates the required queuing time, and returns the queuing time to the user. The execution flow of the estimation algorithm is as follows: It searches the database for keyword information of the queuing request. If similar request data is matched, the time data information is returned directly. If no similar request is matched, based on the industry, discipline, type, and other information of the queuing request, it queries the inference length information of the request from the basic database using SQL statements, and then calculates the inference length according to the calculation formula (see step 2 in section 2.2.1 for the specific formula). After performing algorithmic calculations with computing resources, the estimated inference time is obtained.

[0181] 4. The database can be used as an independent unit and shared among inference services. After each inference operation, the inference program writes data such as request information, inference length, and inference time into the database. The longer the deployment and usage time, the more information the database stores, and the more accurate the estimated time becomes.

[0182] In other words, this application adds a database function. After each inference, the inference program writes the request information, inference length, inference time, and other information into the database for subsequent time estimation. Based on the characteristics of the current request, the inference length and inference time are estimated. Compared with statistically analyzing the time of historical requests and using the average inference time of historical requests as the waiting time of the current request, this application determines the inference time based on the estimated inference length, thus making the method of determining the waiting time more accurate.

[0183] Corresponding to the above method embodiments, this application also provides an inference request processing apparatus, which can be referred to in correspondence with the inference request processing method described above.

[0184] See Figure 3 As shown, the device includes the following modules:

[0185] The request determination module 101 is used to receive the target inference request sent by the client and then obtain the inference request whose processing order is prior to the target inference request.

[0186] The matching and searching module 102 is used to extract keywords from the reasoning request and search for historical reasoning request records that match the keywords in the database.

[0187] The inference length estimation module 103 is used to determine the approximate inference length of the inference request by using the inference length in the historical inference request record if a historical inference request record is found; if no historical inference request record is found, the category to which the inference request belongs in the database is determined, and the approximate inference length is determined by using the average inference length corresponding to the category.

[0188] The inference time estimation module 104 is used to determine the inference time of the inference request using an approximate inference length;

[0189] The waiting feedback module 105 is used to provide feedback to the client on the waiting time for the target inference request based on the inference time.

[0190] Using the apparatus provided in this application embodiment, after receiving a target inference request sent by a client, the system obtains inference requests that precede the target inference request in the processing order; extracts keywords from the inference request and searches for historical inference request records that match the keywords in the database; if a historical inference request record is found, the approximate inference length of the inference request is determined using the inference length in the historical inference request record; if no historical inference request record is found, the system determines the category to which the inference request belongs in the database and determines the approximate inference length using the average inference length corresponding to the category; the system determines the inference time of the inference request using the approximate inference length; and based on the inference time, it feeds back the waiting time of the target inference request to the client.

[0191] In this application, upon receiving a target inference request from a client, inference requests that precede the target inference request in the processing order can be obtained. Then, keywords of the inference request are extracted. By matching these keywords in a database, it is determined whether there are any historical inference request records similar to the target inference request. If data is found, based on the principle that similar inference requests have similar inference lengths, the approximate inference length of the inference request can be determined based on the inference length in the historical inference request records. If no data is found, the approximate inference length of the inference request is determined based on the category to which the inference request belongs in the database and the average inference length corresponding to that category. After determining the approximate inference length, the inference time of the inference request can be predicted based on the inference length. Based on this inference time, a more accurate waiting time for the target inference request can be fed back to the client.

[0192] Technical Effects: The inference time in this application fully considers the differences between various inference requests and incorporates information such as historical inference request records and average inference lengths across different categories. Therefore, this inference time more closely reflects actual inference times, resulting in more accurate waiting times reported to the client and effectively improving user experience.

[0193] In one specific embodiment of this application, the inference length estimation module is specifically used to obtain the number of times a historical inference request record appears, and the inference length of the historical inference request corresponding to each appearance.

[0194] The average inference length of keyword-related inference requests is determined by using the frequency of occurrence and inference length.

[0195] The average inference length is determined as the approximate inference length.

[0196] In one specific embodiment of this application, the inference length estimation module is specifically used to determine the industry, discipline, and type of the inference request in the database;

[0197] Obtain the average inference length for the industry, the average inference length for the discipline, and the average inference length for the type from the database;

[0198] We obtain an approximate inference length by weighting and summing the average inference lengths of the industry, discipline, and type.

[0199] In one specific embodiment of this application, the inference time estimation module is specifically used to obtain the current inference computing power;

[0200] The ratio of approximate inference length to inference computing power is defined as inference time.

[0201] In one specific embodiment of this application, the request determination module is specifically used to determine whether a reasoning feedback can be immediately provided to the target reasoning request.

[0202] If so, the language class model is used to infer the target inference request, and the inference results are fed back to the client in real time;

[0203] If not, insert the target inference request into the waiting queue;

[0204] Read the inference request that precedes the target inference request from the waiting queue.

[0205] In one specific embodiment of this application, it further includes:

[0206] The database creation module is used to search the database for historical reasoning request records that match the keywords.

[0207] Create the database;

[0208] Create statistical items for the inference service in the database and set the average inference length corresponding to each statistical item; wherein, the statistical items include the category classification of inference requests under different dimensions, and the dimensions include industry, discipline and type;

[0209] Construct multiple keywords and determine the initial inference length and inference time for each keyword inference request;

[0210] Establish a mapping relationship between keywords and the statistical items they belong to under different dimensions;

[0211] Set the inference weight percentage for different dimensions;

[0212] Correspondingly, it also includes:

[0213] The database update module is used to record the target inference request record corresponding to the target inference request in the database after feeding back the inference result of the target inference request to the client; wherein, the target inference request record includes the keyword of the target inference request, the actual inference length, and the actual inference time;

[0214] Based on the mapping relationship between keywords and their corresponding statistical items under different dimensions, the statistical item to which the target inference request record belongs is determined in the database;

[0215] Based on the actual inference length, calculate the average inference length corresponding to each statistical item to which the target inference request record belongs;

[0216] Based on the average inference length corresponding to the statistical item to which the target inference request record belongs, update the average inference length of the statistical item to which the target inference request record belongs in the database.

[0217] In one specific embodiment of this application, the inference request includes multiple requests, and the waiting feedback module is specifically used to sum up the inference times corresponding to the multiple inference requests to obtain the cumulative inference time.

[0218] The cumulative reasoning time is defined as the waiting time;

[0219] The waiting time is fed back to the client.

[0220] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the reasoning request processing method described above can be referred to in correspondence.

[0221] See Figure 4 As shown, the electronic device includes:

[0222] Memory 332 is used to store computer programs;

[0223] The processor 322 is used to implement the steps of the reasoning request processing method in the above method embodiments when executing a computer program.

[0224] For details, please refer to Figure 5 , Figure 5This is a schematic diagram of a specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0225] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0226] The steps in the reasoning request processing method described above can be implemented by the structure of an electronic device.

[0227] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the inference request processing method described above.

[0228] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the reasoning request processing method described in the above method embodiments.

[0229] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.

[0230] Corresponding to the above method embodiments, this application also provides a computer program product. The computer program product described below can be referred to in conjunction with the reasoning request processing method described above.

[0231] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the above-described inference request processing method.

[0232] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0233] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0234] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0235] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0236] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for processing inference requests, characterized in that, The inference end applied to language models includes: After receiving the target inference request sent by the client, if the number of users and the number of requests exceed the computing card's load, the inference request whose processing order precedes the target inference request is obtained; the target inference request is a language type problem. Extract the keywords of the reasoning request and search the database for historical reasoning request records that match the keywords; wherein, extracting the keywords of the reasoning request includes: after slicing the reasoning request, matching each slice with the keywords in the database, and using the matched keywords as the keywords of the reasoning request; If the historical reasoning request record is found, the approximate reasoning length of the reasoning request is determined using the reasoning length in the historical reasoning request record; If the historical reasoning request record is not found, the category to which the reasoning request belongs in the database is determined, and the approximate reasoning length is determined using the average reasoning length corresponding to the category. The inference time of the inference request is determined using the approximate inference length. Based on the inference time, the waiting time for the target inference request is fed back to the client; The determination of the reasoning time for the reasoning request using the approximate reasoning length includes: Obtain the current inference computing power; The ratio of the approximate inference length to the inference computing power is defined as the inference time; wherein, the formula for calculating the inference time is: Where S is the server's inference computing power. This is the final estimated inference time; The process of determining the approximate inference length of a reasoning request using the inference lengths in the historical inference request records includes: Obtain the number of occurrences of the historical reasoning request record, and the reasoning length of the corresponding historical reasoning request for each occurrence; Using the occurrence count and the inference length, determine the average inference length of the keyword-related inference request; The average inference length is determined as the approximate inference length; wherein, the formula for calculating the approximate inference length is: Where M represents the number of times the requested keyword appears in the database. Let be the inference length of the i-th record in the database. This is the average inference length of the request; Before searching the database for historical reasoning request records that match the keyword, the process also includes: Create the database; Create statistical items for the inference service in the database and set the average inference length for each statistical item; wherein, the statistical items include the category classification of inference requests under different dimensions, and the dimensions include industry, discipline and type; Construct multiple keywords and determine the initial inference length and inference time for each keyword inference request; Establish a mapping relationship between keywords and the statistical items they belong to under different dimensions; Set the inference weight percentage for different dimensions; Accordingly, after feeding back the inference result of the target inference request to the client, the method further includes: The database records the target inference request record corresponding to the target inference request; wherein the target inference request record includes the keyword of the target inference request, the actual inference length, and the actual inference time; Based on the mapping relationship between keywords and the statistical items they belong to under different dimensions, the statistical item to which the target reasoning request record belongs is determined in the database; Based on the actual inference length, calculate the average inference length corresponding to each statistical item to which the target inference request record belongs; Based on the average inference length corresponding to the statistical item to which the target inference request record belongs, update the average inference length of the statistical item to which the target inference request record belongs in the database.

2. The method according to claim 1, characterized in that, Determining the category to which the inference request belongs in the database, and determining the approximate inference length using the average inference length corresponding to the category, includes: Determine the industry, discipline, and type to which the inference request belongs in the database; Obtain the average inference length of the industry, the average inference length of the discipline, and the average inference length of the type from the database; The approximate inference length is obtained by weighted summing of the average inference length of the industry, the average inference length of the discipline, and the average inference length of the type.

3. The method according to claim 1, characterized in that, Obtaining inference requests whose processing order precedes the target inference request includes: Determine whether a reasoning response can be immediately provided to the target reasoning request; If so, the language class model is used to infer the target inference request, and the inference result is fed back to the client in real time; If not, the target inference request is inserted into the waiting queue; Read the reasoning request that precedes the target reasoning request from the waiting queue.

4. The method according to any one of claims 1 to 3, characterized in that, The inference requests include multiple requests. Based on the inference time, the waiting time for the target inference request is fed back to the client, including: The inference times corresponding to multiple inference requests are summed to obtain the cumulative inference time; The cumulative inference time is determined as the waiting time; The waiting time is then fed back to the client.

5. A reasoning request processing apparatus, characterized in that, The inference end applied to language models includes: The request determination module is used to receive the target inference request sent by the client, and, if the number of users and the number of requests exceed the load of the computing card, to obtain the inference requests whose processing order precedes the target inference request; the target inference request is a language type problem. The matching and searching module is used to extract the keywords of the reasoning request and search the database for historical reasoning request records that match the keywords; wherein, extracting the keywords of the reasoning request includes: after slicing the reasoning request, matching each slice with the keywords in the database, and using the matched keywords as the keywords of the reasoning request; The inference length estimation module is used to determine the approximate inference length of the inference request by using the inference length in the historical inference request record if the historical inference request record is found; if the historical inference request record is not found, the module determines the category to which the inference request belongs in the database and determines the approximate inference length by using the average inference length corresponding to the category. The inference time estimation module is used to determine the inference time of the inference request using the approximate inference length, including: obtaining the current inference computing power; determining the ratio of the approximate inference length to the inference computing power as the inference time; wherein, the formula for calculating the inference time is: Where S is the server's inference computing power. The final estimated inference time; wherein, using the inference length in the historical inference request record, the approximate inference length of the inference request is determined, including: obtaining the occurrence count of the historical inference request record and the inference length of the historical inference request corresponding to each occurrence; using the occurrence count and the inference length, determining the average inference length of the keyword-related inference requests; and determining the average inference length as the approximate inference length; wherein, the formula for calculating the approximate inference length is: Where M represents the number of times the requested keyword appears in the database. Let be the inference length of the i-th record in the database. This is the average inference length of the request; The waiting feedback module is used to provide feedback to the client on the waiting time for the target inference request based on the inference time. The database creation module is used to create the database before searching for historical inference request records that match the keywords; to create statistical items for inference services in the database and set the average inference length corresponding to each statistical item; wherein, the statistical items include the category classification of inference requests under different dimensions, and the dimensions include industry, discipline and type; to construct multiple keywords and determine the initial inference length and inference time of the inference requests corresponding to the keywords; to establish a mapping relationship between the keywords and the statistical items to which they belong under different dimensions; and to set the inference weight ratio of different dimensions. Accordingly, the database update module is used to record the target inference request record corresponding to the target inference request in the database after feeding back the inference result of the target inference request to the client; wherein, the target inference request record includes the keyword of the target inference request, the actual inference length, and the actual inference time; based on the mapping relationship between the keyword and the statistical item to which it belongs under different dimensions, the statistical item to which the target inference request record belongs is determined in the database; based on the actual inference length, the average inference length corresponding to the statistical item to which the target inference request record belongs is calculated; and based on the average inference length corresponding to the statistical item to which the target inference request record belongs, the average inference length of the statistical item to which the target inference request record belongs is updated in the database.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the inference request processing method as described in any one of claims 1 to 4 when executing the computer program.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the inference request processing method as described in any one of claims 1 to 4.

Images