Request processing method and apparatus, electronic device and storage medium

Abstract

The present application relates to the technical field of computers, and in particular to a request processing method and apparatus, an electronic device, and a storage medium, for use in preventing hallucinations in large models due to high concurrency, and improving the availability of the large models. The method comprises: acquiring a request to be processed, initially acquiring from a token bucket a token corresponding to said request, and adding 1 to the total number of concurrencies of a current sliding time window; if the initial acquisition of the token fails, on the basis of the total number of concurrencies, and the throughput, average delay and maximum acceptable delay of a large model, determining the number of retries and a retry interval duration of re-acquiring the token; and on the basis of the number of retries and the retry interval duration, re-acquiring the token from the token bucket, and when the re-acquisition of the token succeeds, performing inference on said request on the basis of the large model, and generating and returning an answer. In this way, the number of retries and the retry interval duration are dynamically adjusted on the basis of the real-time total number of concurrencies, and requests are scattered to reach large models, thereby improving the response success rate and availability.

Description

A request processing method, apparatus, electronic device, and storage medium

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411686713.8, filed on November 22, 2024, with the title “A Request Processing Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a request processing method, apparatus, electronic device, and storage medium. Background Technology

[0004] With the development of artificial intelligence technology, the application of large models is becoming more and more common. Large models often need to process a large number of requests from different users or devices at the same time, which can lead to high concurrency.

[0005] Due to the complexity and slow speed of reasoning in large models, hallucinatory responses are prone to occur in high-concurrency scenarios, and they may even be unable to recover on their own for a long time.

[0006] Therefore, how to provide a request processing method to prevent large models from experiencing illusions due to high concurrency and improve the availability of large models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] This application provides a request processing method, apparatus, electronic device, and storage medium to prevent large models from experiencing illusions due to high concurrency and to improve the usability of large models.

[0008] The specific technical solutions provided in this application are as follows:

[0009] Firstly, a request processing method is provided, the method including:

[0010] Get the pending requests, and initially get the token corresponding to the pending request from the token bucket, and increment the total number of concurrent requests in the current sliding time window by 1;

[0011] If the initial token acquisition fails, the number of retries and the retry interval are determined based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model.

[0012] Based on the number of retries and the retry interval, a token is re-acquired from the token bucket. When the token is successfully re-acquired, the large model is used to infer the request to be processed, generate and return the answer.

[0013] In one possible implementation, the number of retries and the retry interval for re-acquiring the token are determined based on the total number of concurrent connections, as well as the throughput, average latency, and maximum acceptable latency of the large model, including:

[0014] The maximum acceptable retry duration is determined based on the maximum acceptable latency and the average latency.

[0015] Based on throughput and total concurrency, determine the success rate of acquiring a token in the current sliding time window;

[0016] The number of retries is determined based on the acceptable maximum retry duration and success rate;

[0017] The retry interval is determined based on the maximum acceptable retry duration and the number of retries.

[0018] By employing the above method, the time intervals are evenly distributed based on the maximum acceptable retry duration. This allows for a longer wait before attempting to acquire a token in high-concurrency scenarios (i.e., when the total number of concurrent requests is larger), reducing competition pressure. In low-concurrency scenarios (i.e., when the total number of concurrent requests is smaller), token acquisition occurs faster, reducing waiting time, resulting in faster responses, increased retries, and improved response success rates. Furthermore, since the total number of concurrent requests varies for each request within the current sliding event window, the calculated retry interval also differs. This dispersed retry intervals, based on the varying concurrency levels of different requests, also disperses the time it takes for a successfully acquired token to reach the large model, reducing the probability of the large model experiencing "illusion" events.

[0019] In one possible embodiment, obtaining the request to be processed includes:

[0020] The system receives a question request from a requesting object and retrieves the question request based on a question-and-answer database, which is used to store historical question-and-answer pairs.

[0021] If no answer is found for the question request, the question request will be treated as a pending request.

[0022] By using the above method to answer questions using a question-and-answer database, we can reduce the repeated generation of answers for similar questions, reduce the total number of concurrent requests, and improve the answering speed, thereby enhancing the user experience.

[0023] In one possible embodiment, after reasoning about the request to be processed based on a large model, generating and returning an answer, the process further includes:

[0024] Store the question-and-answer pairs for the requests to be processed in the question-and-answer database.

[0025] The above methods have enriched the question-and-answer database.

[0026] In one possible embodiment, the method further includes:

[0027] If the token is successfully obtained on the first attempt, the system will infer the request to be processed based on the large model, generate and return the answer, and store the question-answer pair of the request to be processed in the question-answer database.

[0028] In one possible embodiment, the method further includes:

[0029] If reacquiring the token fails, return a prompt message to the request object of the pending request and add the pending request to the unanswered message queue.

[0030] In one possible embodiment, the method further includes:

[0031] The GPU utilization rate of the image processor is monitored periodically. When the GPU utilization rate is lower than the preset utilization rate threshold, unanswered requests are retrieved from the unanswered message queue and processed.

[0032] When an answer is received for an unanswered request, the question-answer pair for the unanswered request is stored in the question-answer database.

[0033] The above methods enrich the question-answer pairs in the question-answer database, increase the hit probability of the question-answer database retrieval, and increase the probability that unanswered questions will be asked again by the questioner, making it easier for the questioner to quickly obtain the answers to unanswered questions, while making full use of GPU resources.

[0034] Secondly, a request processing apparatus is provided, the apparatus comprising:

[0035] The processing module is used to obtain requests to be processed, initially obtain the token corresponding to the request to be processed from the token bucket, and increment the total number of concurrent requests in the current sliding time window by 1.

[0036] The determination module is used to determine the number of retries and the retry interval if the initial token acquisition fails, based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model.

[0037] The retry module is used to re-acquire tokens from the token bucket based on the number of retries and the retry interval. When the token is successfully re-acquired, the module performs reasoning on the request to be processed based on the large model, generates and returns the answer.

[0038] In one possible embodiment, when determining the number of retries and the retry interval for reacquiring a token based on the total number of concurrent connections, as well as the throughput, average latency, and maximum acceptable latency of the large model, the determining module is further configured to:

[0039] The maximum acceptable retry duration is determined based on the maximum acceptable latency and the average latency.

[0040] Based on throughput and total concurrency, determine the success rate of acquiring a token in the current sliding time window;

[0041] The number of retries is determined based on the acceptable maximum retry duration and success rate;

[0042] The retry interval is determined based on the maximum acceptable retry duration and the number of retries.

[0043] In one possible embodiment, when obtaining a request to be processed, the processing module is further configured to:

[0044] The system receives a question request from a requesting object and retrieves the question request based on a question-and-answer database, which is used to store historical question-and-answer pairs.

[0045] If no answer is found for the question request, the question request will be treated as a pending request.

[0046] In one possible embodiment, after reasoning about the request to be processed based on a large model, generating and returning an answer, the device further includes a storage module, which is used for:

[0047] Store the question-and-answer pairs for the requests to be processed in the question-and-answer database.

[0048] In one possible embodiment, the determining module is further configured to:

[0049] If the token is successfully obtained on the first attempt, the system will infer the request to be processed based on the large model, generate and return the answer, and store the question-answer pair of the request to be processed in the question-answer database.

[0050] In one possible embodiment, the retry module is also used for:

[0051] If reacquiring the token fails, return a prompt message to the request object of the pending request and add the pending request to the unanswered message queue.

[0052] In one possible embodiment, the device further includes a monitoring module, which is used for:

[0053] The GPU utilization rate of the image processor is monitored periodically. When the GPU utilization rate is lower than the preset utilization rate threshold, unanswered requests are retrieved from the unanswered message queue and processed.

[0054] When an answer is received for an unanswered request, the question-answer pair for the unanswered request is stored in the question-answer database.

[0055] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described in the first aspect above.

[0056] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0057] In this embodiment, a request to be processed is acquired, and the token corresponding to the request is initially obtained from the token bucket. The total concurrency of the current sliding time window is incremented by 1. If the initial token acquisition fails, the number of retries and the retry interval are determined based on the total concurrency, the throughput of the large model, the average latency, and the maximum acceptable latency. Then, based on the number of retries and the retry interval, the token is re-acquired from the token bucket. When the re-acquisition of the token is successful, the large model is used to infer the request to be processed, generate, and return the answer. In this way, the token bucket addresses high-concurrency scenarios, improving the availability of the large model. Different requests correspond to different total concurrency levels, and the calculated retry intervals are distributed, ensuring that requests reach the large model in a dispersed manner, reducing the probability of illusions in the large model and preventing it from experiencing illusions due to high concurrency. Furthermore, a fixed token rate limiting algorithm can easily trigger traffic thresholds, leading to low response rates in the large model. Therefore, within the acceptable latency range, the number of retries and the retry interval are dynamically adjusted based on the real-time total concurrency, improving the response success rate. Attached Figure Description

[0058] Figure 1 is a system architecture diagram of the request processing system in an embodiment of this application;

[0059] Figure 2 is a flowchart illustrating the implementation of a request processing method provided in an embodiment of this application;

[0060] Figure 3 is a schematic diagram of another request processing method in an embodiment of this application;

[0061] Figure 4 is a schematic diagram of the structure of a request processing device provided in an embodiment of this application;

[0062] Figure 5 is a schematic diagram of the structure of the electronic device in the embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0064] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0065] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0066] The acquisition, transmission, storage, and use of data in this application all comply with the requirements of relevant national laws and regulations.

[0067] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0068] Frequently Asked Questions (FAQ): Questions are answered by searching a pre-organized database of possible frequently asked questions.

[0069] Token bucket: A common flow control algorithm where tokens are generated at a fixed rate, typically a certain number per second. These tokens are placed in a bucket with a fixed maximum capacity. Whenever a request arrives, the system attempts to retrieve a token from the bucket. If there is a token in the bucket, the request is allowed to proceed, and one token is removed from the bucket. If there is no token, the request is rejected or delayed.

[0070] Sliding window rate limiting algorithm: An algorithm used to control the data flow rate. It divides the time window into multiple small periods, each with its own counter. As time slides, expired data from the small periods is deleted, thereby controlling the flow more precisely.

[0071] The design concept of the embodiments of this application is briefly introduced below:

[0072] With the development of artificial intelligence technology, the application of large models is becoming more and more common. Large models often need to process a large number of requests from different users or devices at the same time, which can lead to high concurrency.

[0073] Due to the complexity and slow speed of reasoning in large models, hallucinatory responses are prone to occur in high-concurrency scenarios, and they may even be unable to recover on their own for a long time.

[0074] In existing technologies, high-concurrency scenarios are addressed by optimizing the performance of the large model itself.

[0075] However, the above methods often involve a trade-off between accuracy and performance, which may increase computing resource requirements, latency, complexity, and long-term maintenance costs.

[0076] Therefore, how to prevent large models from experiencing illusions due to high concurrency and improve their usability without optimizing their performance is a problem that urgently needs to be solved by those skilled in the art.

[0077] In view of this, this application provides a request processing method, apparatus, electronic device, and storage medium. The method involves acquiring a request to be processed, initially acquiring the token corresponding to the request from a token bucket, and incrementing the total concurrency of the current sliding time window by 1. If the initial token acquisition fails, the number of retries and the retry interval are determined based on the total concurrency, the throughput of the large model, the average latency, and the maximum acceptable latency. Then, based on the number of retries and the retry interval, the token is reacquired from the token bucket. When the token acquisition is successful, the large model is used to infer the request to be processed, generate, and return an answer. In this way, the token bucket addresses high-concurrency scenarios, improves the availability of the large model, distributes the total concurrency for different requests, and disperses the calculated retry intervals, ensuring that requests reach the large model in a distributed manner, reducing the probability of illusions in the large model and preventing illusions caused by high concurrency. In addition, fixed token rate limiting algorithms can easily trigger traffic thresholds, resulting in low response rates for large models. Therefore, within acceptable latency ranges, the number of retries and the retry interval are dynamically adjusted based on the real-time total concurrency, thereby improving the response success rate.

[0078] Referring to Figure 1, this is a system architecture diagram of the request processing system in an embodiment of this application. The request processing system includes a load testing module 110, a FAQ module 120, a rate limiting module 130, a large model inference generation module 140, and a monitoring module 150.

[0079] Load testing module 110: Performs load testing on the large model generation module to obtain the throughput and average latency of the large model, and defines the maximum acceptable latency.

[0080] FAQ Module 120: Maintain a high-quality question and answer database. Based on this database, retrieve user-sent questions to obtain answers directly without relying on large model inference.

[0081] Rate limiting module 130: Based on the token bucket algorithm, sliding window counting algorithm and dynamic retry mechanism, it limits the rate of query requests before entering the large model.

[0082] Large model inference generation module 140: generates answers to question requests and stores the question-answer pairs corresponding to the question requests into the question-answer database.

[0083] Monitoring module 150: Periodically monitors the GPU utilization rate of the image processor. When the GPU utilization rate is lower than the preset utilization rate threshold, it retrieves unanswered requests from the unanswered message queue for processing. When the answer corresponding to the unanswered request is obtained, it stores the question-answer pair of the unanswered request in the question-answer database.

[0084] Referring to Figure 2, which is a flowchart of a request processing method provided in an embodiment of this application, the specific implementation process of the method is as follows:

[0085] Step 20: Obtain the pending requests, and initially obtain the token corresponding to the pending request from the token bucket, and increment the total number of concurrent requests in the current sliding time window by 1.

[0086] Among them, the requests to be processed are those that require the generation of answers through large model reasoning. The current sliding time window is the sliding time window in the sliding window rate limiting algorithm. The statistical duration of each sliding time window is set to, for example, 1 second.

[0087] In this embodiment of the application, after obtaining the request to be processed, the token corresponding to the request to be processed is initially obtained from the token bucket, and the counter of the current sliding time window is incremented by 1. The counter count is the total number of concurrent requests in the current sliding time window.

[0088] Among them, the rate at which tokens are issued and the maximum capacity of the token bucket are determined based on the throughput of the large model.

[0089] Optionally, in this application embodiment, a possible implementation is provided for obtaining the request to be processed, and the specific steps are as follows:

[0090] Step 200: Receive the question request from the requesting object and retrieve the question request based on the question-and-answer database.

[0091] The question-and-answer database is used to store historical question-and-answer pairs. The question-and-answer database can be a FAQ database, but this application embodiment does not impose any restrictions on it. The question-and-answer pair is a question-and-answer pair.

[0092] In this embodiment of the application, based on the question-answering database, the target question contained in the question request is retrieved. If a similar question is retrieved that is similar to the target question, the answer to the similar question is used as the answer to the target question.

[0093] In addition, it is worth noting that in the embodiments of this application, historical question-and-answer pairs are periodically clustered and low-quality filtered to maintain a high-quality question-and-answer database.

[0094] Step 201: Determine whether no answer corresponding to the question request has been found. If yes, proceed to step 202; otherwise, proceed to step 203.

[0095] In this embodiment of the application, if a similar question to the target question contained in the question request is found, it is determined that the answer to the question request has been found, and the answer to the similar question is used as the answer to the target question. If no similar question to the target question contained in the question request is found, it is determined that the answer to the question request has not been found.

[0096] Step 202: Treat the question request as a pending request.

[0097] In this embodiment of the application, if no answer is found for the question request, the question request is treated as a pending request.

[0098] Step 203: Return the answer to the request object.

[0099] In this embodiment of the application, if an answer corresponding to the question request is found, the answer is returned to the requesting object.

[0100] By using a question-and-answer database to answer questions first, we can reduce the repeated generation of answers for similar questions, reduce the total number of concurrent requests, and improve the response speed, thereby enhancing the user experience.

[0101] Step 21: If the initial token acquisition fails, determine the number of retries and the retry interval based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model.

[0102] Among them, throughput refers to the number of requests processed by a large model per unit of time, which can also be understood as the amount of input data processed per second. Average latency refers to the average time required for a large model to process one request and generate a response. Maximum acceptable latency refers to the maximum model response latency that a user can tolerate in large model applications.

[0103] In addition, it is worth noting that in the embodiments of this application, the throughput and average latency of the large model are obtained by stress testing the large model, and the maximum acceptable latency is defined.

[0104] Optionally, in this application embodiment, to determine the number of retries and the retry interval, a possible implementation is provided, with the following specific steps:

[0105] Step 210: Determine the maximum acceptable retry duration based on the maximum acceptable latency and the average latency.

[0106] In this embodiment of the application, the difference between the maximum acceptable latency and the average latency is calculated to obtain the maximum acceptable retry duration.

[0107] The maximum acceptable retry duration is expressed as: T = MR. T is the maximum acceptable retry duration, M is the maximum acceptable delay, and R is the average delay.

[0108] For example, assuming the maximum acceptable latency is 24s and the average latency is 18s, then the maximum acceptable retry time is 24-18=6s.

[0109] Step 211: Based on throughput and total concurrency, determine the success rate of acquiring a token in the current sliding time window.

[0110] In this embodiment of the application, the ratio of throughput to total concurrency is calculated to obtain the success rate of obtaining a token in the current sliding time window.

[0111] The success rate of acquiring a token within the current sliding time window is expressed as: P = Q / S. P represents the success rate of acquiring a token within the current sliding time window, Q represents the throughput, and S represents the total number of concurrent connections.

[0112] For example, assuming a throughput of 5 qps and a total concurrency of 6 seconds, the success rate of obtaining a token in the current sliding time window is 5 / 6.

[0113] Step 212: Determine the number of retries based on the acceptable maximum retry duration and success rate.

[0114] In this embodiment of the application, the product of the maximum acceptable retry duration and the success rate is calculated to obtain the number of retries.

[0115] The number of retries is represented as: NR = T * P. T is the maximum acceptable retry duration, and P is the success rate of obtaining a token within the current sliding time window.

[0116] For example, assuming the maximum acceptable retry time is 6 seconds, and the success rate of obtaining a token in the current sliding time window is 5 / 6, then the number of retries to obtain the token again is 6 * 5 / 6 = 5 times.

[0117] Step 213: Determine the retry interval based on the maximum acceptable retry duration and the number of retries.

[0118] In this embodiment of the application, the ratio of the maximum acceptable retry duration to the number of retries is calculated to obtain the retry interval duration.

[0119] The retry interval is expressed as: TTL = T / NR. T is the maximum acceptable retry duration, and NR is the number of retries.

[0120] For example, assuming the maximum acceptable retry time is 6 seconds and the number of retries to reacquire the token is 5, then the retry interval for reacquiring the token is 6 / 5 = 1.2 seconds.

[0121] In this way, the time intervals are evenly distributed according to the maximum acceptable retry duration. This allows for a longer wait before attempting to acquire a token in high-concurrency scenarios (i.e., when the total number of concurrent requests is larger), reducing competition pressure. In low-concurrency scenarios (i.e., when the total number of concurrent requests is smaller), tokens are acquired more quickly, reducing waiting time, resulting in faster responses, increased retries, and improved response success rates. Furthermore, since the total number of concurrent requests varies within the current sliding event window, the calculated retry intervals also differ. By dispersing the retry intervals based on the varying concurrency levels of different requests, the time it takes for a successfully acquired token to reach the large model is also relatively dispersed, reducing the probability of the large model experiencing "illusion" events.

[0122] Based on the above embodiments, taking a large model throughput of 5 qps as an example, an average latency of 18s, a maximum acceptable latency of 24s, and a sliding time window length of 1s as an example, the following example illustrates how to determine the number of retries and the retry interval for reacquiring the token in this embodiment:

[0123] Example 1: When the total number of concurrent requests to be processed within a certain second has reached 5, and the current request to be processed fails to acquire a token on its first attempt, the total number of concurrent requests is now 6. Calculate the number of retries and the retry interval for the current request to be processed:

[0124] The number of retries NR = (MR) * Q / S = (24 - 18) * 5 / 6 = 5 times.

[0125] The retry interval TTL = (MR) / NR = 6 / 5 = 1.2s.

[0126] That is, it retryes to obtain the token for the current pending request every 1.2 seconds, and retryes 5 times.

[0127] Example 2: When the total number of concurrent requests to be processed within a certain second has reached 11, and the current request fails to acquire a token on its first attempt, the total number of concurrent requests is now 12. Calculate the number of retries and the retry interval for the current request to be processed:

[0128] Number of retries NR=(MR)*Q / S=(24-18)*5 / 12=2 times.

[0129] The retry interval TTL = (MR) / NR = (24-18) / 2 = 3s.

[0130] That is, it retryes to obtain the token of the current pending request every 3 seconds, and retryes twice.

[0131] Example 3: When the total number of concurrent requests to be processed within a certain second has reached 60, and the current request fails to acquire a token on its first attempt, the total number of concurrent requests is now 61. Calculate the number of retries and the retry interval for the current request to be processed:

[0132] The number of retries NR = (MR) * Q / S = (24 - 18) * 5 / 61 = 0 times.

[0133] That is, requests with a total concurrency exceeding 60 will no longer attempt to reacquire tokens, but will be directly added to the unanswered message queue.

[0134] In this way, as concurrent requests increase, the retry interval increases while the number of retries decreases. When the calculated number of retries is less than 1, no further retries are performed. The pending request is directly added to the unanswered message queue. At low concurrency levels, a higher number of retries ensures that each pending request receives a response; at high concurrency levels, fewer retries, or even no retries, reduce contention and give subsequent requests more opportunities to be answered.

[0135] In addition, in this embodiment of the application, the token corresponding to the request to be processed is obtained from the token bucket for the first time. If the token is obtained successfully for the first time, the request to be processed is reasoned based on the large model, the answer is generated and returned, and the question-answer pair of the request to be processed is stored in the question-answer library.

[0136] Step 22: Based on the number of retries and the retry interval, re-acquire tokens from the token bucket. When the token is successfully re-acquired, inference is performed on the request to be processed based on the large model, and an answer is generated and returned.

[0137] In this embodiment, the token acquisition is retried at retry intervals, and the retry count is repeated a certain number of times. When the token is successfully acquired again, the large model is used to infer the request to be processed, and an answer is generated and returned.

[0138] For example, assuming the retry interval is 1.2s and the number of retries is 5, then the token will be retried every 1.2s, and this will be repeated 5 times.

[0139] Optionally, in this embodiment of the application, after successfully reacquiring the token, reasoning about the request to be processed based on the large model, generating and returning the answer, the question-answer pair of the request to be processed is stored in the question-answer library to enrich the question-answer library.

[0140] In this embodiment of the application, the token is retried at each retry interval, and the number of retries is repeated. When the token is retried and fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

[0141] The notification message can be something like "System busy".

[0142] Optionally, in this application embodiment, a possible implementation is provided for processing the unanswered message queue, with the following specific steps:

[0143] Step A1: Periodically monitor the GPU utilization rate. When the GPU utilization rate is less than the preset utilization rate threshold, retrieve unanswered requests from the unanswered message queue and process them.

[0144] The preset occupancy rate threshold can be 60%, but this embodiment does not impose any restrictions on it.

[0145] In this embodiment of the application, the processing of unanswered requests includes the following steps:

[0146] Step A11: Based on the question-and-answer database, retrieve unanswered requests.

[0147] Step A12: If an answer to an unanswered request is found, then it is confirmed that the answer to the unanswered request has been obtained.

[0148] Step A13: If no answer is found for the unanswered request, the token corresponding to the unanswered request is initially obtained from the token bucket, and the total number of concurrent requests in the current sliding time window is incremented by 1.

[0149] Step A14: If the token acquisition is successful on the first attempt, reason about the unanswered requests based on the large model and generate answers to the unanswered requests.

[0150] Step A15: If the initial token acquisition fails, determine the number of retries and the retry interval based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model. Then, based on the number of retries and the retry interval, reacquire the token from the token bucket.

[0151] Step A16: When the token is successfully reacquired, reason about the unanswered request based on the large model and generate the answer to the unanswered request.

[0152] Step A17: If reacquiring the token fails, put the unanswered request back into the unanswered message queue.

[0153] Additionally, it is worth noting that in this embodiment of the application, when the GPU utilization rate is greater than or equal to a preset utilization rate threshold, the consumption of unanswered message queues is stopped.

[0154] Step A2: When the answer to an unanswered request is obtained, store the question-answer pair for the unanswered request in the question-answer database.

[0155] This enriches the question-answer pairs in the question-answer database, increases the hit probability of the database retrieval, and increases the probability that unanswered questions will be asked again by the questioner, making it easier for the questioner to quickly obtain answers to unanswered questions, while making full use of GPU resources.

[0156] Based on the above embodiments, referring to Figure 3, another flowchart of the request processing method in this application embodiment is shown, specifically including:

[0157] Step 30: Receive the question request from the requesting object and search for the question request based on the question-and-answer database.

[0158] Step 31: Determine whether the answer to the question request has been found. If yes, proceed to step 32; otherwise, proceed to step 33.

[0159] Step 32: Return the answer.

[0160] Step 33: Treat the question request as a pending request, obtain the token corresponding to the pending request from the token bucket, and increment the total number of concurrent requests in the current sliding time window by 1.

[0161] Step 34: Determine whether the token acquisition was successful. If yes, proceed to step 35; otherwise, proceed to step 36.

[0162] Step 35: Based on the large model, reason about the request to be processed, generate answers, and store the question-answer pairs of the request to be processed in the question-answer database.

[0163] Step 36: Based on the total number of concurrent connections, as well as the throughput, average latency, and maximum acceptable latency of the large model, determine the number of retries and the retry interval for re-acquiring the token.

[0164] Step 37: Attempt to acquire a token at retry intervals.

[0165] Step 38: Determine whether a token has been obtained. If yes, proceed to step 35; otherwise, proceed to step 39.

[0166] Step 39: Determine if the number of attempts has exceeded the number of retries. If yes, proceed to step 310; otherwise, proceed to step 37.

[0167] Step 310: Return a prompt message to the requesting object and add the pending request to the unanswered message queue.

[0168] In this embodiment of the application, the GPU utilization rate of the image processor is monitored periodically. When the GPU utilization rate is less than a preset utilization rate threshold, unanswered requests are retrieved from the unanswered message queue and processed. When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer database.

[0169] Based on the same inventive concept, this application provides a request processing device. Referring to FIG4, the device includes:

[0170] The processing module 401 is used to obtain the request to be processed, obtain the token corresponding to the request to be processed from the token bucket for the first time, and increment the total number of concurrent requests in the current sliding time window by 1.

[0171] The determination module 402 is used to determine the number of retries and the retry interval if the initial token acquisition fails, based on the total number of concurrent requests, as well as the throughput, average latency and maximum acceptable latency of the large model.

[0172] The retry module 403 is used to re-acquire tokens from the token bucket based on the number of retries and the retry interval. When the token is successfully re-acquired, the module performs reasoning on the request to be processed based on the large model, generates and returns the answer.

[0173] In one possible embodiment, when determining the number of retries and the retry interval for reacquiring a token based on the total number of concurrent connections, as well as the throughput, average latency, and maximum acceptable latency of the large model, the determining module 402 is further configured to:

[0174] The maximum acceptable retry duration is determined based on the maximum acceptable latency and the average latency.

[0175] Based on throughput and total concurrency, determine the success rate of acquiring a token in the current sliding time window;

[0176] The number of retries is determined based on the acceptable maximum retry duration and success rate;

[0177] The retry interval is determined based on the maximum acceptable retry duration and the number of retries.

[0178] In one possible embodiment, when obtaining a request to be processed, the processing module 401 is further configured to:

[0179] The system receives a question request from a requesting object and retrieves the question request based on a question-and-answer database, which is used to store historical question-and-answer pairs.

[0180] If no answer is found for the question request, the question request will be treated as a pending request.

[0181] In one possible embodiment, after reasoning about the request to be processed based on a large model, generating and returning an answer, the device further includes a storage module 404, which is used for:

[0182] Store the question-and-answer pairs for the requests to be processed in the question-and-answer database.

[0183] In one possible embodiment, the determining module 402 is further configured to:

[0184] If the token is successfully obtained on the first attempt, the system will infer the request to be processed based on the large model, generate and return the answer, and store the question-answer pair of the request to be processed in the question-answer database.

[0185] In one possible embodiment, the retry module 403 is further configured to:

[0186] If reacquiring the token fails, return a prompt message to the request object of the pending request and add the pending request to the unanswered message queue.

[0187] In one possible embodiment, the device further includes a monitoring module 405, which is used for:

[0188] The GPU utilization rate of the image processor is monitored periodically. When the GPU utilization rate is lower than the preset utilization rate threshold, unanswered requests are retrieved from the unanswered message queue and processed.

[0189] When an answer is received for an unanswered request, the question-answer pair for the unanswered request is stored in the question-answer database.

[0190] Based on the above embodiments, Figure 5 shows a schematic diagram of the structure of the electronic device in the embodiment of this application.

[0191] This application provides an electronic device that may include a processor 510 (Center Processing Unit, CPU), a memory 520, an input device 530, and an output device 540. The input device 530 may include a keyboard, a mouse, a touch screen, etc., and the output device 540 may include a display device, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0192] The memory 520 may include read-only memory (ROM) and random access memory (RAM), and provides the processor 510 with program instructions and data stored in the memory 520. In this embodiment, the memory 520 may be used to store the program of any request processing method in this embodiment.

[0193] The processor 510 executes any request processing method in the embodiments of this application according to the program instructions stored in the memory 520 by calling the program instructions.

[0194] Based on the above embodiments, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the request processing method in any of the above method embodiments.

[0195] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0197] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0198] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0199] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A request processing method, characterized in that, include: Get the pending requests, and initially obtain the token corresponding to the pending requests from the token bucket, and increment the total number of concurrent requests in the current sliding time window by 1; If the initial acquisition of the token fails, the number of retries and the retry interval are determined based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model. Based on the number of retries and the retry interval, the token is retrieved again from the token bucket. When the token is successfully retrieved again, the pending request is inferred based on the large model to generate and return an answer.

2. The method as described in claim 1, characterized in that, The determination of the number of retries and retry interval for re-acquiring the token based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model, includes: Based on the maximum acceptable latency and the average latency, determine the maximum acceptable retry duration; Based on the throughput and the total number of concurrent requests, determine the success rate of acquiring the token in the current sliding time window; The number of retries is determined based on the acceptable maximum retry duration and the success rate; The retry interval is determined based on the maximum acceptable retry duration and the number of retries.

3. The method as described in claim 1, characterized in that, The process of obtaining requests to be processed includes: The system receives a question request from a requesting object and retrieves the question request based on a question-and-answer database, wherein the question-and-answer database is used to store historical question-and-answer pairs. If no answer is found for the question request, the question request will be treated as a pending request.

4. The method as described in claim 3, characterized in that, After reasoning about the request to be processed based on the large model, generating and returning the answer, the process further includes: The question-and-answer pairs of the requests to be processed are stored in the question-and-answer database.

5. The method as described in claim 1, characterized in that, The method further includes: If the token is successfully acquired on the first attempt, the system will infer the request to be processed based on the large model, generate and return an answer, and store the question-answer pair of the request to be processed in the question-answer database.

6. The method as described in claim 1, characterized in that, The method further includes: If reacquiring the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

7. The method as described in claim 1, characterized in that, The method further includes: The GPU utilization rate of the image processor is monitored periodically. When the GPU utilization rate is less than a preset utilization rate threshold, unanswered requests are retrieved from the unanswered message queue and processed. When an answer is obtained for an unanswered request, the question-answer pair for the unanswered request is stored in the question-answer database.

8. A request processing apparatus, characterized in that, include: The processing module is used to obtain the request to be processed, obtain the token corresponding to the request to be processed from the token bucket for the first time, and increment the total number of concurrent requests in the current sliding time window by 1. The determination module is used to determine the number of retries and the retry interval if the initial acquisition of the token fails, based on the total number of concurrent requests, as well as the throughput, average latency and maximum acceptable latency of the large model. The retry module is used to retrieve the token from the token bucket based on the number of retries and the retry interval. When the token is successfully retrieved, the module performs reasoning on the pending request based on the large model, generates and returns an answer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images