Request processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a request processing method and device, electronic equipment and a storage medium, which are used for preventing illusion of a large model due to high concurrency and improving the availability of the large model. The method comprises the following steps: acquiring a to-be-processed request, initially acquiring a token corresponding to the to-be-processed request from a token bucket, and adding 1 to the total concurrent number of a current sliding time window; if the token is failed to be acquired for the first time, determining the number of retry times and the retry interval duration for re-acquiring the token based on the total concurrent number and the throughput, the average time delay and the maximum acceptable time delay of the large model; and on the basis of the number of times of retry and the retry interval duration, re-acquiring the token from the token bucket, and when the token is successfully re-acquired, reasoning the to-be-processed request on the basis of the large model, and generating and returning an answer. Thus, according to the real-time total concurrent amount, the number of retry times and the retry interval duration are dynamically adjusted, the requests are dispersed to the large model, and the response success rate and availability are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a request processing method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application of large models is becoming more and more common. Large models often need to process a large number of requests from different users or devices at the same time, resulting in high concurrency.

[0003] Due to the complexity and slow reasoning speed of large models, hallucinations are likely to occur in high-concurrency scenarios, and they may even fail to recover for a long time.

[0004] Therefore, how to provide a request processing method to prevent large models from experiencing hallucinations due to high concurrency and improve the availability of large models is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The embodiments of the present application provide a request processing method, device, electronic device, and storage medium to prevent hallucinations in large models due to high concurrency and improve the availability of large models.

[0006] The specific technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, a request processing method is provided, the method comprising:

[0008] Get the pending request, get the token corresponding to the pending request from the token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1;

[0009] If the initial token acquisition fails, the number of retries and the retry interval for re-acquiring the token are determined based on the total number of concurrency, as well as the throughput, average latency, and maximum acceptable latency of the large model;

[0010] Based on the number of retries and the length of the retry interval, the token is retrieved from the token bucket. When the token is retrieved successfully, the request to be processed is inferred based on the large model to generate and return an answer.

[0011] In a possible embodiment, based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of a large model, the number of retries and the retry interval for reacquiring a token are determined, including:

[0012] Based on the maximum acceptable delay and average delay, determine the maximum acceptable retry duration;

[0013] Based on the throughput and total number of concurrent connections, determine the success rate of acquiring tokens in the current sliding time window;

[0014] Determine the number of retries based on the maximum acceptable retry duration and success rate;

[0015] Determine the retry interval length based on the maximum acceptable retry duration and the number of retries.

[0016] Through the above method, the time interval of each time is evenly distributed according to the acceptable maximum retry time, so that in high concurrency scenarios, that is, when the total number of concurrency is larger, it takes a longer time to grab the token to reduce competition pressure. In low concurrency scenarios, that is, when the total number of concurrency is small, it is faster to grab the token, reduce waiting time, respond faster, increase the number of retries, and improve the response success rate. At the same time, when each request in the current sliding event window comes in, the corresponding total number of concurrency is different, and the calculated retry interval is also different. According to the different total number of concurrency of different requests, the retry interval is dispersed, and the time to reach the large model when the token is successfully obtained is also relatively dispersed, reducing the probability of hallucinations in the large model.

[0017] In a possible embodiment, obtaining a pending request includes:

[0018] receiving a question request from a request object, and searching the question request based on a question-answer database, wherein the question-answer database is used to store historical question-answer pairs;

[0019] If the answer corresponding to the question request is not retrieved, the question request will be treated as a pending request.

[0020] Through the above method, the question request is answered using the question and answer library first, which can reduce the repeated generation of answers to similar questions, reduce the total number of concurrency, and increase the answer speed, thereby improving the user experience.

[0021] In a possible embodiment, after reasoning about the request to be processed based on the large model and generating and returning an answer, the following steps are also included:

[0022] The question-answer pairs of pending requests are stored in the question-answer library.

[0023] Through the above methods, the question and answer database is enriched.

[0024] In a possible embodiment, the method further includes:

[0025] If the token is acquired successfully for the first time, the request to be processed is inferred based on the large model, the answer is generated and returned, and the question and answer pair of the request to be processed is stored in the question and answer library.

[0026] In a possible embodiment, the method further includes:

[0027] When re-acquisition of the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

[0028] In a possible embodiment, the method further includes:

[0029] Regularly monitor the GPU occupancy rate of the image processor. When the GPU occupancy rate is less than a preset occupancy rate threshold, take out the unanswered requests from the unanswered message queue and process the unanswered requests.

[0030] When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer library.

[0031] Through the above method, the question-answer pairs in the question-answer library are enriched, the hit probability of the question-answer library retrieval is increased, and the probability of the unsuccessful questions being asked again by the question object is high, which makes it easier for the question object to quickly obtain the answer to the unsuccessful question, while making full use of the GPU resources.

[0032] In a second aspect, a request processing device is provided, the device comprising:

[0033] The processing module is used to obtain pending requests, obtain the token corresponding to the pending request from the token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1;

[0034] A determination module, used to determine the number of retries and the retry interval for re-acquiring the token if the initial acquisition of the token fails, based on the total number of concurrency, and the throughput, average latency, and maximum acceptable latency of the large model;

[0035] The retry module is used to re-acquire the token from the token bucket based on the number of retries and the retry interval. When the token is successfully re-acquired, the request to be processed is inferred based on the large model to generate and return an answer.

[0036] In a possible embodiment, when determining the number of retries and the retry interval for reacquiring a token based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of a large model, the determination module is further configured to:

[0037] Based on the maximum acceptable delay and average delay, determine the maximum acceptable retry duration;

[0038] Based on the throughput and total number of concurrent connections, determine the success rate of acquiring tokens in the current sliding time window;

[0039] Determine the number of retries based on the maximum acceptable retry duration and success rate;

[0040] Determine the retry interval length based on the maximum acceptable retry duration and the number of retries.

[0041] In a possible embodiment, when obtaining the request to be processed, the processing module is further used to:

[0042] receiving a question request from a request object, and searching the question request based on a question-answer database, wherein the question-answer database is used to store historical question-answer pairs;

[0043] If the answer corresponding to the question request is not retrieved, the question request will be treated as a pending request.

[0044] In a possible embodiment, after reasoning on the request to be processed based on the large model and generating and returning an answer, the device further includes a storage module, which is used to:

[0045] The question-answer pairs of pending requests are stored in the question-answer library.

[0046] In a possible embodiment, the determination module is further configured to:

[0047] If the token is acquired successfully for the first time, the request to be processed is inferred based on the large model, the answer is generated and returned, and the question and answer pair of the request to be processed is stored in the question and answer library.

[0048] In a possible embodiment, the retry module is further configured to:

[0049] When re-acquisition of the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

[0050] In a possible embodiment, the device further includes a monitoring module, and the monitoring module is used to:

[0051] Regularly monitor the GPU occupancy rate of the image processor. When the GPU occupancy rate is less than a preset occupancy rate threshold, take out the unanswered requests from the unanswered message queue and process the unanswered requests.

[0052] When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer library.

[0053] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0054] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0055] In the embodiment of the present application, a pending request is obtained, and the token corresponding to the pending request is obtained from the token bucket for the first time, and the total number of concurrency of the current sliding time window is increased by 1. If the token fails to be obtained for the first time, the number of retries and the length of the retry interval for re-acquiring the token are determined based on the total number of concurrency, the throughput, average delay and maximum acceptable delay of the large model, and then the token is re-acquired from the token bucket based on the number of retries and the length of the retry interval. When the token is successfully re-acquired, the pending request is inferred based on the large model to generate and return the answer. In this way, through the token bucket, high concurrency scenarios are coped with and the availability of the large model is improved. The total number of concurrency corresponding to different requests is different, and the calculated retry interval is dispersed. The requests are dispersed to the large model, which reduces the probability of hallucination of the large model and prevents the large model from hallucination due to high concurrency. In addition, the fixed token current limiting algorithm can easily trigger the flow threshold, resulting in a low response rate of the large model. Therefore, the number of retries and the length of the retry interval are dynamically adjusted according to the real-time total concurrency within the acceptable delay range, thereby improving the response success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a system architecture diagram of a request processing system in an embodiment of the present application;

[0057] Figure 2 A flowchart of a request processing method provided in an embodiment of the present application;

[0058] Figure 3 Another flowchart of the request processing method in the embodiment of the present application is shown;

[0059] Figure 4 A schematic diagram of the structure of a request processing device provided in an embodiment of the present application;

[0060] Figure 5 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0062] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0063] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0064] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0065] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0066] Frequently-asked questions (FAQ): Organize some possible FAQs in advance and search and answer the questions.

[0067] Token Bucket: It is a common flow control algorithm. Tokens are generated at a fixed rate, usually a certain number of tokens per second. These tokens are placed in a bucket with a fixed maximum capacity. Whenever a request comes, the system will try to get a token from the bucket. If there is a token in the bucket, the request is allowed to execute and a token is taken from the bucket. If there is no token, the request will be rejected or delayed.

[0068] Sliding window current limiting algorithm: an algorithm used to control the data flow rate. It divides the time window into multiple small cycles. Each small cycle has its own counter. As time slides, expired small cycle data is deleted, thereby controlling the traffic more accurately.

[0069] The following is a brief introduction to the design concept of the embodiment of the present application:

[0070] With the development of artificial intelligence technology, the application of large models is becoming more and more common. Large models often need to process a large number of requests from different users or devices at the same time, resulting in high concurrency.

[0071] Due to the complexity and slow reasoning speed of large models, hallucinations are likely to occur in high-concurrency scenarios, and they may even fail to recover for a long time.

[0072] In the existing technology, high-concurrency scenarios are dealt with by optimizing the performance of the large model itself.

[0073] However, the above approaches often involve a trade-off between accuracy and performance, and may increase computing resource requirements, latency, complexity, and long-term maintenance costs.

[0074] Therefore, how to prevent the large model from experiencing hallucinations due to high concurrency while not optimizing the performance of the large model itself and improve the availability of the large model is an urgent problem that technicians in this field need to solve.

[0075] In view of this, in an embodiment of the present application, a request processing method, device, electronic device and storage medium are provided to obtain pending requests, and to obtain the token corresponding to the pending request from the token bucket for the first time, and to increase the total concurrency of the current sliding time window by 1. If the initial acquisition of the token fails, the number of retries and the length of the retry interval for re-acquiring the token are determined based on the total number of concurrency, as well as the throughput, average latency and maximum acceptable latency of the large model, and then the token is re-acquired from the token bucket based on the number of retries and the length of the retry interval. When the token is successfully re-acquired, the pending request is inferred based on the large model to generate and return an answer. In this way, through the token bucket, high concurrency scenarios are coped with and the availability of the large model is improved. The total concurrency corresponding to different requests is different, and the calculated retry intervals are dispersed. The requests are dispersed to reach the large model, which reduces the probability of hallucinations in the large model and prevents the large model from hallucinations due to high concurrency. In addition, the fixed token current limiting algorithm can easily trigger the traffic threshold, resulting in low response rate for large models. Therefore, the number of retries and the length of the retry interval are dynamically adjusted according to the real-time total concurrency within the acceptable delay range, thereby improving the response success rate.

[0076] See also Figure 1 , which is a system architecture diagram of a request processing system in an embodiment of the present application. The request processing system includes a stress testing module 110, a FAQ module 120, a current limiting module 130, a large model reasoning generation module 140, and a monitoring module 150.

[0077] Stress testing module 110: performs stress testing on the large model generation module to obtain the throughput and average latency of the large model and define the maximum acceptable latency.

[0078] FAQ module 120: Maintain a high-quality question and answer database, and retrieve the question requests sent by users based on the question and answer database to directly obtain answers to the question requests without large model reasoning.

[0079] Current limiting module 130: Based on the token bucket algorithm, sliding window counting algorithm and dynamic retry mechanism, it limits the current of question requests before entering the large model.

[0080] The large model reasoning generation module 140 generates an answer to the question request and stores the question-answer pair corresponding to the question request in the question-answer library.

[0081] Monitoring module 150: Regularly monitor the GPU occupancy rate of the image processor. When the GPU occupancy rate is lower than a preset occupancy rate threshold, take out the unanswered request from the unanswered message queue for processing. When the answer corresponding to the unanswered request is obtained, the question and answer pair of the unanswered request is stored in the question and answer library.

[0082] See also Figure 2 As shown, it is an implementation flow chart of a request processing method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0083] Step 20: Get the pending request, and get the token corresponding to the pending request from the token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1.

[0084] Among them, the pending request is a request that needs to generate an answer through large model reasoning, the current sliding time window is the sliding time window in the sliding window current limiting algorithm, and the statistical duration of each sliding time window is set, for example, the statistical duration is set to 1s.

[0085] In an embodiment of the present application, after obtaining the pending request, the token corresponding to the pending request is obtained from the token bucket for the first time, and the counter of the current sliding time window is increased by 1, and the counter statistic is the total number of concurrencies in the current sliding time window.

[0086] Among them, based on the throughput of the large model, the rate of issuing tokens and the maximum capacity of the token bucket are determined.

[0087] Optionally, in the embodiment of the present application, a possible implementation is provided to obtain a request to be processed, and the specific steps are as follows:

[0088] Step 200: Receive a question request from a request object, and search the question request based on a question and answer database.

[0089] Among them, the question and answer library is used to store historical question and answer pairs. The question and answer library can be a FAQ library, which is not limited in the embodiments of the present application. The question and answer pair is a question-answer pair.

[0090] In the embodiment of the present application, based on the question and answer database, the target question contained in the question request is searched, and if similar questions similar to the target question are searched, the answers to the similar questions are used as the answers to the target question.

[0091] In addition, it is worth noting that in the embodiment of the present application, historical question and answer pairs are regularly clustered and low-quality filtered to maintain a high-quality question and answer library.

[0092] Step 201: Determine whether the answer corresponding to the question request is not retrieved, if so, execute step 202, otherwise, execute step 203.

[0093] In an embodiment of the present application, if a similar question similar to the target question included in the question request is retrieved, it is determined that the answer corresponding to the question request is retrieved, and the answer to the similar question is used as the answer to the target question. If a similar question similar to the target question included in the question request is not retrieved, it is determined that the answer corresponding to the question request is not retrieved.

[0094] Step 202: The question request is treated as a request to be processed.

[0095] In the embodiment of the present application, if the answer corresponding to the question request is not retrieved, the question request will be treated as a pending request.

[0096] Step 203: Return the answer to the request object.

[0097] In the embodiment of the present application, if the answer corresponding to the question request is retrieved, the answer is returned to the request object.

[0098] In this way, answering the question request using the question and answer library first can reduce the repeated generation of answers to similar questions, reduce the total number of concurrency, and increase the answer speed, thereby improving the user experience.

[0099] Step 21: If the initial token acquisition fails, the number of retries and the retry interval for re-acquiring the token are determined based on the total number of concurrency, as well as the throughput, average latency, and maximum acceptable latency of the large model.

[0100] Among them, throughput refers to the number of requests processed by the large model per unit time, which can also be understood as the amount of input data processed per second. Average latency refers to the average time required for the large model to process a request and generate a response. Maximum acceptable latency refers to the maximum value of the model response delay that users can tolerate in large model applications.

[0101] In addition, it is worth noting that in the embodiments of the present application, by stress testing the large model, the throughput and average latency of the large model are obtained, and the maximum acceptable latency is defined.

[0102] Optionally, in the embodiment of the present application, a possible embodiment is provided to determine the number of retries and the length of the retry interval, and the specific steps are as follows:

[0103] Step 210: Determine an acceptable maximum retry duration based on the maximum acceptable delay and the average delay.

[0104] In the embodiment of the present application, the difference between the maximum acceptable delay and the average delay is calculated to obtain the acceptable maximum retry duration.

[0105] The maximum acceptable retry time is expressed as: T = MR. T is the maximum acceptable retry time, M is the maximum acceptable delay, and R is the average delay.

[0106] For example, assuming that the maximum acceptable delay is 24 seconds and the average delay is 18 seconds, the maximum acceptable retry duration is 24-18=6 seconds.

[0107] Step 211: Based on the throughput and the total number of concurrent connections, determine the success rate of acquiring tokens in the current sliding time window.

[0108] In the embodiment of the present application, the ratio of the throughput to the total number of concurrency is calculated to obtain the success rate of acquiring the token in the current sliding time window.

[0109] The success rate of acquiring tokens in the current sliding time window is expressed as: P = Q / S. P is the success rate of acquiring tokens in the current sliding time window, Q is the throughput, and S is the total number of concurrency.

[0110] For example, assuming the throughput is 5qps and the total concurrency is 6s, the success rate of acquiring tokens in the current sliding time window is 5 / 6.

[0111] Step 212: Determine the number of retries based on the acceptable maximum retry duration and success rate.

[0112] In the embodiment of the present application, the product of the maximum acceptable retry duration and the success rate is calculated to obtain the number of retries.

[0113] The number of retries is expressed as: NR = T*P. T is the maximum acceptable retry duration, and P is the success rate of acquiring the token in the current sliding time window.

[0114] For example, assuming that the maximum acceptable retry duration is 6 seconds, and the success rate of acquiring the token in the current sliding time window is 5 / 6, the number of retries for re-acquiring the token is 6*5 / 6=5 times.

[0115] Step 213: Determine the retry interval duration based on the acceptable maximum retry duration and the number of retries.

[0116] In the embodiment of the present application, the ratio of the maximum acceptable retry duration to the number of retries is calculated to obtain the retry interval duration.

[0117] The retry interval is expressed as: TTL = T / NR. T is the maximum acceptable retry interval, and NR is the number of retries.

[0118] For example, assuming that the maximum acceptable retry duration is 6 seconds, and the number of retries for re-acquiring the token is 5 times, the retry interval for re-acquiring the token is 6 / 5=1.2 seconds.

[0119] In this way, the time interval of each retry is evenly distributed according to the maximum acceptable retry time, so that in high-concurrency scenarios, that is, when the total number of concurrency is larger, it takes a longer time to grab the token, reducing competition pressure. In low-concurrency scenarios, that is, when the total number of concurrency is small, it is faster to grab the token, reduce waiting time, respond faster, increase the number of retries, and improve the response success rate. At the same time, when each request in the current sliding event window comes in, the corresponding total number of concurrency is different, and the calculated retry interval is also different. According to the different total number of concurrency of different requests, the retry interval is dispersed, and the time to reach the large model when the token is successfully obtained is also relatively dispersed, reducing the probability of hallucinations in the large model.

[0120] Based on the above embodiment, taking the large model throughput of 5qps as an example, the average delay is 18s, the maximum acceptable delay is 24s, and the length of each sliding time window is 1s as an example, the number of retries and the retry interval for reacquiring the token in the embodiment of the present application are determined by example:

[0121] Example 1: When the total number of concurrent pending requests within a certain 1 second has reached 5, and the current pending request arrives, the initial token acquisition fails. At this time, the total number of concurrent requests is 6. Calculate the number of retries and the retry interval of the current pending request:

[0122] Number of retries NR = (MR)*Q / S = (24-18)*5 / 6 = 5 times.

[0123] The retry interval duration TTL=(MR) / NR=6 / 5=1.2s.

[0124] That is, retry to obtain the token of the current pending request every 1.2 seconds, and retry 5 times.

[0125] Example 2: When the total number of concurrent pending requests within a certain 1 second has reached 11, and the current pending request arrives, the initial token acquisition fails. At this time, the total number of concurrent requests is 12. Calculate the number of retries and the retry interval of the current pending request:

[0126] Number of retries NR = (MR)*Q / S = (24-18)*5 / 12 = 2 times.

[0127] The retry interval duration TTL = (MR) / NR = (24-18) / 2 = 3s.

[0128] That is, retry to obtain the token of the current pending request every 3 seconds, and retry twice.

[0129] Example 3: When the total number of concurrent pending requests within a certain 1 second has reached 60, and the current pending request arrives, the initial token acquisition fails. At this time, the total number of concurrent requests is 61. Calculate the number of retries and the retry interval of the current pending request:

[0130] Number of retries NR = (MR)*Q / S = (24-18)*5 / 61 = 0 times.

[0131] That is, after the total number of concurrent requests exceeds 60, the pending requests will no longer attempt to re-acquire the token, and will be directly added to the unanswered message queue.

[0132] In this way, as the number of concurrent requests increases, the retry interval increases and the number of retries decreases. When the calculated number of retries is less than 1, no more retries are performed. The pending request is directly added to the unanswered message queue. When the concurrency is low, a large number of retries ensures that each pending request can be answered. When the concurrency is high, a small number of retries or even no retries can reduce competition pressure and give subsequent requests more opportunities to be answered.

[0133] In addition, in an embodiment of the present application, a token corresponding to a pending request is initially obtained from a token bucket. If the initial acquisition of the token is successful, the pending request is inferred based on a large model, an answer is generated and returned, and the question and answer pair of the pending request is stored in a question and answer library.

[0134] Step 22: Based on the number of retries and the retry interval, the token is retrieved from the token bucket. When the token is retrieved successfully, the request to be processed is inferred based on the large model to generate and return an answer.

[0135] In the embodiment of the present application, the token is retried every retry interval, and the retry number is repeated. When the token is successfully re-acquired, the request to be processed is inferred based on the large model, and an answer is generated and returned.

[0136] For example, if the retry interval is 1.2 seconds and the number of retries is 5, then the token will be retried every 1.2 seconds for 5 times.

[0137] Optionally, in an embodiment of the present application, after the token is successfully re-acquired and the request to be processed is inferred based on the large model and the answer is generated and returned, the question and answer pairs of the request to be processed are stored in the question and answer library to enrich the question and answer library.

[0138] In the embodiment of the present application, an attempt is made to obtain the token every retry interval, and the retry is repeated for a number of times. When re-acquisition of the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

[0139] The prompt message may be a reply such as "system busy".

[0140] Optionally, in an embodiment of the present application, a possible embodiment is provided for processing an unanswered message queue, and the specific steps are as follows:

[0141] Step A1: regularly monitor the GPU occupancy rate of the image processor. When the GPU occupancy rate is less than a preset occupancy rate threshold, take out the unanswered request from the unanswered message queue and process the unanswered request.

[0142] Among them, the preset occupancy rate threshold may be 60%, which is not limited in the comparison in the embodiment of the present application.

[0143] In the embodiment of the present application, processing the unanswered request includes the following steps:

[0144] Step A11: Based on the question and answer database, unanswered requests are retrieved.

[0145] Step A12: If the answer corresponding to the unanswered request is retrieved, it is determined that the answer corresponding to the unanswered request is obtained.

[0146] Step A13: If the answer corresponding to the unanswered request is not retrieved, the token corresponding to the unanswered request is obtained from the token bucket for the first time, and the total number of concurrency in the current sliding time window is increased by 1.

[0147] Step A14: If the initial acquisition of the token is successful, the unanswered requests are inferred based on the large model to generate answers to the unanswered requests.

[0148] Step A15: If the initial acquisition of the token fails, the number of retries and the retry interval for re-acquiring the token are determined based on the total number of concurrency, as well as the throughput, average latency and maximum acceptable latency of the large model, and then the token is re-acquired from the token bucket based on the number of retries and the retry interval.

[0149] Step A16: When the token is successfully retrieved, the unanswered requests are inferred based on the large model to generate answers to the unanswered requests.

[0150] Step A17: When re-acquisition of the token fails, the unanswered request is put back into the unanswered message queue.

[0151] In addition, it is worth noting that, in the embodiment of the present application, when the GPU occupancy rate is greater than or equal to the preset occupancy rate threshold, consumption of the unanswered message queue is stopped.

[0152] Step A2: When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer library.

[0153] In this way, the question-answer pairs in the question-answer library are enriched, the hit probability of the question-answer library retrieval is increased, and the probability of the unanswered questions being asked again by the question object is high, which makes it easier for the question object to quickly obtain the answers to the unanswered questions and makes full use of the GPU resources.

[0154] Based on the above embodiments, see Figure 3 FIG. 1 is another flowchart of a request processing method in an embodiment of the present application, which specifically includes:

[0155] Step 30: Receive a question request from the request object, and search the question request based on the question and answer database.

[0156] Step 31: Determine whether the answer corresponding to the question request is retrieved, if so, execute step 32, otherwise, execute step 33.

[0157] Step 32: Return the answer.

[0158] Step 33: The question request is treated as a pending request, a token corresponding to the pending request is obtained from the token bucket, and the total number of concurrent requests in the current sliding time window is increased by 1.

[0159] Step 34: Determine whether the token acquisition is successful, if so, execute step 35, otherwise, execute step 36.

[0160] Step 35: Reasoning is performed on the request to be processed based on the large model, an answer is generated, and the question-answer pair of the request to be processed is stored in the question-answer library.

[0161] Step 36: Based on the total number of concurrent users, as well as the throughput, average latency, and maximum acceptable latency of the large model, determine the number of retries and the retry interval for reacquiring the token.

[0162] Step 37: Try to obtain a token every retry interval.

[0163] Step 38: Determine whether the token is obtained. If so, execute step 35; otherwise, execute step 39.

[0164] Step 39: Determine whether the number of attempts exceeds the number of retries. If so, execute step 310; otherwise, execute step 37.

[0165] Step 310: Return a prompt message to the request object, and add the pending request to the unanswered message queue.

[0166] In an embodiment of the present application, the GPU occupancy rate of the image processor is monitored at regular intervals. When the GPU occupancy rate is less than a preset occupancy rate threshold, unanswered requests are taken out from the unanswered message queue and processed. When the answers corresponding to the unanswered requests are obtained, the question and answer pairs of the unanswered requests are stored in the question and answer library.

[0167] Based on the same inventive concept, the present application embodiment provides a request processing device, please refer to Figure 4 , the device comprises:

[0168] Processing module 401 is used to obtain pending requests, obtain tokens corresponding to pending requests from a token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1;

[0169] A determination module 402 is used to determine the number of retries and the length of the retry interval for re-acquiring the token if the initial acquisition of the token fails based on the total number of concurrency, and the throughput, average latency and maximum acceptable latency of the large model;

[0170] The retry module 403 is used to re-acquire the token from the token bucket based on the number of retries and the retry interval. When the token is successfully re-acquired, the request to be processed is inferred based on the large model to generate and return an answer.

[0171] In a possible embodiment, when determining the number of retries and the retry interval for reacquiring a token based on the total number of concurrent requests, and the throughput, average latency, and maximum acceptable latency of a large model, the determination module 402 is further configured to:

[0172] Based on the maximum acceptable delay and average delay, determine the maximum acceptable retry duration;

[0173] Based on the throughput and total number of concurrent connections, determine the success rate of acquiring tokens in the current sliding time window;

[0174] Determine the number of retries based on the maximum acceptable retry duration and success rate;

[0175] Determine the retry interval length based on the maximum acceptable retry duration and the number of retries.

[0176] In a possible embodiment, when obtaining the request to be processed, the processing module 401 is further used to:

[0177] receiving a question request from a request object, and searching the question request based on a question-answer database, wherein the question-answer database is used to store historical question-answer pairs;

[0178] If the answer corresponding to the question request is not retrieved, the question request will be treated as a pending request.

[0179] In a possible embodiment, after inferring the request to be processed based on the large model, generating and returning an answer, the device further includes a storage module 404, which is used to:

[0180] The question-answer pairs of pending requests are stored in the question-answer library.

[0181] In a possible embodiment, the determination module 402 is further configured to:

[0182] If the token is acquired successfully for the first time, the request to be processed is inferred based on the large model, the answer is generated and returned, and the question and answer pair of the request to be processed is stored in the question and answer library.

[0183] In a possible embodiment, the retry module 403 is further configured to:

[0184] When re-acquisition of the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to the unanswered message queue.

[0185] In a possible embodiment, the device further includes a monitoring module 405, and the monitoring module 405 is used to:

[0186] Regularly monitor the GPU occupancy rate of the image processor. When the GPU occupancy rate is less than a preset occupancy rate threshold, take out the unanswered requests from the unanswered message queue and process the unanswered requests.

[0187] When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer library.

[0188] Based on the above embodiments, see Figure 5 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0189] An embodiment of the present application provides an electronic device, which may include a processor 510 (Center Processing Unit, CPU), a memory 520, an input device 530 and an output device 540, etc. The input device 530 may include a keyboard, a mouse, a touch screen, etc., and the output device 540 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0190] The memory 520 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 510 with program instructions and data stored in the memory 520. In the embodiment of the present application, the memory 520 may be used to store the program of any request processing method in the embodiment of the present application.

[0191] The processor 510 calls the program instructions stored in the memory 520, and the processor 510 is used to execute any request processing method in the embodiments of the present application according to the obtained program instructions.

[0192] Based on the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the request processing method in any of the above method embodiments is implemented.

[0193] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0194] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0197] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A request processing method, characterized in that: include: Obtain pending requests, obtain the token corresponding to the pending requests from the token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1; If the initial acquisition of the token fails, the number of retries and the retry interval for re-acquiring the token are determined based on the total number of concurrent requests, as well as the throughput, average latency, and maximum acceptable latency of the large model; Based on the number of retries and the retry interval, the token is retrieved from the token bucket. When the token is retrieved successfully, the pending request is inferred based on the large model to generate and return an answer.

2. The method according to claim 1, characterized in that Determining the number of retries and the retry interval for reacquiring the token based on the total number of concurrent connections, as well as the throughput, average latency, and maximum acceptable latency of the large model, includes: Determining an acceptable maximum retry duration based on the maximum acceptable delay and the average delay; Determine a success rate of acquiring the token in the current sliding time window based on the throughput and the total number of concurrent connections; Determining the number of retries based on the acceptable maximum retry duration and the success rate; The retry interval duration is determined based on the acceptable maximum retry duration and the number of retries.

3. The method according to claim 1, characterized in that The obtaining of pending requests includes: receiving a question request from a request object, and searching the question request based on a question-answer database, wherein the question-answer database is used to store historical question-answer pairs; If the answer corresponding to the question request is not retrieved, the question request is treated as a request to be processed.

4. The method according to claim 3, characterized in that After the request to be processed is inferred based on the large model and an answer is generated and returned, the method further includes: The question-answer pair of the request to be processed is stored in the question-answer library.

5. The method according to claim 1, characterized in that The method further comprises: If the token is successfully acquired for the first time, the request to be processed is inferred based on the large model, an answer is generated and returned, and the question-answer pair of the request to be processed is stored in a question-answer library.

6. The method according to claim 1, characterized in that The method further comprises: When re-acquisition of the token fails, a prompt message is returned to the request object of the pending request, and the pending request is added to an unanswered message queue.

7. The method according to claim 1, characterized in that The method further comprises: Regularly monitor the GPU occupancy rate of the image processor, and when the GPU occupancy rate is less than a preset occupancy rate threshold, take out the unanswered request from the unanswered message queue and process the unanswered request; When the answer corresponding to the unanswered request is obtained, the question-answer pair of the unanswered request is stored in the question-answer library.

8. A request processing device, characterized in that: include: A processing module, used to obtain pending requests, obtain a token corresponding to the pending request from a token bucket for the first time, and increase the total number of concurrent requests in the current sliding time window by 1; A determination module, configured to determine the number of retries and the length of the retry interval for re-acquiring the token if the initial acquisition of the token fails, based on the total number of concurrent requests, and the throughput, average latency, and maximum acceptable latency of the large model; A retry module is used to re-acquire the token from the token bucket based on the number of retries and the length of the retry interval, and when the token is successfully re-acquired, the pending request is inferred based on the large model to generate and return an answer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images