Request processing method and apparatus, computer device, and readable storage medium

Abstract

The request processing method and device, computer device and readable storage medium provided by the embodiments of the present application consider that the bundling width of the sequence generation model deployed on the server has a great influence on the time consumption and quality of the sequence generation operation, and a bundling width adaptive to the number of requests is selected to achieve a relatively optimized processing effect. For a case where the number of real-time requests is greater than or equal to a preset number of requests, the preset number of requests in the request queue are combined and input into the sequence generation model for sequence generation operation, while for a case where the number of real-time requests is less than the preset number of requests, a temporary bundling width is determined according to the number of real-time requests, and all the requests to be processed are combined and input into the sequence generation model for sequence generation operation. The temporary bundling width when the number of real-time requests to be processed is small is greater than the preset bundling width, which guarantees the output quality and inference speed, realizes the elastic allocation of server computing power, and improves the server throughput.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a request processing method, apparatus, computer device, and readable storage medium. Background Technology

[0002] Sequence generation models have gradually moved from academia to industry in recent years, and are widely used in chatbots, automatic summarization, automatic poetry writing, and automatic music composition. Deep learning-based sequence generation models typically generate sequences using search algorithms such as beamsearch, top-k, and top-p. Their inference speed is determined by two important parameters: beam width and sequence length. Beam width represents the search width at which the model generates the next symbol. The value of beam width significantly affects both the time consumption and the quality of the generated sequence. When the beam width is 1, the search strategy degenerates into a greedy search, resulting in the shortest search time but also the worst average quality of the generated sequence. As the beam width increases, the average quality of the generated sequence improves, but the time consumption also gradually increases. When deploying sequence generation models on servers, choosing an appropriate search beam width and balancing throughput and the quality of the generated sequence is a crucial issue.

[0003] The sequence length represents the number of symbols contained in the generated sequence. Since generating each symbol requires one model inference, the time required to generate a complete sequence is linearly related to the sequence length. In real-world deployment scenarios, multiple users often send requests to the server, and each request requires generating a complete sequence. Assuming the sequence length is i and the search width is j, i model inferences are needed, processing j samples in parallel each time. It is evident that sequence generation models in real-world deployment scenarios have significantly higher computational requirements and time overhead compared to other tasks such as classification and regression.

[0004] Therefore, there is an urgent need to provide a processing solution that can improve server throughput while ensuring the quality of the generated sequence in actual deployment scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a request processing method, apparatus, computer device, and readable storage medium.

[0006] In a first aspect, embodiments of this application provide a request processing method, the method comprising:

[0007] Get the real-time number of requests waiting to be processed in the request queue;

[0008] If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width.

[0009] If the number of real-time requests is less than the preset number of requests, then a temporary bundle width is determined based on the number of real-time requests, and all requests to be processed are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the temporary bundle width.

[0010] Wherein, the preset bundle width is smaller than the temporary bundle width.

[0011] According to one specific embodiment of this application, determining the temporary bundle width based on the number of real-time requests includes:

[0012] The temporary bundle width is determined based on the number of real-time requests and the maximum bundle width of the sequence generation model.

[0013] According to one specific embodiment of this application, determining the temporary bundle width based on the number of real-time requests and the maximum bundle width of the sequence generation model includes:

[0014] The temporary bundle width is calculated using the formula m = Floor(K, n);

[0015] Where m represents the temporary bundle width, K represents the maximum bundle width of the sequence generation model, and n represents the actual number of requests of the sequence generation model.

[0016] According to one specific embodiment of this application, before obtaining the real-time number of requests to be processed in the request queue, the method further includes:

[0017] Create an empty request queue;

[0018] If a pending request for sequence generation is received, the pending request is added to the request queue in order of receipt time.

[0019] The step of merging a preset number of pending requests in the request queue into the input sequence generation model for sequence generation includes:

[0020] According to the order in which they were added to the request queue, the first preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation.

[0021] According to a specific embodiment of this application, the step of merging the pre-preset number of pending requests in the request queue into the sequence generation model for sequence generation further includes:

[0022] Newly received pending requests are added to the request queue in the order they were received.

[0023] If it is detected that the sequence generation model has completed the sequence generation operation, the process will continue to jump to obtain the real-time number of requests pending in the request queue.

[0024] According to one specific embodiment of this application, after obtaining the number of real-time requests to be processed in the request queue, the method further includes:

[0025] If the number of real-time requests is zero, the sequence generation operation is performed when the number of real-time requests to be processed in the request queue reaches the minimum merging number; wherein, 1 ≤ the minimum merging number is less than the preset number of requests.

[0026] According to one specific embodiment of this application, the merging of pending requests includes:

[0027] Each text-formatted request to be processed is segmented into a multi-dimensional term, and the multi-dimensional terms are then converted into a one-hot encoded format.

[0028] All the multidimensional terms corresponding to the requests to be processed are combined into a multidimensional tensor corresponding to the input format of the sequence generation model. The dimensions of the multidimensional tensor are [batch_size, max_sequence_length, dictionary_size], where batch_size represents the number of real-time requests, max_sequence_length represents the maximum sequence length of the sequence generation model, and dictionary_size represents the total number of terms in the dictionary.

[0029] Secondly, embodiments of this application provide a request processing apparatus, the apparatus comprising:

[0030] The acquisition module is used to obtain the real-time number of requests waiting to be processed in the request queue;

[0031] Processing module, used for,

[0032] If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width.

[0033] If the number of real-time requests is less than the preset number of requests, then a temporary bundle width is determined based on the number of real-time requests, and all requests to be processed are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the temporary bundle width.

[0034] Wherein, the preset bundle width is smaller than the temporary bundle width.

[0035] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the computer program executes the request processing method provided in the first aspect when the processor is running.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a processor, executes the request processing method provided in the first aspect.

[0037] The request processing method, apparatus, computer equipment, and readable storage medium provided in this application consider the significant impact of the bundle width of the sequence generation model deployed on the server on the time consumption and quality of sequence generation operations. A bundle width adapted to the number of requests is selected to achieve a more optimized processing effect. When the number of real-time requests is greater than or equal to a preset number of requests, the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operations. When the number of real-time requests is less than the preset number of requests, a temporary bundle width is determined based on the number of real-time requests, and all pending requests are merged and input into the sequence generation model for sequence generation operations. When the number of real-time requests to be processed is small, the temporary bundle width is greater than the preset bundle width, ensuring output quality and inference speed, achieving elastic allocation of server computing power, and improving server throughput. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation on the scope of protection of this application. In the various drawings, similar components are numbered similarly.

[0039] Figure 1 A flowchart illustrating a request processing method provided in an embodiment of this application is shown;

[0040] Figure 2 This illustration shows another flowchart of the request processing method provided in an embodiment of this application;

[0041] Figure 3 This paper shows a schematic diagram of the structure of a request processing apparatus provided in an embodiment of this application;

[0042] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0044] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0045] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0046] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0047] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0048] Example 1

[0049] See Figure 1 This is a flowchart illustrating a request processing method provided in an embodiment of this application. Figure 1 As shown, the request processing method mainly includes the following steps:

[0050] Step S101: Obtain the number of real-time requests pending in the request queue;

[0051] The request processing method provided in this embodiment is applied to scenarios where a sequence generation model processes user-input requests. The sequence generation model can be a deep learning model used in chatbots, automatic summarization, automatic poetry writing, and automatic music composition. The requests can be text translation, automatic chat, etc. Any input that can be used as the sequence generation model can be used as the request content. Requests may come from the same user or from different users. For multiple requests that need to be merged, requests that handle the same task and are processed by the same sequence generation model can be merged into a single input for the sequence generation model. Requests waiting for the sequence generation model to process are defined as pending requests. The solution provided in this embodiment merges pending requests for batch processing to optimize the server's throughput in loading the model.

[0052] In practice, the computer device can create a process independent of model computation to maintain a task queue, defined as a request queue, to receive and store pending requests waiting for the current sequence generation model. Received pending requests first enter the request queue, waiting for a certain number of pending requests to be merged and processed. The number of pending requests in the request queue is defined as the real-time request count. The computer device can obtain the real-time request count in the request queue in real time or periodically to choose whether to execute subsequent request merging operations.

[0053] Step S102: If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width.

[0054] Step S103: If the number of real-time requests is less than the preset number of requests, then the temporary bundle width is determined according to the number of real-time requests, and all requests to be processed are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the temporary bundle width.

[0055] Wherein, the preset bundle width is smaller than the temporary bundle width.

[0056] In real-world applications, the number of requests received within a given timeframe is typically variable and unpredictable, with a low number of requests per unit time and occasional peaks. To provide a universally applicable solution for long-term or routine situations, this study considers the number of requests that a sequence generation model can process in a merged manner and the impact of different request numbers on output quality and inference speed. A preset request number is defined to determine the number of requests that can be merged under critical conditions, limiting the number of requests to be processed in each merge to be less than or equal to this preset request number.

[0057] Before each triggering of a sequence generation request, the computer device obtains the real-time number of requests to be processed in the request queue, compares this real-time number with the preset number of requests, and performs different merging operations in two different cases.

[0058] In the first scenario, the number of real-time requests is greater than or equal to the preset number of requests, meaning the number of pending requests in the request queue has reached the maximum number of requests that can be merged. In this case, the preset number of pending requests can be extracted from the request queue, merged, and used as input to the sequence generation model for sequence generation. If the number of requests is a fixed preset number, then the bundle width of the sequence generation model in this case is also a preset bundle width; that is, the actual bundle width of the sequence generation model during the search is set to the preset bundle width.

[0059] In the second scenario, the number of real-time requests is less than the preset number of requests, meaning the current number of pending requests in the request queue has not reached the maximum number of requests that can be merged. In this case, all pending requests can be extracted from the request queue, merged, and used as input to the sequence model for sequence generation. Since different numbers of requests require different amounts of computing power, the appropriate bundle width for the current number of pending requests can be calculated and defined as the temporary bundle width. The actual bundle width used by the sequence generation model during the search is then set as the temporary bundle width.

[0060] Based on the constraint relationship between the bundle width and the single-round request sheet, the fewer the number of pending requests to be processed in the merged input, the larger the bundle width can be selected to prioritize ensuring the quality of the sequence generation model's output; conversely, the larger the number of pending requests to be processed in the merged input, the smaller the bundle width can be selected to prioritize ensuring the inference speed of the sequence generation model. Therefore, for the above two cases, the preset bundle width when merging a preset number of input requests is smaller than the temporary bundle width.

[0061] Based on the above embodiments, a specific scheme for obtaining the temporary cluster width is added. According to a specific embodiment of this application, determining the temporary cluster width based on the number of real-time requests includes:

[0062] The temporary bundle width is determined based on the number of real-time requests and the maximum bundle width of the sequence generation model.

[0063] Sequence generation models have a maximum bundle width based on their operational properties. When the actual number of real-time requests being merged does not reach the preset number of requests under critical conditions, a temporary bundle width can be determined based on the current number of real-time requests and the maximum bundle width of the sequence generation model to comprehensively ensure both output quality and inference speed.

[0064] Specifically, determining the temporary bundle width based on the number of real-time requests and the maximum bundle width of the sequence generation model may include:

[0065] The temporary bundle width is calculated using the formula m = Floor(K, n);

[0066] Where m represents the temporary bundle width, K represents the maximum bundle width of the sequence generation model, and n represents the actual number of requests of the sequence generation model.

[0067] When input merging is triggered, n pending requests extracted from the request queue are fed into the sequence generation model for sequence generation. During the search, the temporary bundle width m is set to K divided by n (rounded down), where K is the preset maximum bundle width batchsize, the value of which is determined by available computing resources, expected response time, and throughput. At this time, the actual batchsize of the sequence generation model is m*n, where m*n≤K.

[0068] The specific value of parameter K depends on the size of the graphics processor's video memory. The larger K is, the larger the tensor of the model input at one time, the more video memory is consumed, and the longer it takes to complete a single operation. However, more requests are processed per operation, so the model throughput is also greater. In practical use, it is necessary to first determine the maximum allowable response time and the maximum allowable video memory allocation, and then maximize K while satisfying these two constraints.

[0069] The request merging operation involved in the above steps typically involves merging requests according to the format requirements of the model input. According to a specific embodiment of this application, the merging of requests to be processed may specifically include:

[0070] Each text-formatted request to be processed is segmented into a multi-dimensional term, and the multi-dimensional terms are then converted into a one-hot encoded format.

[0071] All the multidimensional terms corresponding to the requests to be processed are combined into a multidimensional tensor corresponding to the input format of the sequence generation model. The dimensions of the multidimensional tensor are [batch_size, max_sequence_length, dictionary_size], where batch_size represents the number of real-time requests, max_sequence_length represents the maximum sequence length of the sequence generation model, and dictionary_size represents the total number of terms in the dictionary.

[0072] The sequence generation model is a deep learning model. Its input is a multidimensional tensor. A tensor can contain multiple data points, called a batch. The index of the first dimension of the tensor indicates which data point is in the current batch.

[0073] Here, data type conversion is required to transform the user input data type (such as text) into a data type suitable for model input (multidimensional tensor). The conversion method varies depending on the specific model type. In natural language processing, a common method is to first segment the text and then convert it into one-hot encoding. In this case, the dimensions of the input tensor are [batch_size, max_sequence_length, dictionary_size]. batch_size represents the number of requests in a batch, max_sequence_length represents the maximum sequence length accepted by the model. If the length exceeds this, the sequence is truncated; if the length is less than this, the sequence is padded with special symbols. dictionary_size represents the total number of words in the dictionary. This tensor can be imagined as a cube filled with 0s or 1s, where the value corresponding to the specific dimension index [i, jk] is 1, indicating that the word at the j-th position of the i-th data in the batch after segmentation is the k-th word in the dictionary; otherwise, the value is 0.

[0074] The graphics computing unit has a higher computing speed than the CPU when processing tensor operations. The specific principle involves hardware knowledge. Here, we combine multiple user requests into a single tensor for operation, which takes less time than converting each request into a tensor and inputting it serially into the model for operation.

[0075] Of course, considering that the input formats of different sequence generation models may be different, the request merging method will also be adjusted accordingly, which will not be elaborated here.

[0076] The request processing method provided in this application takes into account the significant impact of the bundle width of the sequence generation model deployed on the server on the time consumption and quality of the sequence generation operation. It selects a bundle width that is adapted to the number of requests to achieve a more optimized processing effect. When the number of real-time requests to be processed is small, the temporary bundle width is greater than the preset bundle width to ensure output quality and inference speed, realize the elastic allocation of server computing power, and improve server throughput.

[0077] like Figure 2 One embodiment shown adds related operations for triggering the process before and after request processing. In this specific embodiment, before obtaining the real-time number of requests to be processed in the request queue, the method may further include:

[0078] S201, Create an empty request queue;

[0079] S202, if a pending request for participating in the sequence generation operation is received, the pending requests are added to the request queue in order of their reception time;

[0080] The step of merging a preset number of pending requests in the request queue into the input sequence generation model for sequence generation includes:

[0081] S203, according to the order in which the requests were added to the request queue, the first preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation.

[0082] The computer device creates an empty request queue T. Requests received from users and awaiting processing are first added to this queue T. Subsequent requests are retrieved from the queue when a merging operation is triggered. To ensure efficient request processing, requests are added to and retrieved from the queue according to a first-in, first-out (FIFO) principle. Specifically, requests are added to the queue sequentially based on their reception time and retrieved sequentially based on the time they were added.

[0083] Regarding the first scenario mentioned above, if the number of real-time requests remaining in the request queue is greater than the preset number of requests, then the first preset number of requests to be processed are extracted and merged into the input according to the order in which they were added. If the number of real-time requests is exactly equal to or equal to the preset number of requests, then all requests to be processed can be extracted and merged into the input.

[0084] Furthermore, in specific implementations, the step of merging the pre-preset number of pending requests in the request queue into the input sequence generation model for sequence generation may further include:

[0085] Newly received pending requests are added to the request queue in the order they were received.

[0086] If it is detected that the sequence generation model has completed the sequence generation operation, the process will continue to jump to obtain the real-time number of requests pending in the request queue.

[0087] While performing the current request extraction, input merging, and sequence generation operations, the computer device will continuously receive new requests to be processed. At this time, the newly received requests to be processed can be added to the request queue to wait for the next request extraction, input merging, and sequence generation operation.

[0088] It should be noted that there are multiple triggering conditions for adjacent operations in request merging processing. For example, the triggering condition can be the completion of each sequence generation operation. That is, after detecting that the sequence generation model has completed the sequence generation operation, the system jumps to obtain the real-time number of requests to be processed in the request queue and then selects whether to perform the next sequence generation operation. Of course, time, period, etc. can also be used as triggering conditions to trigger the next sequence generation operation. In this embodiment, the completion status of the current sequence generation operation can be preferred as the triggering condition. In this way, new requests are extracted from the queue immediately after the previous round of model calculation is completed, while periodic or time-segmented extraction will cause unnecessary delays.

[0089] Of course, to address potential anomalies during the aforementioned request retrieval operation, such as an empty request queue, a waiting operation can be limited to avoid unnecessary processes. According to a specific embodiment of this application, after obtaining the real-time number of requests awaiting processing in the request queue, the method may further include:

[0090] If the number of real-time requests is zero, the sequence generation operation is performed when the number of real-time requests to be processed in the request queue reaches the minimum merging number; wherein, 1 ≤ the minimum merging number is less than the preset number of requests.

[0091] To avoid wasting computing power, when the request queue is empty, a minimum number of merges must be waited before executing the sequence generation operation. The minimum number of merges can be set to 1. This ensures that each user request receives a timely response, preventing previously received requests from waiting indefinitely when no new requests arrive, thus avoiding the abnormal situation where requests cannot be processed for an extended period.

[0092] The request processing scheme provided in this embodiment increases the search bundle width when the number of requests processed is less than N. A wider bundle width results in higher quality generated sequences. If a single request triggers the request merging operation, the model's search bundle width K reaches its maximum value, resulting in the highest quality generated sequences. When server traffic is low, such as when only one request is processed at a time, the search algorithm prioritizes the quality of the generated sequences. Only when traffic peaks and queues become congested, does the number of requests processed at a time increase to N, at which point the search bundle width decreases accordingly. The search algorithm sacrifices some quality to improve efficiency. This method of improving the throughput of the sequence generation model and achieving a flexible balance between the quality and speed of generated sequences facilitates the implementation of sequence generation models in industrial applications, promoting their transformation from academic research to practical application.

[0093] Example 2

[0094] See Figure 3This is a block diagram of a request processing device provided in an embodiment of this application. Figure 3 As shown, the request processing device 300 mainly includes:

[0095] The acquisition module 301 is used to acquire the real-time number of requests pending in the request queue;

[0096] Processing module 302 is used for,

[0097] If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width.

[0098] If the number of real-time requests is less than the preset number of requests, then a temporary bundle width is determined based on the number of real-time requests, and all requests to be processed are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the temporary bundle width.

[0099] Wherein, the preset bundle width is smaller than the temporary bundle width.

[0100] According to one specific embodiment of this application, the processing module 302 is specifically used for:

[0101] The temporary bundle width is determined based on the number of real-time requests and the maximum bundle width of the sequence generation model.

[0102] According to one specific embodiment of this application, the processing module 302 is specifically used for:

[0103] The temporary bundle width is calculated using the formula m = Floor(K, n);

[0104] Where m represents the temporary bundle width, K represents the maximum bundle width of the sequence generation model, and n represents the actual number of requests of the sequence generation model.

[0105] According to one specific embodiment of this application, the acquisition module 301 is specifically used for:

[0106] Create an empty request queue;

[0107] If a pending request for sequence generation is received, the pending request is added to the request queue in order of receipt time.

[0108] The processing module 302 is specifically used for:

[0109] According to the order in which they were added to the request queue, the first preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation.

[0110] According to one specific embodiment of this application, the acquisition module 301 is specifically used for:

[0111] Newly received pending requests are added to the request queue in the order they were received.

[0112] If it is detected that the sequence generation model has completed the sequence generation operation, the processing module 302 is specifically used to continue to jump to obtain the real-time number of requests to be processed in the request queue.

[0113] According to one specific embodiment of this application, the processing module 302 is specifically used for:

[0114] If the number of real-time requests is zero, the sequence generation operation is performed when the number of real-time requests to be processed in the request queue reaches the minimum merging number; wherein, 1 ≤ the minimum merging number is less than the preset number of requests.

[0115] According to one specific embodiment of this application, the processing module 302 is specifically used for:

[0116] Each text-formatted request to be processed is segmented into a multi-dimensional term, and the multi-dimensional terms are then converted into a one-hot encoded format.

[0117] All the multidimensional terms corresponding to the requests to be processed are combined into a multidimensional tensor corresponding to the input format of the sequence generation model. The dimensions of the multidimensional tensor are [batch_size, max_sequence_length, dictionary_size], where batch_size represents the number of real-time requests, max_sequence_length represents the maximum sequence length of the sequence generation model, and dictionary_size represents the total number of terms in the dictionary.

[0118] The request processing apparatus provided in this application considers the significant impact of the bundle width of the sequence generation model deployed on the server on the time consumption and quality of the sequence generation operation. It selects a bundle width adapted to the number of requests to achieve a more optimized processing effect. When the number of real-time requests is greater than or equal to a preset number of requests, the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation. When the number of real-time requests is less than the preset number of requests, a temporary bundle width is determined based on the number of real-time requests, and all pending requests are merged and input into the sequence generation model for sequence generation. When the number of real-time requests is small, the temporary bundle width is greater than the preset bundle width, ensuring output quality and inference speed, achieving elastic allocation of server computing power, and improving server throughput. The specific implementation process of the request processing apparatus provided in this embodiment can be found in the specific implementation process of the request processing method provided in the foregoing embodiments, and will not be repeated here.

[0119] Example 3

[0120] Furthermore, this application also provides a computer device, including a memory and a processor. The memory is used to store a computer program, and the computer program executes a request processing method when the processor runs. The executed request processing method can be the request processing method provided in the aforementioned embodiment 1.

[0121] Specifically, such as Figure 4 As shown, the computer device 400 provided in this embodiment includes:

[0122] The components include a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, a processor 410, and a power supply 411. Those skilled in the art will understand that... Figure 4 The computer device structures shown are not intended to limit the computer device. A computer device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In the embodiments of this application, the computer device includes, but is not limited to, mobile phones, tablets, laptops, handheld computers, in-vehicle computer devices, wearable devices, and pedometers.

[0123] It should be understood that, in this embodiment, the radio frequency unit 401 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 410; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 401 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 401 can also communicate with networks and other devices through a wireless communication system.

[0124] The computer device provides users with wireless broadband internet access through network module 402, such as helping users send and receive emails, browse web pages, and access streaming media.

[0125] The audio output unit 403 can convert audio data received by the radio frequency unit 401 or the network module 402 or stored in the memory 409 into audio signals and output them as sound. Furthermore, the audio output unit 403 can also provide audio output related to specific functions performed by the computer device 400 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 403 includes a speaker, a buzzer, and a receiver, etc.

[0126] Input unit 404 is used to receive audio or video signals. Input unit 404 may include a graphics processing unit (GPU) 4041 and a microphone 4042. The GPU 4041 processes image data of still images or videos acquired by an image capture computer device (such as a camera) in video capture mode or image capture mode. The processed image frames can be played back on display unit 406. The image frames processed by GPU 4041 can be stored in memory 409 (or other storage media) or transmitted via radio frequency unit 401 or network module 402. Microphone 4042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 401 in telephone call mode.

[0127] The computer device 400 also includes at least one sensor 405, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 4061 according to the ambient light level, and the proximity sensor can turn off the display panel 4061 and / or backlight when the computer device 400 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the computer device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 405 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0128] Display unit 406 is used for video playback of information input by the user or information provided to the user. Display unit 406 may include display panel 4061, which may be configured in the form of a liquid crystal display (LCD), organic light-emitting diode (OLED), or the like.

[0129] User input unit 407 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the computer device. Specifically, user input unit 407 includes a touch panel 4071 and other input devices 4072. Touch panel 4071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 4071). Touch panel 4071 may include two parts: a touch detection computer device and a touch controller. The touch detection computer device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection computer device, converts it into touch point coordinates, and sends it to the processor 410, which receives and executes commands from the processor 410. In addition, touch panel 4071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 4071, user input unit 407 may also include other input devices 4072. Specifically, other input devices 4072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0130] Furthermore, the touch panel 4071 can cover the display panel 4061. When the touch panel 4071 detects a touch operation on or near it, it transmits the information to the processor 410 to determine the type of touch event. Subsequently, the processor 410 provides corresponding visual output on the display panel 4061 based on the type of touch event. Although in Figure 4 In this embodiment, the touch panel 4071 and the display panel 4061 are two independent components to realize the input and output functions of the computer device. However, in some embodiments, the touch panel 4071 and the display panel 4061 can be integrated to realize the input and output functions of the computer device. The specific implementation is not limited here.

[0131] Interface unit 408 serves as an interface for connecting external computer devices to computer device 400. For example, the external computer device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting to a computer device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 408 can be used to receive input (e.g., data, power, etc.) from the external computer device and transmit the received input to one or more elements within computer device 400, or it can be used to transfer data between computer device 400 and the external computer device.

[0132] The memory 409 can be used to store software programs and various data. The memory 409 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 409 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0133] Processor 410 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines. By running or executing software programs and / or modules stored in memory 409, and by calling data stored in memory 409, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Processor 410 may include one or more processing units; preferably, processor 410 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 410.

[0134] The computer device 400 may also include a power supply 411 (such as a battery) for supplying power to various components. Preferably, the power supply 411 is logically connected to the processor 410 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0135] In addition, the computer device 400 includes some functional modules not shown, which will not be described in detail here.

[0136] Furthermore, embodiments of this application provide a computer-readable storage medium storing a computer program that executes a request processing method when running on a processor. The executed request processing method may be the request processing method provided in the aforementioned embodiment 1.

[0137] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0138] The computer device and readable storage medium provided in this application, considering the significant impact of the bundle width of the sequence generation model deployed on the server on the time consumption and quality of sequence generation operations, select a bundle width adapted to the number of requests to achieve a more optimized processing effect. When the number of real-time requests is greater than or equal to a preset number of requests, the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operations. When the number of real-time requests is less than the preset number of requests, a temporary bundle width is determined based on the number of real-time requests, and all pending requests are merged and input into the sequence generation model for sequence generation operations. When the number of real-time requests to be processed is small, the temporary bundle width is greater than the preset bundle width, ensuring output quality and inference speed, realizing elastic allocation of server computing power, and improving server throughput. The specific implementation process of the provided computer device and readable storage medium can be found in the specific implementation process of the request processing method provided in the foregoing embodiments, and will not be repeated here.

[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0141] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A request processing method, characterized in that, The method includes: Get the real-time number of requests waiting to be processed in the request queue; If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width. If the number of real-time requests is less than the preset number of requests, all pending requests will be merged and input into the sequence generation model for sequence generation. The actual bundle width of the sequence generation model is the temporary bundle width, which is equal to the maximum bundle width of the sequence generation model divided by the number of real-time requests. The maximum bundle width is determined based on computing resources, expected response time, and throughput. Wherein, the preset bundle width is smaller than the temporary bundle width.

2. The method according to claim 1, characterized in that, include: The temporary bundle width is calculated according to the formula m=Floor(K,n); Where m represents the temporary bundle width, K represents the maximum bundle width of the sequence generation model, and n represents the actual number of requests of the sequence generation model.

3. The method according to any one of claims 1 to 2, characterized in that, Before obtaining the real-time number of requests to be processed in the request queue, the method further includes: Create an empty request queue; If a pending request for sequence generation is received, the pending request is added to the request queue in order of receipt time. The step of merging a preset number of pending requests in the request queue into the input sequence generation model for sequence generation includes: According to the order in which they were added to the request queue, the first preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation.

4. The method according to claim 3, characterized in that, The step of merging the pre-preset number of pending requests in the request queue into the input sequence generation model for sequence generation also includes: Newly received pending requests are added to the request queue in the order they were received. If it is detected that the sequence generation model has completed the sequence generation operation, the process will continue to jump to obtain the real-time number of requests pending in the request queue.

5. The method according to claim 1, characterized in that, After obtaining the real-time number of requests to be processed in the request queue, the method further includes: If the number of real-time requests is zero, the sequence generation operation is performed when the number of real-time requests to be processed in the request queue reaches the minimum merging number; wherein, 1 ≤ the minimum merging number is less than the preset number of requests.

6. The method according to claim 1, characterized in that, The merging of pending requests includes: Each text-formatted request to be processed is segmented into a multi-dimensional term, and the multi-dimensional terms are then converted into a one-hot encoded format. All the multidimensional terms corresponding to the requests to be processed are combined into a multidimensional tensor corresponding to the input format of the sequence generation model. The dimensions of the multidimensional tensor are [batch_size, max_sequence_length, dictionary_size], where batch_size represents the number of real-time requests, max_sequence_length represents the maximum sequence length of the sequence generation model, and dictionary_size represents the total number of terms in the dictionary.

7. A request processing apparatus, characterized in that, The device includes: The acquisition module is used to obtain the real-time number of requests waiting to be processed in the request queue; Processing module, used for, If the number of real-time requests is greater than or equal to the preset number of requests, then the preset number of pending requests in the request queue are merged and input into the sequence generation model for sequence generation operation. The actual bundle width of the sequence generation model is the preset bundle width. If the number of real-time requests is less than the preset number of requests, all pending requests will be merged and input into the sequence generation model for sequence generation. The actual bundle width of the sequence generation model is the temporary bundle width, which is equal to the maximum bundle width of the sequence generation model divided by the number of real-time requests. The maximum bundle width is determined based on computing resources, expected response time, and throughput. Wherein, the preset bundle width is smaller than the temporary bundle width.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that executes the request processing method according to any one of claims 1 to 6 when the processor is running.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when run on a processor, executes the request processing method according to any one of claims 1 to 6.

Images