Service request scheduling method, equipment and device based on time awareness and medium

Abstract

The invention discloses a service request scheduling method, equipment and device based on time awareness and a medium, and relates to the technical field of data transmission. And through the pre-training model, the pre-estimated completion time of each service request is obtained through prediction, so that the prediction accuracy of the service request in the request scheduling completion time is improved. And calculating the remaining time of each service request in the corresponding reasoning stage through the estimated completion time, and if the sum of the remaining time of each service request is greater than or equal to 0, indicating that the remaining time corresponding to each service request in the scheduling process is sufficient and is sufficient to complete scheduling in the current round. And if the time is less than 0, indicating that the remaining time of each service request of the current round is insufficient and all scheduling cannot be completed, deferring the target service request to the next round of scheduling, and recalculating the remaining time of the remaining service requests until the scheduling of the service requests is completed. And the final remaining service requests in the current round are ensured to be scheduled, and the overtime phenomenon is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a time-aware service request scheduling method, equipment, device and medium. Background Art

[0002] The pre-filling stage and the decoding stage are two key stages in the reasoning process of the pre-trained model. In the corresponding parallel processing process when different requests arrive at different times, it is necessary to ensure that the delay requirements of the decoding stage of the original request are met, and also to ensure that the delay requirements of the pre-filling stage of the new request are met.

[0003] In the conventional processing corresponding to multiple business requests, if the corresponding timeout period is strictly required, there will be a certain timeout phenomenon in the scheduling and processing process of multiple business requests, resulting in reduced reliability and efficiency of system processing.

[0004] Therefore, how to avoid timeout in the scheduling process of multiple service requests to improve the reliability and efficiency of system processing is an urgent problem that those skilled in the art need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a time-aware service request scheduling method, equipment, device and medium to solve the problem that the timeout phenomenon in the scheduling process of multiple service requests leads to reduced reliability and efficiency of system processing.

[0006] In order to solve the above technical problems, the present invention provides a time-aware service request scheduling method, comprising:

[0007] Obtain the pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output the estimated completion time corresponding to each business request;

[0008] Determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0009] When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that each service request has completed scheduling in the current round;

[0010] When the sum of the remaining times of each business request is less than 0, the remaining times of each business request are sorted to determine the target business request, and the target business request is postponed to the next round of scheduling; the maximum estimated completion time is determined among the remaining business requests except the target business request, and the new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and the process returns to the step of comparing the sum of the remaining times of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0011] On the one hand, the process of determining the pre-training model includes:

[0012] Get the initial pre-trained model;

[0013] Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request;

[0014] Inputting the request task length, hardware board parameters and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling phase and decoding phase processing on each historical business request to obtain the corresponding first completion time;

[0015] If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model;

[0016] If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements to obtain the pre-trained model.

[0017] On the other hand, the remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time of each business request, including:

[0018] Get the current inference stage of each business request in the pre-trained model;

[0019] If the current reasoning stage of the business request is the pre-filling stage, the arrival time and current time of the business request are obtained; the remaining pre-filling time of the pre-filling stage of the business request is determined according to the arrival time of the business request, the pre-filling timeout threshold time, the current time and the first estimated completion time;

[0020] If the current reasoning stage of the business request is the decoding stage, the arrival time of the business request, the current time and the actual request business length of the business request are obtained; the remaining decoding time of the decoding stage of the business request is determined according to the arrival time of the business request, the pre-filled timeout threshold time, the actual request business length, the decoding timeout threshold time, the current time and the first estimated completion time.

[0021] On the other hand, when the service request is a first service request, determining the remaining pre-filling time of the pre-filling phase of the service request according to the arrival time of the first service request, the pre-filling timeout threshold time, the current time, and the first estimated completion time includes:

[0022] Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time;

[0023] The current time of the first service request and the first estimated completion time of the first service request are added together to obtain a second time;

[0024] Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request;

[0025] Correspondingly, when the service request is a second service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time and the first estimated completion time, including:

[0026] Adding the arrival time of the second service request and the pre-filling timeout threshold time to obtain a third time;

[0027] Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request;

[0028] The fourth time is obtained by multiplying the actual requested service length of the second service request by the decoding timeout threshold time;

[0029] Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time;

[0030] The third time and the fourth time are added to obtain a sixth time;

[0031] The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

[0032] On the other hand, the remaining time of each service request is sorted to determine the target service request, including:

[0033] Sort the service requests from large to small according to the remaining time of each service request to obtain sorted service requests;

[0034] Get the arrival time of each business request;

[0035] The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

[0036] On the other hand, the new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time corresponding to the remaining business requests except the target business request and the maximum estimated completion time, including:

[0037] Obtaining a maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request;

[0038] Get the current inference stage of each other business request in the pre-trained model;

[0039] If the current reasoning stage of the remaining business requests is the pre-filling stage, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling stage of the remaining business requests is determined according to the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time and the maximum preset completion time;

[0040] If the current inference stage of the remaining business requests is the decoding stage, then obtain the arrival time, current time and actual request business length of the remaining business requests; determine the remaining decoding time of the decoding stage of the business request based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

[0041] On the other hand, when the remaining service request is a third service request, determining the remaining pre-filling time of the pre-filling phase of the remaining service requests according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0042] The arrival time of the third service request and the pre-filling timeout threshold time are added together to obtain a seventh time;

[0043] The eighth time is obtained by adding the current time of the third business request, the first estimated completion time and the maximum estimated completion time of the third business request;

[0044] Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request;

[0045] Correspondingly, when the remaining service request is a fourth service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time, including:

[0046] The arrival time of the fourth service request and the pre-filling timeout threshold time are added together to obtain a ninth time;

[0047] Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request;

[0048] The actual requested service length of the fourth service request is multiplied by the decoding timeout threshold to obtain a tenth time;

[0049] The current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time are added together to obtain an eleventh time;

[0050] The ninth time and the tenth time are added together to obtain the twelfth time;

[0051] The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

[0052] In order to solve the above technical problems, the present invention further provides a time-aware service request scheduling device, comprising:

[0053] An acquisition module is used to acquire a pre-trained model and a request task length of each business request; and input the request task length of each business request into the pre-trained model to output an estimated completion time corresponding to each business request;

[0054] The first determination module is used to determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0055] The second determination module is used to determine that all service requests have completed scheduling in the current round when the sum of the remaining time of each service request is greater than or equal to 0;

[0056] The third determination module is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the reasoning stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0057] In order to solve the above technical problems, the present invention further provides a time-aware service request scheduling device, comprising:

[0058] Memory for storing computer programs;

[0059] A processor is used to implement the steps of the time-aware service request scheduling method as described above when executing the computer program.

[0060] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the time-aware service request scheduling method as described above are implemented.

[0061] The present invention provides a time-aware business request scheduling method, comprising: obtaining a pre-training model and a request task length of each business request; inputting the request task length of each business request into the pre-training model to output an estimated completion time corresponding to each business request; determining the remaining time of the reasoning phase corresponding to the pre-training model according to the estimated completion time of each business request; comparing the sum of the remaining time of each business request with 0; when the sum of the remaining time of each business request is greater than or equal to 0, determining that each business request has completed scheduling in the current round; when the sum of the remaining time of each business request is less than 0, sorting the remaining time of each business request to determine a target business request, and postponing the target business request to the next round of scheduling; determining the maximum estimated completion time among the remaining business requests except the target business request; and determining the new remaining time of the reasoning phase corresponding to the pre-training model according to the estimated completion time corresponding to the remaining business requests except the target business request and the maximum estimated completion time, and returning to the step of comparing the sum of the remaining time of each business request with 0 until each business request has completed scheduling; wherein the urgency of the target business request is less than the urgency of the remaining business requests.

[0062] The beneficial effect of the present invention is to obtain the estimated completion time of each business request through the prediction of the pre-trained model, so as to improve the prediction accuracy of the completion time of the business request in the request scheduling. The remaining time of each business request in the corresponding reasoning stage is calculated by the estimated completion time, and the sum of the remaining time of each business request is compared with 0. If it is greater than or equal to 0, it means that the remaining time corresponding to each business request in the scheduling process is sufficient to complete the scheduling in the current round. If it is less than 0, it means that the remaining time of each business request in the current round is insufficient and all scheduling cannot be completed. It is necessary to select the target business request with less urgency and postpone it to the next round of scheduling, and recalculate the remaining time of the remaining business requests until the scheduling of the business request is completed. Through the remaining time calculation method corresponding to the timeout indicator and the timeout strategy of the remaining time, it is guaranteed that the remaining business requests in the current round are scheduled, avoiding the occurrence of timeouts, and improving the reliability and efficiency of system processing.

[0063] Secondly, training iterations are performed on the basis of the original initial pre-training model, so that it is combined with the scenario of the business request in this embodiment, suitable for the prediction of the estimated completion time in the business scenario, and the accuracy of the prediction is improved. For the pre-filling remaining time and decoding remaining time determined in the corresponding reasoning stage under different business requests, the corresponding estimated completion time and the current arrival time corresponding to each business request are used as the basis, and the corresponding timeout threshold time of the decoding and pre-filling stages are added, so that the determined remaining time is more accurate. The target business request is determined by the order of the arrival time of the business request and the order of the remaining time of the business request, so that the determination process is simple and clear. For the new pre-filling remaining time and decoding remaining time determined in the corresponding reasoning stage under different business requests, the corresponding estimated completion time, the current arrival time corresponding to each business request and the maximum preset completion time are used as the basis, and the corresponding timeout threshold time of the decoding and pre-filling stages are added to ensure that the scheduling is realized within the maximum preset completion time, so that the determined remaining time is more accurate.

[0064] In addition, the present invention also provides a time-aware service request scheduling device, apparatus and medium, which have the same beneficial effects as the above-mentioned time-aware service request scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1Scheduling timing diagrams for regular multiple requests;

[0067] Figure 2 A flowchart of a time-aware service request scheduling method provided by an embodiment of the present invention;

[0068] Figure 3 A structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention;

[0069] Figure 4 A structural diagram of a time-aware service request scheduling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The core of the present invention is to provide a time-aware service request scheduling method, equipment, device and medium to solve the problem that the timeout phenomenon in the scheduling process of multiple service requests leads to reduced reliability and efficiency of system processing.

[0072] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0073] When the system processes the scheduling of multiple business requests at the same time, it is necessary to ensure that the non-first word delay of the request meets the preset requirements, and it is also necessary to ensure that the first word delay of the new request meets the preset requirements, so that the scheduling process of multiple business requests in the current round avoids timeout delays. The reasoning of the pre-trained model is decomposed into two independent stages in the reasoning process, the pre-filling stage and the decoding stage, to improve the operation efficiency and scalability of the pre-trained model. In the pre-filling stage, the system pre-processes the input data and performs operations such as data loading, preprocessing, and feature extraction. The purpose is to prepare all necessary information so that the model can be quickly reasoned in the decoding stage. The decoding stage is the actual model reasoning stage, which uses the data prepared in the pre-filling stage for calculation and generates output results. For example, the first word "today" of the question-answering question "What day is today" is the first word of the output result, which is the current pre-filling stage; the second word "week" of the output result is a non-first word, which is the current decoding stage. The delays corresponding to the two stages are the first word delay (Time To First Token, TTFT) and the non-first word delay (Time Per Output Token, TPOT).

[0074] Figure 1 Schedule a sequence diagram for a regular multiple request, such as Figure 1 As shown, the slash in the box represents the decoding inference stage, and the horizontal and vertical lines in the boxes represent the pre-filling stage. Requests 1-3 are running requests, and requests 4 and 5 are requests that arrive at the 51st ms. The inference scheduling needs to run the pre-filling stage first, and then the decoding stage. After the pre-filling stage is completed, the delay corresponding to the first word unit output by the language model (the first word unit delay) can be obtained, and the subsequent decoding stage outputs the delays corresponding to the subsequent words one by one (the non-first word unit delay). It is necessary to ensure that the non-first word delay of the request meets the preset requirements, and it is also necessary to ensure that the first word unit delay of the new request meets the preset requirements. However, corresponding to the business scenario with strict time, the adjustability of the corresponding multiple request scheduling is poor, resulting in serious timeouts and reduced system reliability. The time-aware business request scheduling method provided by the present invention can solve the above technical problems.

[0075] Figure 2 A flowchart of a time-aware service request scheduling method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0076] S11: obtaining a pre-trained model and a request task length of each business request; and inputting the request task length of each business request into the pre-trained model to output an estimated completion time corresponding to each business request;

[0077] S12: determining the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; comparing the sum of the remaining time of each business request with 0;

[0078] S13: When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that each service request has completed scheduling in the current round;

[0079] S14: When the sum of the remaining time of each business request is less than 0, sort the remaining time of each business request to determine the target business request, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request; and determine the new remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to step S12 of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0080] Specifically, time perception is to ensure that data is scheduled and transmitted between corresponding business requests within a certain and predictable time range, providing reliable request scheduling and transmission guarantees for time-sensitive business requests. The pre-trained model is a machine learning model with large-scale parameters and complex computing structure. It learns common features such as language and vision by pre-training on a large amount of data, so as to handle various complex tasks. The request task length of each business request usually refers to the time required to process a business request, including the entire process from receiving the request to sending the response. In different application scenarios, the optimization of the request task length can improve the overall performance of the system and user experience. The request task length of each business request is an important indicator to measure the efficiency of business request processing. Through various technical means and best practices, the request task length can be effectively shortened and the system performance can be improved.

[0081] Inputting the request task length of the business request into the pre-training model can obtain the estimated completion time corresponding to each business request. In other words, the completion time of each business request can be predicted by the pre-training model, so as to determine the remaining time of step S12.

[0082] How to predict in step S11 is not limited here, and the specific model of the pre-trained model used is also not limited. It can be a conventional pre-trained model or a pre-trained model for the current business request and the corresponding hardware board parameter setting, which can be set according to the actual situation. This embodiment uses a pre-trained model to predict the completion time of the business request, and improves the accuracy of the prediction through the calculation of the model.

[0083] In step S12, the remaining time of the reasoning stage is determined according to the estimated completion time of each business request. It can be understood that the reasoning stage corresponding to each business request is a pre-filling stage or a decoding stage, and the specific parameters of the remaining time of the reasoning stage are determined by the estimated completion time of each business request. A preset time corresponding to each business request can be provided here, and the corresponding remaining time is obtained based on the preset time minus the estimated completion time. The preset time here can be the same time corresponding to each business request, or different time parameters can be set for each business request, which is not limited here. In addition, the remaining time can also be determined in combination with the actual arrival time of each business request, the pre-filled timeout threshold, etc., and the remaining time can also be calculated by a specific algorithm, which is not limited here and can be set according to actual conditions.

[0084] The sum of the remaining time of each service request in steps S13 and S14 is to regard each service request as a set, and comprehensively consider the remaining time corresponding to each service request in the set. If the sum of the remaining time is greater than or equal to 0, it means that there is surplus remaining time in the current set, that is, there will be remaining time after all service requests are scheduled, or the scheduling time is just used up after the scheduling is completed. If the sum of the remaining time is less than 0, it means that there is a serious timeout in the current set. The serious timeout here is based on all service requests. Non-urgent service requests in the current set can be scheduled in the next round to relieve the remaining time for scheduling the remaining service requests in the current set of the current round. It should be noted that the sum of the remaining time may include service requests with a remaining time less than 0, a remaining time equal to 0, or a remaining time greater than 0, as long as the sum of the remaining time is calculated and determined.

[0085] In step S14, when the sum of the remaining time is less than 0, it is necessary to consider selecting the target business request from the remaining time of each business request, and determine the target business request by time sorting, so as to postpone the target business request to the next round of scheduling. In addition, corresponding to the sorting operation, it is helpful to identify that the number of target business requests here is not limited, it can be one or more, and the remaining time corresponding to the target business request can be the maximum remaining time, or it can be the remaining time corresponding to the middle among multiple business requests. Other factors can also be considered here, such as the arrival time of the business request. If it arrives the latest and the remaining time is in the middle, it can be delayed to the next round to complete the scheduling without timeout, and this is used as the target business request. The urgency of the target business request is less than that of the other business requests. It should be noted that the urgency here can be determined by the remaining time, or it can be determined based on multiple factors such as the arrival time and remaining time of the target business request, which are not limited here.

[0086] After excluding the target business request, the remaining business requests are taken as a new set, and the corresponding remaining time is recalculated in the new set. Here, the remaining time of the reasoning stage is mainly updated, and the maximum remaining time corresponding to all the remaining business requests is also fully considered. Therefore, the maximum estimated completion time is required here, and the new remaining time is determined based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests. In the embodiment of determining the remaining time in step S12 above, the maximum estimated completion time is added to determine the updated new remaining time.

[0087] Based on the new remaining time, the sum of the remaining time in step S12 is compared with 0 to make the judgment before scheduling again until all service requests are scheduled.

[0088] The embodiment of the present invention provides a time-aware business request scheduling method, comprising: obtaining a pre-trained model and a request task length of each business request; inputting the request task length of each business request into the pre-trained model to output an estimated completion time corresponding to each business request; determining the remaining time of the reasoning phase corresponding to the pre-trained model according to the estimated completion time of each business request; comparing the sum of the remaining time of each business request with 0; when the sum of the remaining time of each business request is greater than or equal to 0, determining that each business request has completed scheduling in the current round; when the sum of the remaining time of each business request is less than 0, sorting the remaining time of each business request to determine a target business request, and postponing the target business request to the next round of scheduling; determining the maximum estimated completion time among the remaining business requests except the target business request; and determining a new remaining time of the reasoning phase corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and returning to the step of comparing the sum of the remaining time of each business request with 0 until each business request has completed scheduling; wherein the urgency of the target business request is less than the urgency of the remaining business requests. The estimated completion time of each business request is predicted by the pre-trained model to improve the prediction accuracy of the completion time of the business request in the request scheduling. The remaining time of each business request in the corresponding reasoning stage is calculated by the estimated completion time, and the sum of the remaining time of each business request is compared with 0. If it is greater than or equal to 0, it means that the remaining time corresponding to each business request in the scheduling process is sufficient to complete the scheduling in the current round. If it is less than 0, it means that the remaining time of each business request in the current round is insufficient and all scheduling cannot be completed. It is necessary to select the target business request with less urgency and postpone it to the next round of scheduling, and recalculate the remaining time of the remaining business requests until the scheduling of the business request is completed. Through the remaining time calculation method corresponding to the timeout indicator and the timeout strategy of the remaining time, it is ensured that the remaining business requests in the current round are scheduled, avoiding the occurrence of timeouts, and improving the reliability and efficiency of system processing.

[0089] In some embodiments, the process of determining the pre-trained model includes:

[0090] Get the initial pre-trained model;

[0091] Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request;

[0092] Inputting the request task length, hardware board parameters and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling phase and decoding phase processing on each historical business request to obtain the corresponding first completion time;

[0093] If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model;

[0094] If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements to obtain the pre-trained model.

[0095] It can be understood that the pre-trained model in this embodiment is based on the initial pre-trained model setting. There is no limitation on the specific model of the initial pre-trained model. It can be a neural network model in deep learning or other models. It is mainly used to process and understand complex systems of natural language.

[0096] The input parameters of the pre-trained model include the request task length corresponding to each historical business request, the hardware board parameters and the pre-trained model parameters themselves. Based on these three parameters, in the initial pre-trained model, the hardware board is used to process each historical business request in the pre-filling stage and the decoding stage to obtain the corresponding first completion time. Hardware board parameters usually refer to the specifications and performance indicators of the expansion card installed on the computer motherboard, mainly including hardware type, interface type, bus type, memory size, output interface and other parameters. The pre-trained model parameters are the configuration and scale of the model in the pre-training stage, including but not limited to the number of layers, number of hidden units, number of attention heads, total model parameters, etc.

[0097] If the current corresponding first completion time meets the preset requirements, the initial pre-trained model is used as the final pre-trained model. If it does not meet the preset requirements, the training continues until the output first completion time meets the preset requirements, and the pre-trained model is determined. The preset requirements here can be that the first completion time does not exceed a certain threshold, or can be based on the number of iterations, etc., which are not limited here and can be set according to actual conditions.

[0098] The present embodiment provides training iterations based on the original initial pre-training model, so that it is combined with the business request scenario of the present embodiment, is suitable for predicting the estimated completion time in the business scenario, and improves the accuracy of the prediction.

[0099] In some other embodiments, the process of determining the pre-trained model includes:

[0100] Get the request task length corresponding to each historical business request;

[0101] Using a hardware board to process each historical service request in a pre-filling phase and a decoding phase, respectively, to obtain a corresponding first completion time;

[0102] The first completion time of the middle collection point between each two adjacent collection points is obtained by averaging the first completion time determined by each two adjacent collection points according to the request task length corresponding to each historical business request;

[0103] The pre-training model is obtained by performing linear fitting processing on all the first completion times and the collection points of the request task length corresponding to each historical business request.

[0104] In this embodiment, the pre-training model takes into account the linear fitting process of the data, and processes the request task lengths corresponding to each historical business request in the pre-filling stage and the decoding stage to obtain the respective first completion times, forming a mapping relationship between the two parameters of request task length and time. In order to expand the collection points, it is necessary to average the first completion times of two adjacent collection points to obtain the first completion time of the middle collection point between every two adjacent collection points, and perform linear fitting processing based on each collection point and the corresponding first completion time to determine the functional relationship as the pre-training model.

[0105] The pre-training model provided in this embodiment is determined based on two parameters, namely, the requested task length and the first completion time, and is obtained through a linear fitting relationship, thereby simplifying the determination process of the pre-training model and improving data processing efficiency.

[0106] In some embodiments, when the service request is a first service request, determining the remaining pre-filling time of the pre-filling phase of the service request according to the arrival time of the first service request, the pre-filling timeout threshold time, the current time, and the first estimated completion time includes:

[0107] Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time;

[0108] The current time of the first service request and the first estimated completion time of the first service request are added together to obtain a second time;

[0109] Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request;

[0110] Correspondingly, when the service request is a second service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time and the first estimated completion time, including:

[0111] Adding the arrival time of the second service request and the pre-filling timeout threshold time to obtain a third time;

[0112] Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request;

[0113] The fourth time is obtained by multiplying the actual requested service length of the second service request by the decoding timeout threshold time;

[0114] Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time;

[0115] The third time and the fourth time are added to obtain a sixth time;

[0116] The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

[0117] Specifically, the remaining time for each reasoning stage is as follows:

[0118] 1. Pre-filling time in the pre-filling phase:

[0119] T1=t1+t2-(t3+t4);

[0120] Among them, t1 is the arrival time of the first business request, t2 is the pre-filling timeout threshold time; t1+t2 is the first time; t3 is the current time of the first business request, t4 is the first estimated completion time of the first business request; t3+t4 is the second time; T1 is the remaining pre-filling time of the first business request.

[0121] 2. Remaining decoding time in the decoding phase:

[0122] T2=(t11+t2)+(L1-1)*t22-(t13+t14);

[0123] Among them, t11 is the arrival time of the second business request, t2 is the pre-filling timeout threshold time; t11+t2 is the third time; L1 is the requested service length of the second business request, L1-1 is the actual requested service length of the second business request; t22 is the decoding timeout threshold time, (L1-1)*t22 is the fourth time; t13 is the current time of the second business request, t14 is the first estimated completion time of the second business request; t13+t14 is the fifth time, (t11+t2)+(L1-1)*t22 is the sixth time, and T2 is the remaining decoding time of the second business request.

[0124] It should be noted that L1-1 in this embodiment takes into account that 1 is the length unit of a token, and the length unit of the first token is in the pre-filling stage and needs to be deleted during the calculation and decoding stage.

[0125] This embodiment provides the pre-filling remaining time and decoding remaining time determined in the corresponding inference stage under different business requests, based on the corresponding estimated completion time and the current arrival time corresponding to each business request, and at the same time adds the timeout threshold time corresponding to the decoding and pre-filling stages, so that the determined remaining time is more accurate.

[0126] In some embodiments, sorting the remaining time of each service request to determine the target service request includes:

[0127] Sort the service requests from large to small according to the remaining time of each service request to obtain sorted service requests;

[0128] Get the arrival time of each business request;

[0129] The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

[0130] Specifically, for the target service request, there is no limit on the specific number, which can be one or more. The order of arrival time of each service request is referenced to select whether the latest service request is within the remaining time of the first N service requests, where N is a positive integer. If so, it will be set as the target service request. The longer the remaining time, the more time there is for the service request to be postponed to the next round for processing.

[0131] The target service request provided in this embodiment is determined by sorting the order of arrival time of the service requests and the order of remaining time of the service requests, so that the determination process is simple and clear.

[0132] In some embodiments, determining a new remaining time of the reasoning phase corresponding to the pre-trained model according to the estimated completion time corresponding to the remaining business requests except the target business request and the maximum estimated completion time includes:

[0133] Obtain the maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request;

[0134] Get the current inference stage of each other business request in the pre-trained model;

[0135] If the current reasoning stage of the remaining business requests is the pre-filling stage, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling stage of the remaining business requests is determined according to the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time and the maximum preset completion time;

[0136] If the current inference stage of the remaining business requests is the decoding stage, then obtain the arrival time, current time and actual request business length of the remaining business requests; determine the remaining decoding time of the decoding stage of the business request based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

[0137] In combination with the above embodiments, the new remaining time corresponding to each reasoning stage under the remaining business requests takes into account the corresponding maximum preset completion time under the current set to ensure that the scheduling problem of the remaining business requests is completed within the current maximum preset completion time.

[0138] In some embodiments, when the remaining service requests are third service requests, determining the remaining pre-filling time of the pre-filling phase of the remaining service requests according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0139] The arrival time of the third service request and the pre-filling timeout threshold time are added together to obtain a seventh time;

[0140] The eighth time is obtained by adding the current time of the third business request, the first estimated completion time and the maximum estimated completion time of the third business request;

[0141] Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request;

[0142] Correspondingly, when the remaining service request is a fourth service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time, including:

[0143] The arrival time of the fourth service request and the pre-filling timeout threshold time are added together to obtain a ninth time;

[0144] Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request;

[0145] The actual requested service length of the fourth service request is multiplied by the decoding timeout threshold to obtain a tenth time;

[0146] The current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time are added together to obtain an eleventh time;

[0147] The ninth time and the tenth time are added together to obtain the twelfth time;

[0148] The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

[0149] In combination with the above embodiment, the corresponding new pre-filling remaining time is:

[0150] T3= t1'+t2-(t3'+t4'+t5');

[0151] Among them, t1' is the arrival time of the third business request, t2 is the pre-filling timeout threshold time; t1'+t2 is the seventh time; t3' is the current time of the third business request, t4' is the first estimated completion time of the third business request; t5' is the maximum estimated completion time; t3'+t4'+ t5' is the eighth time; T3 is the pre-filling remaining time of the third business request.

[0152] The corresponding new decoding remaining time is:

[0153] T4=(t11'+t2)+(L1'-1)*t22-(t13'+t14'+ t5');

[0154] Among them, t11' is the arrival time of the fourth business request, t2 is the pre-filling timeout threshold time; t11'+t2 is the ninth time; L1' is the request service length of the fourth business request, L1'-1 is the actual request service length of the fourth business request; t22 is the decoding timeout threshold time, (L1'-1)*t22 is the tenth time; t13' is the current time of the fourth business request, t14' is the first estimated completion time of the fourth business request; t5' is the maximum estimated completion time; t13'+t14'+t5' is the eleventh time, (t11'+t2)+(L1'-1)*t22 is the twelfth time, and T4 is the remaining decoding time of the fourth business request.

[0155] This embodiment provides new pre-filling remaining time and decoding remaining time determined in the corresponding inference stage under different business requests. It is based on the corresponding estimated completion time, the current arrival time corresponding to each business request and the maximum preset completion time, and at the same time adds the timeout threshold time corresponding to the decoding and pre-filling stages to ensure scheduling within the maximum preset completion time, so that the determined remaining time is more accurate.

[0156] In addition, the deployment scheduling scheme of the present invention is as follows:

[0157] Set a scheduling request set S;

[0158] The present invention first sets the current scheduling request set S, where S includes existing requests and newly added requests. Each request in S will be evaluated by a specific calculation method in subsequent steps to determine whether it should be scheduled in the current round.

[0159] Calculate the time parameters of the request;

[0160] For each request s in the set S, the required computing time is calculated by a specific computing function Q, which is recorded as T. At the same time, the remaining time P of the request s is calculated. The calculation process includes:

[0161] Prefill remaining time: The prefill remaining time is equal to the request arrival time plus the prefill timeout threshold, minus the sum of the current time and the calculation time.

[0162] Decoding remaining time: The decoding remaining time is equal to the request arrival time plus the pre-fill timeout threshold, plus (output length minus 1) multiplied by the decoding timeout threshold, minus the sum of the current time and the calculation time.

[0163] Determine the cumulative sum of remaining time;

[0164] After calculating the remaining time of all requests, if the cumulative sum of all remaining time is greater than 0, it means that all requests in the current round can be scheduled. Otherwise, go to the next step to further process the request set S.

[0165] Request sorting;

[0166] Sort the requests in set S according to their remaining timeouts to obtain a sorted request set D. This sorting helps identify more urgent requests for priority processing.

[0167] Pop up the request and update the remaining time;

[0168] Pop a request from the sorted set D, that is, the request will be postponed to the next round in this round of scheduling. After the request is popped, its remaining time is recalculated, including:

[0169] Use function Q to calculate the computation time of the request in set D, denoted as U.

[0170] Update the remaining prefill time to the requested arrival time plus the prefill timeout threshold, minus the sum of the current time, the calculation time, and the maximum value in U.

[0171] Update the decoding remaining time to the requested arrival time plus the prefill timeout threshold plus (output length minus 1) multiplied by the decoding timeout threshold, minus the sum of the current time, the computation time, and the maximum value in U.

[0172] Re-evaluate the remaining cumulative time;

[0173] In the updated set D, the cumulative sum of the remaining time is recalculated. If the cumulative sum is greater than 0, all requests in set D are scheduled; otherwise, requests in D are continuously popped and the remaining time is updated until the condition is met or the set D is empty.

[0174] The ultimate scheduling solution;

[0175] If the D set is empty, the scheduling scheme with the largest remaining time is used to ensure that the system can process requests as efficiently as possible within the time constraints.

[0176] The solution involved in the present invention is applicable to both graphics processing unit (GPU) scenarios and non-GPU scenarios, and is not limited here.

[0177] The above describes in detail various embodiments corresponding to the time-aware service request scheduling method. On this basis, the present invention also discloses a time-aware service request scheduling device corresponding to the above method. Figure 3 The structure diagram of a time-aware service request scheduling device provided by an embodiment of the present invention. Figure 3 As shown, the time-aware service request scheduling device includes:

[0178] The acquisition module 11 is used to acquire the pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output the estimated completion time corresponding to each business request;

[0179] The first determination module 12 is used to determine the remaining time of the reasoning phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0180] The second determination module 13 is used to determine that all service requests have completed scheduling in the current round when the sum of the remaining time of each service request is greater than or equal to 0;

[0181] The third determination module 14 is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the inference stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until each business request is scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0182] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0183] For an introduction to a time-aware service request scheduling device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above time-aware service request scheduling method.

[0184] Figure 4 A structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention, such as Figure 4 As shown, the device comprises:

[0185] A memory 21, used for storing computer programs;

[0186] The processor 22 is used to implement the steps of the time-aware service request scheduling method when executing a computer program.

[0187] The time-aware service request scheduling device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0188] Among them, the processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0189] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the time-aware service request scheduling method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, data involved in the time-aware service request scheduling method, etc.

[0190] In some embodiments, the time-aware service request scheduling device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0191] Those skilled in the art can understand that Figure 4 The structure shown in the figure does not constitute a limitation on the time-aware service request scheduling device, and may include more or fewer components than those shown in the figure.

[0192] The processor 22 implements the time-aware service request scheduling method provided by any of the above embodiments by calling the instructions stored in the memory 21 .

[0193] For an introduction to a time-aware service request scheduling device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above time-aware service request scheduling method.

[0194] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 22, the steps of the above-mentioned time-aware service request scheduling method are implemented.

[0195] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0196] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above time-aware service request scheduling method.

[0197] Furthermore, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the time-aware service request scheduling method when executed by a processor.

[0198] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above time-aware service request scheduling method.

[0199] The above is a detailed introduction to a time-aware service request scheduling method, device, apparatus and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0200] It should also be noted that, in this specification, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A time-aware service request scheduling method, characterized in that: include: Get the pre-trained model and the request task length of each business request; The request task length of each business request is input into the pre-trained model to output the estimated completion time corresponding to each business request; Determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0; When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that each service request has completed scheduling in the current round; When the sum of the remaining times of each business request is less than 0, the remaining times of each business request are sorted to determine the target business request, and the target business request is postponed to the next round of scheduling; the maximum estimated completion time is determined among the remaining business requests except the target business request, and the new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and the process returns to the step of comparing the sum of the remaining times of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

2. The time-aware service request scheduling method according to claim 1, characterized in that: The process of determining the pre-training model includes: Get the initial pre-trained model; Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request; Inputting the request task length, hardware board parameters and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling phase and decoding phase processing on each historical business request to obtain the corresponding first completion time; If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model; If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements to obtain the pre-trained model.

3. The time-aware service request scheduling method according to claim 2, characterized in that: The remaining time of the inference phase corresponding to the pre-trained model is determined based on the estimated completion time of each business request, including: Get the current inference stage of each business request in the pre-trained model; If the current reasoning stage of the business request is the pre-filling stage, the arrival time and current time of the business request are obtained; the remaining pre-filling time of the pre-filling stage of the business request is determined according to the arrival time of the business request, the pre-filling timeout threshold time, the current time and the first estimated completion time; If the current reasoning stage of the business request is the decoding stage, the arrival time of the business request, the current time and the actual request business length of the business request are obtained; the remaining decoding time of the decoding stage of the business request is determined according to the arrival time of the business request, the pre-filled timeout threshold time, the actual request business length, the decoding timeout threshold time, the current time and the first estimated completion time.

4. The time-aware service request scheduling method according to claim 3, characterized in that: When the service request is a first service request, determining a pre-filling remaining time of a pre-filling phase of the service request according to an arrival time of the first service request, a pre-filling timeout threshold time, a current time, and a first estimated completion time includes: Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time; The current time of the first service request and the first estimated completion time of the first service request are added together to obtain a second time; Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request; Correspondingly, when the service request is a second service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time and the first estimated completion time, including: Adding the arrival time of the second service request and the pre-filling timeout threshold time to obtain a third time; Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request; The fourth time is obtained by multiplying the actual requested service length of the second service request by the decoding timeout threshold time; Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time; The third time and the fourth time are added to obtain a sixth time; The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

5. The time-aware service request scheduling method according to any one of claims 1 to 4, characterized in that: Sort the remaining time of each service request to determine the target service request, including: Sort the service requests from large to small according to the remaining time of each service request to obtain sorted service requests; Get the arrival time of each business request; The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

6. The time-aware service request scheduling method according to claim 5, characterized in that: The new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time corresponding to the remaining business requests except the target business request and the maximum estimated completion time, including: Obtaining a maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request; Get the current inference stage of each other business request in the pre-trained model; If the current reasoning stage of the remaining business requests is the pre-filling stage, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling stage of the remaining business requests is determined according to the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time and the maximum preset completion time; If the current inference stage of the remaining business requests is the decoding stage, then obtain the arrival time, current time and actual request business length of the remaining business requests; determine the remaining decoding time of the decoding stage of the business request based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

7. The time-aware service request scheduling method according to claim 6, characterized in that: When the remaining service requests are third service requests, determining the remaining pre-filling time of the pre-filling phase of the remaining service requests according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes: The arrival time of the third service request and the pre-filling timeout threshold time are added together to obtain a seventh time; The eighth time is obtained by adding the current time of the third business request, the first estimated completion time and the maximum estimated completion time of the third business request; Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request; Correspondingly, when the remaining service request is a fourth service request, the decoding remaining time of the decoding phase of the service request is determined according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual request service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time, including: The arrival time of the fourth service request and the pre-filling timeout threshold time are added together to obtain a ninth time; Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request; The actual requested service length of the fourth service request is multiplied by the decoding timeout threshold to obtain a tenth time; The current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time are added together to obtain an eleventh time; The ninth time and the tenth time are added together to obtain the twelfth time; The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

8. A time-aware service request scheduling device, characterized in that: include: The acquisition module is used to obtain the pre-trained model and the request task length of each business request; The request task length of each business request is input into the pre-trained model to output the estimated completion time corresponding to each business request; The first determination module is used to determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0; The second determination module is used to determine that all service requests have completed scheduling in the current round when the sum of the remaining time of each service request is greater than or equal to 0; The third determination module is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the reasoning stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

9. A time-aware service request scheduling device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the time-aware service request scheduling method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the time-aware service request scheduling method according to any one of claims 1 to 7 are implemented.

Images