Time-aware service request scheduling method, device, apparatus, and medium

Abstract

The present invention discloses a time-aware business request scheduling method, equipment, device and medium, and relates to the field of data transmission technology. The estimated completion time of each business request is predicted by a pre-trained model to improve the prediction accuracy of the business request in the request scheduling completion time. The remaining time of each business request in the corresponding reasoning stage is calculated based on the estimated completion time. If the sum of the remaining time of each business request is greater than or equal to 0, it means that the corresponding remaining time of each business request in the scheduling process is sufficient to complete the scheduling in the current round. If it is less than 0, it means that the remaining time of each business request in the current round is insufficient and all scheduling cannot be completed. The target business request is postponed to the next round of scheduling, and the remaining time of the remaining business requests is recalculated until the scheduling of the business request is completed. Ensure that the remaining business requests in the current round are scheduled to avoid timeouts.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a time-aware service request scheduling method, equipment, device and medium. Background Art

[0002] The pre-filling stage and the decoding stage are two key stages in the inference process of the pre-trained model. In the parallel processing corresponding to different requests arriving at different times, it is necessary to ensure that the delay requirements of the decoding stage of the original request are met, and also to ensure that the delay requirements of the pre-filling stage of the new request are met.

[0003] In the conventional processing corresponding to multiple business requests, if the corresponding timeout period is strictly required, there will be a certain timeout phenomenon in the scheduling and processing process of multiple business requests, resulting in reduced reliability and efficiency of system processing.

[0004] Therefore, how to avoid timeout in the scheduling process of multiple service requests to improve the reliability and efficiency of system processing is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a time-aware service request scheduling method, equipment, device and medium to solve the problem of timeout in the scheduling process of multiple service requests leading to reduced reliability and efficiency of system processing.

[0006] To solve the above technical problems, the present invention provides a time-aware service request scheduling method, comprising:

[0007] Obtain the pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output the estimated completion time corresponding to each business request;

[0008] Determine the remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0009] When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that all service requests have completed scheduling in the current round;

[0010] When the sum of the remaining times of each business request is less than 0, the remaining times of each business request are sorted to determine the target business request, and the target business request is postponed to the next round of scheduling; the maximum estimated completion time is determined among the remaining business requests except the target business request, and the new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and the process returns to the step of comparing the sum of the remaining times of each business request with 0 until the scheduling of each business request is completed; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0011] On the one hand, the process of determining the pre-trained model includes:

[0012] Get the initial pre-trained model;

[0013] Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request;

[0014] Inputting the request task length, hardware board parameters, and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling and decoding phases on each historical business request to obtain the corresponding first completion time;

[0015] If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model;

[0016] If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request respectively to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements and the pre-trained model is obtained.

[0017] On the other hand, the remaining time of the inference phase corresponding to the pre-trained model is determined based on the estimated completion time of each business request, including:

[0018] Get the current inference stage of each business request in the pre-trained model;

[0019] If the current reasoning phase of the business request is the pre-filling phase, the arrival time and current time of the business request are obtained; and the remaining pre-filling time of the pre-filling phase of the business request is determined according to the arrival time of the business request, the pre-filling timeout threshold time, the current time, and the first estimated completion time.

[0020] If the current inference stage of the business request is the decoding stage, the arrival time of the business request, the current time and the actual request business length of the business request are obtained; the remaining decoding time of the decoding stage of the business request is determined according to the arrival time of the business request, the pre-filled timeout threshold time, the actual request business length, the decoding timeout threshold time, the current time and the first estimated completion time.

[0021] On the other hand, when the service request is a first service request, determining the remaining pre-filling time of the pre-filling phase of the service request according to the arrival time of the first service request, the pre-filling timeout threshold time, the current time, and the first estimated completion time includes:

[0022] Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time;

[0023] Adding the current time of the first business request and the first estimated completion time of the first business request to obtain a second time;

[0024] Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request;

[0025] Correspondingly, when the service request is a second service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, and the first estimated completion time includes:

[0026] Adding the arrival time of the second service request and the pre-filled timeout threshold time to obtain a third time;

[0027] Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request;

[0028] Multiplying the actual requested service length of the second service request by the decoding timeout threshold time to obtain a fourth time;

[0029] Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time;

[0030] The third time and the fourth time are added together to obtain a sixth time;

[0031] The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

[0032] On the other hand, the remaining time of each service request is sorted to determine the target service request, including:

[0033] Sort the service requests by their remaining time from largest to smallest to obtain sorted service requests;

[0034] Get the arrival time of each business request;

[0035] The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

[0036] On the other hand, the new remaining time of the inference phase corresponding to the pre-trained model is determined based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, including:

[0037] Obtaining a maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request;

[0038] Get the current inference stage of each other business request in the pre-trained model;

[0039] If the current reasoning phase of the remaining business requests is the pre-filling phase, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling phase of the remaining business requests is determined based on the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time;

[0040] If the current inference stage of the remaining business requests is the decoding stage, the arrival time, current time and actual request business length of the remaining business requests are obtained; the remaining decoding time of the decoding stage of the business request is determined based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

[0041] On the other hand, when the remaining service request is a third service request, determining the remaining pre-filling time of the pre-filling phase of the remaining service request according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0042] Adding the arrival time of the third service request and the pre-filled timeout threshold time to obtain a seventh time;

[0043] An eighth time is obtained by summing the current time of the third business request, the first estimated completion time, and the maximum estimated completion time of the third business request;

[0044] Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request;

[0045] Correspondingly, when the remaining service request is the fourth service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0046] Adding the arrival time of the fourth service request and the pre-filling timeout threshold time to obtain a ninth time;

[0047] Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request;

[0048] Multiplying the actual request service length of the fourth service request by the decoding timeout threshold to obtain a tenth time;

[0049] The eleventh time is obtained by adding the current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time;

[0050] The ninth time and the tenth time are added together to obtain the twelfth time;

[0051] The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

[0052] To solve the above technical problems, the present invention further provides a time-aware service request scheduling device, comprising:

[0053] An acquisition module is used to obtain the pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output the estimated completion time corresponding to each business request;

[0054] A first determination module is configured to determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0055] The second determining module is configured to determine that all service requests have completed scheduling in the current round when the sum of the remaining times of the service requests is greater than or equal to 0;

[0056] The third determination module is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the inference stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0057] To solve the above technical problems, the present invention further provides a time-aware service request scheduling device, comprising:

[0058] memory for storing computer programs;

[0059] The processor is configured to implement the steps of the time-aware service request scheduling method described above when executing the computer program.

[0060] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the time-aware service request scheduling method as described above are implemented.

[0061] The present invention provides a time-aware business request scheduling method, comprising: obtaining a pre-trained model and a request task length of each business request; inputting the request task length of each business request into the pre-trained model to output an estimated completion time corresponding to each business request; determining the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; comparing the sum of the remaining times of each business request with 0; when the sum of the remaining times of each business request is greater than or equal to 0, determining that all business requests have completed scheduling in the current round; when the sum of the remaining times of each business request is less than 0, sorting the remaining times of each business request to determine a target business request, and postponing the target business request to the next round of scheduling; determining the maximum estimated completion time among the remaining business requests except the target business request; and determining a new remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and returning to the step of comparing the sum of the remaining times of each business request with 0 until all business requests have completed scheduling; wherein the urgency of the target business request is less than the urgency of the remaining business requests.

[0062] The beneficial effect of the present invention is to obtain the estimated completion time of each business request through the prediction of the pre-trained model, so as to improve the prediction accuracy of the completion time of the business request in the request scheduling. The remaining time of each business request in the corresponding reasoning stage is calculated by the estimated completion time, and the sum of the remaining time of each business request is compared with 0. If it is greater than or equal to 0, it means that the remaining time corresponding to each business request in the scheduling process is sufficient to complete the scheduling in the current round. If it is less than 0, it means that the remaining time of each business request in the current round is insufficient and all scheduling cannot be completed. It is necessary to select the target business request with less urgency and postpone it to the next round of scheduling, and recalculate the remaining time of the remaining business requests until the scheduling of the business request is completed. Through the remaining time calculation method corresponding to the timeout indicator and the timeout strategy of the remaining time, it is guaranteed that the remaining business requests in the current round are completed, the timeout phenomenon is avoided, and the reliability and efficiency of the system processing are improved.

[0063] Secondly, training iterations are performed based on the original initial pre-trained model, adapting it to the business request scenarios of this embodiment and making it suitable for predicting estimated completion times in business scenarios, thereby improving prediction accuracy. The remaining pre-fill and decoding times determined during the inference phase for different business requests are based on the corresponding estimated completion times and the current arrival times of each business request. Timeout thresholds corresponding to the decoding and pre-fill phases are also incorporated, ensuring greater accuracy in the remaining time determination. The target business request is determined by sorting the arrival times and remaining times of the business requests, simplifying the determination process. The new remaining pre-fill and decoding times determined during the inference phase for different business requests are based on the corresponding estimated completion times, the current arrival times of each business request, and the maximum preset completion time. Timeout thresholds corresponding to the decoding and pre-fill phases are also incorporated to ensure scheduling within the maximum preset completion time, ensuring greater accuracy in the remaining time determination.

[0064] In addition, the present invention also provides a time-aware service request scheduling device, apparatus, and medium, which have the same beneficial effects as the above-mentioned time-aware service request scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0066] Figure 1Scheduling sequence diagrams for regular multiple requests;

[0067] Figure 2 A flowchart of a time-aware service request scheduling method provided by an embodiment of the present invention;

[0068] Figure 3 A structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention;

[0069] Figure 4 This is a structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0071] The core of the present invention is to provide a time-aware service request scheduling method, equipment, device and medium to solve the problem of timeout in the scheduling process of multiple service requests leading to reduced reliability and efficiency of system processing.

[0072] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0073] When the system simultaneously schedules multiple service requests, it must ensure that the latency of non-first tokens of requests meets preset requirements, as well as the latency of first tokens of new requests, to avoid timeouts and delays in the scheduling of multiple service requests in the current round. The inference process of pre-trained models is divided into two independent phases: the pre-population phase and the decoding phase, to improve the efficiency and scalability of pre-trained models. In the pre-population phase, the system pre-processes input data, performing operations such as data loading, preprocessing, and feature extraction. This aims to prepare all necessary information for rapid model inference in the decoding phase. The decoding phase is the actual model inference phase, using the data prepared in the pre-population phase to perform calculations and generate output results. For example, for the question "What day is today?", the first token in the output is "today," representing the pre-population phase; the second token in the output is "week," representing the non-first token, representing the decoding phase. The corresponding delays for these two phases are the first token delay (TTFT) and the non-first token delay (TPOT).

[0074] Figure 1 Scheduling sequence diagrams for regular multiple requests, such as Figure 1 As shown, the slash in the box represents the decoding inference stage, and the horizontal and vertical lines in the box represent the pre-filling stage. Requests 1-3 are running requests, and requests 4 and 5 are requests that arrive at the 51st ms. The inference scheduling needs to run the pre-filling stage first, and then the decoding stage. After the pre-filling stage is completed, the delay corresponding to the first word output by the language model (the first word delay) can be obtained, and the subsequent decoding stage outputs the delay corresponding to the subsequent words one by one (the non-first word delay). It is necessary to ensure that the non-first word delay of the request meets the preset requirements, and it is also necessary to ensure that the first word delay of the new request meets the preset requirements. However, in business scenarios with strict time constraints, the adjustability of the corresponding multiple request scheduling is poor, resulting in serious timeouts and reduced system reliability. The time-aware business request scheduling method provided by the present invention can solve the above technical problems.

[0075] Figure 2 A flowchart of a time-aware service request scheduling method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0076] S11: Obtain a pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output an estimated completion time corresponding to each business request;

[0077] S12: Determine the remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time of each business request; compare the sum of the remaining time of each business request with 0;

[0078] S13: When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that all service requests have completed scheduling in the current round;

[0079] S14: When the sum of the remaining time of each business request is less than 0, sort the remaining time of each business request to determine the target business request, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request; and determine the new remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to step S12 of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0080] Specifically, time awareness is to ensure that data is scheduled and transmitted between corresponding business requests within a certain and predictable time range, providing reliable request scheduling and transmission guarantees for time-sensitive business requests. The pre-trained model is a machine learning model with large-scale parameters and complex computing structure. It learns common features such as language and vision by pre-training on a large amount of data, so as to handle various complex tasks. The request task length of each business request usually refers to the time required to process a business request, including the entire process from receiving the request to sending the response. In different application scenarios, optimizing the request task length can improve the overall performance of the system and user experience. The request task length of each business request is an important indicator to measure the efficiency of business request processing. Through various technical means and best practices, the request task length can be effectively shortened and the system performance can be improved.

[0081] Inputting the request task length of the business request into the pre-trained model can obtain the estimated completion time corresponding to each business request. In other words, the completion time of each business request can be predicted by the pre-trained model, thereby determining the remaining time of step S12.

[0082] There is no limitation on how to predict in step S11, nor on the specific pre-trained model used. It can be a conventional pre-trained model or a pre-trained model tailored to the current service request and the corresponding hardware board parameter settings, and can be set according to actual conditions. This embodiment uses a pre-trained model to predict the completion time of the service request, and improves the accuracy of the prediction through model calculation.

[0083] In step S12, the remaining time of the inference stage is determined based on the estimated completion time of each business request. It can be understood that the inference stage corresponding to each business request is the pre-filling stage or the decoding stage, and the specific parameters of the remaining time of the inference stage are determined by the estimated completion time corresponding to each business request. Here, a preset time corresponding to each business request can be provided, and the corresponding remaining time is obtained based on the preset time minus the estimated completion time. The preset time here can be the same time corresponding to each business request, or different time parameters can be set for each business request, which is not limited here. In addition, the remaining time can also be determined in combination with the actual arrival time of each business request, the pre-filled timeout threshold, etc., and the remaining time can also be calculated by a specific algorithm, which is not limited here and can be set according to actual conditions.

[0084] The sum of the remaining time of each business request in steps S13 and S14 is to regard each business request as a set, and comprehensively consider the remaining time corresponding to each business request in the set. If the sum of the remaining time is greater than or equal to 0, it means that there is surplus remaining time in the current set, that is, there will be remaining time after all business requests are scheduled, or the scheduling time is just used up after the scheduling is completed. If the sum of the remaining time is less than 0, it means that there is a serious timeout in the current set. The serious timeout here is based on all business requests. Non-urgent business requests in the current set can be scheduled in the next round to relieve the remaining time for scheduling the remaining business requests in the current set of the current round. It should be noted that the sum of the remaining time can include business requests with a remaining time less than 0, a remaining time equal to 0, or a remaining time greater than 0, as long as the sum of the remaining time is calculated and determined.

[0085] In step S14, when the sum of the remaining time is less than 0, it is necessary to consider selecting the target business request from the remaining time of each business request, and determine the target business request through time sorting, so as to postpone the target business request to the next round of scheduling. In addition, corresponding to the sorting operation, it is helpful to identify that the number of target business requests here is not limited, it can be one or more, and the remaining time corresponding to the target business request can be the maximum remaining time, or the remaining time corresponding to the middle among multiple business requests. Other factors can also be considered here, such as the arrival time of the business request. If it arrives the latest and the remaining time is in the middle, it can be delayed to the next round to complete the scheduling without timeout, and this is used as the target business request. The urgency of the target business request is less than that of the other business requests. It should be noted that the urgency here can be determined by the remaining time, or it can be determined based on multiple factors such as the arrival time of the target business request and the remaining time, which are not limited here.

[0086] After excluding the target business request, the remaining business requests are re-formed into a new set, and the corresponding remaining times are recalculated in this new set. This mainly updates the remaining time of the inference phase, while also fully considering the maximum remaining time corresponding to all remaining business requests. Therefore, the maximum estimated completion time is required. The new remaining time is determined based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests. In the embodiment of determining the remaining time in step S12 above, the maximum estimated completion time is added to determine the updated new remaining time.

[0087] Based on the new remaining time, the sum of the remaining time in step S12 is compared with 0 to perform the judgment before scheduling again until all service requests are scheduled.

[0088] An embodiment of the present invention provides a time-aware service request scheduling method, comprising: obtaining a pre-trained model and a request task length of each service request; inputting the request task length of each service request into the pre-trained model to output an estimated completion time corresponding to each service request; determining the remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time of each service request; comparing the sum of the remaining times of each service request with 0; when the sum of the remaining times of each service request is greater than or equal to 0, determining that all service requests have completed scheduling in the current round; when the sum of the remaining times of each service request is less than 0, sorting the remaining times of each service request to determine a target service request, and postponing the target service request to the next round of scheduling; determining a maximum estimated completion time among the remaining service requests other than the target service request; and determining a new remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion times and the maximum estimated completion times corresponding to the remaining service requests other than the target service request, and returning to the step of comparing the sum of the remaining times of each service request with 0 until all service requests have completed scheduling; wherein the urgency of the target service request is less than the urgency of the remaining service requests. The estimated completion time of each business request is predicted through the pre-trained model to improve the accuracy of the prediction of the completion time of the business request in the request scheduling. The remaining time of each business request in the corresponding reasoning stage is calculated based on the estimated completion time. The sum of the remaining time of each business request is compared with 0. If it is greater than or equal to 0, it means that the remaining time corresponding to each business request in the scheduling process is sufficient to complete the scheduling in the current round. If it is less than 0, it means that the remaining time of each business request in the current round is insufficient to complete the entire scheduling. In this case, it is necessary to select target business requests with less urgency and postpone them to the next round of scheduling. The remaining time of the remaining business requests is recalculated until the scheduling of the business requests is completed. The remaining time calculation method corresponding to the timeout indicator and the timeout strategy of the remaining time are used to ensure that the remaining business requests in the current round are scheduled, avoid timeouts, and improve the reliability and efficiency of system processing.

[0089] In some embodiments, the process of determining the pre-trained model includes:

[0090] Get the initial pre-trained model;

[0091] Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request;

[0092] Inputting the request task length, hardware board parameters, and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling and decoding phases on each historical business request to obtain the corresponding first completion time;

[0093] If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model;

[0094] If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request respectively to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements and the pre-trained model is obtained.

[0095] It can be understood that the pre-training model in this embodiment is based on the initial pre-training model setting. There is no limitation on the specific model of the initial pre-training model. It can be a neural network model in deep learning or other models. It is mainly used to process and understand complex systems of natural language.

[0096] The input parameters of the pre-trained model include the request task length corresponding to each historical business request, the hardware board parameters, and the pre-trained model parameters themselves. Based on these three parameters, the hardware board is used in the initial pre-trained model to process each historical business request in the pre-filling stage and the decoding stage to obtain the corresponding first completion time. Hardware board parameters generally refer to the specifications and performance indicators of the expansion card installed on the computer motherboard, mainly including parameters such as hardware type, interface type, bus type, memory size, and output interface. The pre-trained model parameters are the configuration and scale of the model during the pre-training stage, including but not limited to the number of layers, number of hidden units, number of attention heads, total number of model parameters, etc.

[0097] If the current corresponding first completion time meets the preset requirements, the initial pre-trained model is used as the final pre-trained model. If it does not meet the preset requirements, training continues until the output first completion time meets the preset requirements, and the pre-trained model is determined. The preset requirements here can include the first completion time not exceeding a certain threshold, or can be based on a limit on the number of iterations, etc., which are not limited here and can be set according to actual circumstances.

[0098] This embodiment provides training iterations based on the original initial pre-training model, so that it is combined with the business request scenario of this embodiment, suitable for predicting the estimated completion time in the business scenario, and improves the accuracy of the prediction.

[0099] In some other embodiments, the process of determining the pre-trained model includes:

[0100] Get the request task length corresponding to each historical business request;

[0101] Using a hardware board to process each historical service request in a pre-filling phase and a decoding phase respectively to obtain a corresponding first completion time;

[0102] The first completion time of the middle collection point between each two adjacent collection points is obtained by averaging the first completion time determined by each two adjacent collection points according to the request task length corresponding to each historical business request;

[0103] A pre-training model is obtained by performing linear fitting processing on all the collection points of the first completion time and the request task length corresponding to each historical business request.

[0104] In this embodiment, the pre-training model takes into account the linear fitting process of the data. The request task lengths corresponding to each historical service request are processed during the pre-population and decoding phases to obtain their respective first completion times, forming a mapping relationship between the request task length and time parameters. To expand the collection points, the first completion times of two adjacent collection points are averaged to obtain the first completion time of the middle collection point between each pair of adjacent collection points. A linear fitting process is performed on each collection point and its corresponding first completion time to determine the functional relationship, which is then used as the pre-training model.

[0105] The pre-training model provided in this embodiment is determined based on two parameters: the requested task length and the first completion time, and is obtained through a linear fitting relationship, which simplifies the determination process of the pre-training model and improves data processing efficiency.

[0106] In some embodiments, when the service request is a first service request, determining the remaining pre-filling time of the pre-filling phase of the service request according to the arrival time of the first service request, the pre-filling timeout threshold time, the current time, and the first estimated completion time includes:

[0107] Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time;

[0108] Adding the current time of the first business request and the first estimated completion time of the first business request to obtain a second time;

[0109] Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request;

[0110] Correspondingly, when the service request is a second service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, and the first estimated completion time includes:

[0111] Adding the arrival time of the second service request and the pre-filled timeout threshold time to obtain a third time;

[0112] Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request;

[0113] Multiplying the actual requested service length of the second service request by the decoding timeout threshold time to obtain a fourth time;

[0114] Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time;

[0115] The third time and the fourth time are added together to obtain a sixth time;

[0116] The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

[0117] Specifically, the remaining time for each inference stage is as follows:

[0118] 1. Pre-filling time in the pre-filling stage:

[0119] T1=t1+t2-(t3+t4);

[0120] Among them, t1 is the arrival time of the first business request, t2 is the pre-filling timeout threshold time; t1+t2 is the first time; t3 is the current time of the first business request, t4 is the first estimated completion time of the first business request; t3+t4 is the second time; T1 is the remaining pre-filling time of the first business request.

[0121] 2. Remaining decoding time in the decoding phase:

[0122] T2=(t11+t2)+(L1-1)*t22-(t13+t14);

[0123] Among them, t11 is the arrival time of the second business request, t2 is the pre-filling timeout threshold time; t11+t2 is the third time; L1 is the requested business length of the second business request, L1-1 is the actual requested business length of the second business request; t22 is the decoding timeout threshold time, (L1-1)*t22 is the fourth time; t13 is the current time of the second business request, t14 is the first estimated completion time of the second business request; t13+t14 is the fifth time, (t11+t2)+(L1-1)*t22 is the sixth time, and T2 is the remaining decoding time of the second business request.

[0124] It should be noted that L1-1 in this embodiment takes into account that 1 is the length unit of a token, and the length unit of the first token is in the pre-filling stage and needs to be deleted during the calculation and decoding stage.

[0125] This embodiment provides the pre-filling remaining time and decoding remaining time determined in the corresponding inference stage under different business requests, based on the corresponding estimated completion time and the current arrival time corresponding to each business request, and at the same time adds the timeout threshold time corresponding to the decoding and pre-filling stages, so that the determined remaining time is more accurate.

[0126] In some embodiments, sorting the remaining time of each service request to determine the target service request includes:

[0127] Sort the service requests by their remaining time from largest to smallest to obtain sorted service requests;

[0128] Get the arrival time of each business request;

[0129] The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

[0130] Specifically, there's no specific limit on the number of target service requests; it can be one or more. The latest service request is selected based on the order of their arrival times. If it's within the remaining time of the previous N service requests, where N is a positive integer, it's set as the target service request. The longer the remaining time, the more time there is left to defer the service request to the next round.

[0131] The target service requests provided in this embodiment are determined by sorting the order of arrival time of the service requests and the order of remaining time of the service requests, making the determination process simple and clear.

[0132] In some embodiments, determining a new remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests other than the target business request includes:

[0133] Obtain the maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request;

[0134] Get the current inference stage of each other business request in the pre-trained model;

[0135] If the current reasoning phase of the remaining business requests is the pre-filling phase, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling phase of the remaining business requests is determined based on the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time;

[0136] If the current inference stage of the remaining business requests is the decoding stage, the arrival time, current time and actual request business length of the remaining business requests are obtained; the remaining decoding time of the decoding stage of the business request is determined based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

[0137] In combination with the above embodiments, the new remaining time corresponding to each reasoning stage under the remaining business requests takes into account the corresponding maximum preset completion time under the current set to ensure that the scheduling problems of the remaining business requests are completed within the current maximum preset completion time.

[0138] In some embodiments, when the remaining service request is a third service request, determining the remaining pre-filling time of the pre-filling phase of the remaining service request according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0139] Adding the arrival time of the third service request and the pre-filled timeout threshold time to obtain a seventh time;

[0140] An eighth time is obtained by summing the current time of the third business request, the first estimated completion time, and the maximum estimated completion time of the third business request;

[0141] Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request;

[0142] Correspondingly, when the remaining service request is the fourth service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes:

[0143] Adding the arrival time of the fourth service request and the pre-filling timeout threshold time to obtain a ninth time;

[0144] Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request;

[0145] Multiplying the actual request service length of the fourth service request by the decoding timeout threshold to obtain a tenth time;

[0146] The eleventh time is obtained by adding the current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time;

[0147] The ninth time and the tenth time are added together to obtain the twelfth time;

[0148] The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

[0149] In combination with the above embodiment, the corresponding new pre-filling remaining time is:

[0150] T3= t1'+t2-(t3'+t4'+t5');

[0151] Among them, t1' is the arrival time of the third business request, t2 is the pre-filling timeout threshold time; t1'+t2 is the seventh time; t3' is the current time of the third business request, t4' is the first estimated completion time of the third business request; t5' is the maximum estimated completion time; t3'+t4'+t5' is the eighth time; T3 is the pre-filling remaining time of the third business request.

[0152] The corresponding new decoding remaining time is:

[0153] T4=(t11'+t2)+(L1'-1)*t22-(t13'+t14'+ t5');

[0154] Among them, t11' is the arrival time of the fourth business request, t2 is the pre-filling timeout threshold time; t11'+t2 is the ninth time; L1' is the request service length of the fourth business request, L1'-1 is the actual request service length of the fourth business request; t22 is the decoding timeout threshold time, (L1'-1)*t22 is the tenth time; t13' is the current time of the fourth business request, t14' is the first estimated completion time of the fourth business request; t5' is the maximum estimated completion time; t13'+t14'+t5' is the eleventh time, (t11'+t2)+(L1'-1)*t22 is the twelfth time, and T4 is the remaining decoding time of the fourth business request.

[0155] This embodiment provides new pre-filling remaining time and decoding remaining time determined for the corresponding inference stage under different business requests. This is based on the corresponding estimated completion time, the current arrival time corresponding to each business request, and the maximum preset completion time. At the same time, the timeout threshold time corresponding to the decoding and pre-filling stages is added to ensure that scheduling is achieved within the maximum preset completion time, making the determined remaining time more accurate.

[0156] In addition, the deployment scheduling scheme of the present invention is as follows:

[0157] Set the scheduling request set S;

[0158] The present invention first sets the current scheduling request set S, where S includes existing requests and newly added requests. Each request in S will be evaluated in subsequent steps using a specific calculation method to determine whether it should be scheduled in the current round.

[0159] Calculate the time parameters of the request;

[0160] For each request s in the set S, the required computing time is calculated by a specific computing function Q, which is recorded as T. At the same time, the remaining time P of the request s is calculated. The calculation process includes:

[0161] Prefill remaining time: The prefill remaining time is equal to the request arrival time plus the prefill timeout threshold, minus the sum of the current time and the calculation time.

[0162] Decoding remaining time: The decoding remaining time is equal to the request arrival time plus the pre-fill timeout threshold, plus (output length minus 1) multiplied by the decoding timeout threshold, minus the sum of the current time and the calculation time.

[0163] Determine the cumulative sum of remaining time;

[0164] After calculating the remaining time of all requests, if the cumulative sum of all remaining times is greater than 0, it means that all requests in the current round can be scheduled. Otherwise, proceed to the next step to further process the request set S.

[0165] Request sorting;

[0166] Sort the requests in set S according to their remaining timeouts to obtain a sorted request set D. This sorting helps identify more urgent requests for priority processing.

[0167] Pop up the request and update the remaining time;

[0168] Pop a request from the sorted set D, which means that the request will be postponed to the next round in this round of scheduling. After popping the request, recalculate its remaining time, including:

[0169] Use function Q to calculate the computation time of the request in set D, denoted as U.

[0170] Update the remaining pre-fill time to the requested arrival time plus the pre-fill timeout threshold, minus the sum of the current time, the calculation time, and the maximum value in U.

[0171] Update the decoding remaining time to the requested arrival time plus the prefill timeout threshold, plus (output length minus 1) multiplied by the decoding timeout threshold, minus the sum of the current time, the computation time, and the maximum value in U.

[0172] Re-evaluate the remaining accumulated time;

[0173] In the updated set D, the cumulative sum of the remaining time is recalculated. If the cumulative sum is greater than 0, all requests in set D are scheduled; otherwise, requests in D are continuously popped and the remaining time is updated until the condition is met or the set D is empty.

[0174] The ultimate scheduling solution;

[0175] If the D set is empty, the system runs according to the scheduling scheme with the largest remaining time to ensure that the system can process requests as efficiently as possible within the time constraints.

[0176] The solution involved in the present invention is applicable to both Graphics Processing Unit (GPU) scenarios and non-GPU scenarios, and is not limited here.

[0177] The above describes in detail various embodiments corresponding to the time-aware service request scheduling method. On this basis, the present invention also discloses a time-aware service request scheduling device corresponding to the above method. Figure 3 This is a structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention. Figure 3 As shown, the time-aware service request scheduling device includes:

[0178] The acquisition module 11 is used to obtain the pre-trained model and the request task length of each business request; and input the request task length of each business request into the pre-trained model to output the estimated completion time corresponding to each business request;

[0179] The first determination module 12 is configured to determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0;

[0180] The second determining module 13 is configured to determine that all service requests have completed scheduling in the current round when the sum of the remaining times of the service requests is greater than or equal to 0;

[0181] The third determination module 14 is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the inference stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

[0182] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, and will not be repeated here.

[0183] For an introduction to a time-aware service request scheduling device provided by the present invention, please refer to the above method embodiment, and the present invention will not go into details here. It has the same beneficial effects as the above-mentioned time-aware service request scheduling method.

[0184] Figure 4 A structural diagram of a time-aware service request scheduling device provided by an embodiment of the present invention, such as Figure 4 As shown, the device includes:

[0185] Memory 21, for storing computer programs;

[0186] The processor 22 is configured to implement the steps of the time-aware service request scheduling method when executing a computer program.

[0187] The time-aware service request scheduling device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0188] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented in at least one hardware form: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a GPU, which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is responsible for processing computing operations related to machine learning.

[0189] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the time-aware service request scheduling method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include but is not limited to data involved in the time-aware service request scheduling method, etc.

[0190] In some embodiments, the time-aware service request scheduling device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0191] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the time-aware service request scheduling device, and may include more or fewer components than shown in the figure.

[0192] The processor 22 implements the time-aware service request scheduling method provided by any of the above embodiments by calling instructions stored in the memory 21 .

[0193] For an introduction to a time-aware service request scheduling device provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above time-aware service request scheduling method.

[0194] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by the processor 22, the steps of the above-mentioned time-aware service request scheduling method are implemented.

[0195] It is understood that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0196] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above time-aware service request scheduling method.

[0197] Furthermore, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the time-aware service request scheduling method when executed by a processor.

[0198] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above time-aware service request scheduling method.

[0199] The above is a detailed introduction to the time-aware service request scheduling method, equipment, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0200] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A time-aware service request scheduling method, characterized in that: include: Get the pre-trained model and the request task length of each business request; The request task length of each business request is input into the pre-trained model to output the estimated completion time corresponding to each business request; Determine the remaining time of the inference phase corresponding to the pre-trained model based on the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0; When the sum of the remaining time of each service request is greater than or equal to 0, it is determined that all service requests have completed scheduling in the current round; When the sum of the remaining times of each business request is less than 0, the remaining times of each business request are sorted to determine the target business request, and the target business request is postponed to the next round of scheduling; the maximum estimated completion time is determined among the remaining business requests except the target business request, and the new remaining time of the inference phase corresponding to the pre-trained model is determined according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and the process returns to the step of comparing the sum of the remaining times of each business request with 0 until the scheduling of each business request is completed; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

2. The time-aware service request scheduling method according to claim 1, characterized in that: The process of determining the pre-training model includes: Get the initial pre-trained model; Obtain the request task length, hardware board parameters, and pre-trained model parameters corresponding to each historical business request; Inputting the request task length, hardware board parameters, and pre-training model parameters corresponding to each historical business request into the initial pre-training model, so as to use the hardware board to perform pre-filling and decoding phases on each historical business request to obtain the corresponding first completion time; If the first completion time corresponding to the pre-filling stage and the decoding stage meets the preset requirements, the initial pre-trained model is used as the pre-trained model; If the first completion time corresponding to the pre-filling stage and the decoding stage does not meet the preset requirements, return to the step of using the hardware board to perform the pre-filling stage and the decoding stage on each historical business request respectively to obtain the corresponding first completion time for calculation until the corresponding first completion time meets the preset requirements and the pre-trained model is obtained.

3. The time-aware service request scheduling method according to claim 2, characterized in that: Determine the remaining time for the inference phase of the pre-trained model based on the estimated completion time of each business request, including: Get the current inference stage of each business request in the pre-trained model; If the current reasoning phase of the business request is the pre-filling phase, the arrival time and current time of the business request are obtained; and the remaining pre-filling time of the pre-filling phase of the business request is determined according to the arrival time of the business request, the pre-filling timeout threshold time, the current time, and the first estimated completion time. If the current inference stage of the business request is the decoding stage, the arrival time of the business request, the current time and the actual request business length of the business request are obtained; the remaining decoding time of the decoding stage of the business request is determined according to the arrival time of the business request, the pre-filled timeout threshold time, the actual request business length, the decoding timeout threshold time, the current time and the first estimated completion time.

4. The time-aware service request scheduling method according to claim 3, characterized in that: When the service request is a first service request, determining a remaining pre-filling time in a pre-filling phase of the service request according to an arrival time of the first service request, a pre-filling timeout threshold time, a current time, and a first estimated completion time includes: Adding the arrival time of the first service request and the pre-filled timeout threshold time to obtain a first time; Adding the current time of the first business request and the first estimated completion time of the first business request to obtain a second time; Subtracting the first time from the second time to obtain a pre-filled remaining time for the first service request; Correspondingly, when the service request is a second service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the second service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, and the first estimated completion time includes: Adding the arrival time of the second service request and the pre-filled timeout threshold time to obtain a third time; Subtract 1 from the requested service length of the second service request to obtain the actual requested service length of the second service request; Multiplying the actual requested service length of the second service request by the decoding timeout threshold time to obtain a fourth time; Adding the current time of the second business request and the first estimated completion time of the second business request to obtain a fifth time; The third time and the fourth time are added together to obtain a sixth time; The sixth time is subtracted from the fifth time to obtain the remaining decoding time of the second service request.

5. The time-aware service request scheduling method according to any one of claims 1 to 4, characterized in that: Sort the remaining time of each business request to determine the target business request, including: Sort the service requests by their remaining time from largest to smallest to obtain sorted service requests; Get the arrival time of each business request; The target service request is determined according to the order of arrival time of each service request and the remaining time of the first N service requests after sorting.

6. The time-aware service request scheduling method according to claim 5, characterized in that: The new remaining time of the inference phase corresponding to the pre-trained model is determined based on the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, including: Obtaining a maximum preset completion time among the estimated completion times corresponding to the remaining business requests except the target business request; Get the current inference stage of each other business request in the pre-trained model; If the current reasoning phase of the remaining business requests is the pre-filling phase, the arrival time and current time of the remaining business requests are obtained; the remaining pre-filling time of the pre-filling phase of the remaining business requests is determined based on the arrival time of the business requests, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time; If the current inference stage of the remaining business requests is the decoding stage, the arrival time, current time and actual request business length of the remaining business requests are obtained; the remaining decoding time of the decoding stage of the business request is determined based on the arrival time, pre-filled timeout threshold time, actual request business length, decoding timeout threshold time, current time, first estimated completion time and maximum preset completion time of the remaining business requests.

7. The time-aware service request scheduling method according to claim 6, characterized in that: When the remaining service request is a third service request, determining the remaining pre-filling time of the pre-filling phase of the remaining service request according to the arrival time of the third service request, the pre-filling timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes: Adding the arrival time of the third service request and the pre-filled timeout threshold time to obtain a seventh time; An eighth time is obtained by summing the current time of the third business request, the first estimated completion time, and the maximum estimated completion time of the third business request; Subtracting the seventh time from the eighth time to obtain the pre-filled remaining time of the third service request; Correspondingly, when the remaining service request is the fourth service request, determining the remaining decoding time of the decoding phase of the service request according to the arrival time of the fourth service request, the pre-filling timeout threshold time, the actual requested service length, the decoding timeout threshold time, the current time, the first estimated completion time, and the maximum preset completion time includes: Adding the arrival time of the fourth service request and the pre-filling timeout threshold time to obtain a ninth time; Subtract 1 from the requested service length of the fourth service request to obtain the actual requested service length of the fourth service request; Multiplying the actual request service length of the fourth service request by the decoding timeout threshold to obtain a tenth time; The eleventh time is obtained by adding the current time of the fourth business request, the first estimated completion time of the fourth business request, and the maximum preset completion time; The ninth time and the tenth time are added together to obtain the twelfth time; The twelfth time is subtracted from the eleventh time to obtain the remaining decryption time of the fourth service request.

8. A time-aware service request scheduling device, characterized in that: include: The acquisition module is used to obtain the pre-trained model and the request task length of each business request; The request task length of each business request is input into the pre-trained model to output the estimated completion time corresponding to each business request; A first determination module is configured to determine the remaining time of the inference phase corresponding to the pre-trained model according to the estimated completion time of each business request; and compare the sum of the remaining time of each business request with 0; The second determining module is configured to determine that all service requests have completed scheduling in the current round when the sum of the remaining times of the service requests is greater than or equal to 0; The third determination module is used to sort the remaining time of each business request to determine the target business request when the sum of the remaining time of each business request is less than 0, and postpone the target business request to the next round of scheduling; determine the maximum estimated completion time among the remaining business requests except the target business request, and determine the new remaining time of the inference stage corresponding to the pre-trained model according to the estimated completion time and the maximum estimated completion time corresponding to the remaining business requests except the target business request, and return to the step of comparing the sum of the remaining time of each business request with 0 until all business requests are scheduled; wherein, the urgency of the target business request is less than the urgency of the remaining business requests.

9. A time-aware service request scheduling device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the time-aware service request scheduling method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the time-aware service request scheduling method according to any one of claims 1 to 7 are implemented.

Images