A scheduling method based on large model output length prediction and related equipment

Abstract

This invention provides a scheduling method and related equipment based on large model output length prediction. The method includes: acquiring a large model input request set, wherein the large model input request set includes multiple large model input requests within a time period; determining the corresponding predicted output length based on each input request; determining the scheduling priority of each input request based on each predicted output length; and controlling the output order of each input request in the large model input request set based on the scheduling priority. The method in this application solves the problem of head-of-line blocking that often occurs in current Model-as-a-Service platforms, improving the utilization rate of platform computing resources and the overall efficiency of request processing.

Description

Technical Field

[0001] This application relates to the technical field of large model output prediction, and in particular to a scheduling method and related equipment based on large model output length prediction. Background Technology

[0002] Model-as-a-Service (MAS) platforms provide large language model inference capabilities through standardized interfaces. When the server adopts simple scheduling strategies such as First-In-First-Out (FIFO), head-of-queue blocking can easily occur, leading to increased queuing latency, decreased system throughput, and reduced resource utilization. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a scheduling method and related equipment based on large model output length prediction, so as to solve the problem of head-of-line blocking that is prone to occur in current model-as-a-service platforms.

[0004] To achieve the above objectives, this application provides a scheduling method based on large model output length prediction, comprising:

[0005] Obtain a large model input request set, wherein the large model input request set includes multiple large model input requests within a certain time period; Based on each input request, determine the corresponding predicted output length; Based on the predicted output length, the scheduling priority of each input request is determined. Based on the scheduling priority, the output order of each input request in the large model input request set is scheduled and controlled.

[0006] Optionally, determining the corresponding prediction output length based on each input request includes: A predetermined template is determined for each of the input requests, wherein the predetermined template has a corresponding statistical output length; Based on the statistical output length of the pre-determined template and the semantics of the input request, the predicted output length corresponding to the input request is determined.

[0007] Optionally, the pre-determining process of the pre-determined template includes: Obtain multiple historical input requests for the large model; Key features are extracted for each of the aforementioned historical input requests; Based on the extracted key features, the historical input requests are classified, and multiple statistical features of each type of historical input request are summarized into a pre-determined template corresponding to that type. The statistical features include at least the statistical output length; the statistical output length is obtained by statistically analyzing the historical output lengths corresponding to at least a portion of the historical input requests in this class.

[0008] Optionally, determining the pre-determined template corresponding to each input request includes: The request instruction of each input request is matched with the key features corresponding to the multiple predetermined templates; In response to determining that the request instruction of the input request matches the key feature corresponding to a predetermined template, the predetermined template is determined as the predetermined template of the input request; If the request instruction of the input request does not match the key features of any of the predetermined templates, then the default template is determined as the predetermined template of the input request, wherein the default template has a predetermined statistical output length.

[0009] Optionally, determining the predicted output length corresponding to the input request based on the statistical output length of the pre-determined template and the semantics of the input request includes: The input request is semantically parsed to extract the implicit semantic features corresponding to the input request, wherein the implicit semantic features are implicit features that affect the output length; Based on the statistical output length of the pre-determined template corresponding to the input request and the explicit semantic features corresponding to the input request, the first output length corresponding to the input request is determined, wherein the explicit semantic features are explicit features that affect the output length; Based on the implicit semantic features and the first output length, the predicted output length corresponding to the input request is determined.

[0010] Optionally, semantic parsing of the input request and extraction of the implicit semantic features corresponding to the input request include: The text of the input request is segmented and encoded to obtain the word sequence corresponding to the input request; Based on a preset truncation threshold, target words are truncated from the head and tail of the word sequence, and the truncated target words are concatenated to obtain a concatenated word sequence. Based on the concatenated word sequence, the implicit semantic features corresponding to the input request are determined.

[0011] Optionally, determining the scheduling priority of each input request based on the length of each predicted output includes: Arrange all the input requests in order of their corresponding predicted output lengths to form a request queue; Based on the order of the request queue, the scheduling priority corresponding to the large model input request set is determined.

[0012] Optionally, determining the scheduling priority of each input request based on the predicted output length further includes: In response to determining that there is a long input request in the request queue whose waiting time exceeds a preset time, the scheduling priority of the long input request is increased, wherein the long input request is an input request whose predicted output length is greater than a preset value.

[0013] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0014] Based on the same inventive concept, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0015] As can be seen from the above, the scheduling method based on large model output length prediction provided in this application includes: obtaining a large model input request set, wherein the large model input request set includes multiple large model input requests within a time period; determining the corresponding predicted output length based on each input request; determining the scheduling priority of each input request based on each predicted output length; and controlling the output order of each input request in the large model input request set based on the scheduling priority. This embodiment obtains each input request in the large model input request set within a time period and determines the corresponding predicted output length based on each input request, providing a precise and reliable basis for determining scheduling priorities and avoiding the blindness of priority division; determining the scheduling priority of each input request based on the predicted output length achieves differentiated management and refined scheduling of input request scheduling, effectively avoiding the head-of-line blocking problem that easily occurs in simple scheduling strategies, while improving the utilization rate of platform computing resources and the overall efficiency of request processing, and ensuring the timeliness and stability of responses to various requests. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating a scheduling method based on large model output length prediction, as shown in an embodiment of this application. Figure 2This is a schematic diagram of a scheduling device based on large model output length prediction, as shown in an embodiment of this application; Figure 3 This is a schematic diagram showing the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] As mentioned in the background, with the rapid iteration of artificial intelligence technology, large language models have been widely applied in various fields such as natural language processing, content generation, and intelligent interaction, giving rise to Model-as-a-Service (MAS) platforms. The core function of a MAS platform is to standardize the deployment of well-trained large language models and provide inference services through a unified, standardized interface. This allows various terminal devices and applications to quickly obtain model inference capabilities through interface calls without needing to deploy models separately, significantly lowering the application threshold of large language models and improving the efficiency of technology implementation. Currently, various internet services and enterprise applications widely rely on the platform's inference services to achieve core functions such as intelligent response, content generation, and data parsing. In the actual operation of a Model-as-a-Service (MAS) platform, the server-side scheduling strategy directly determines the platform's performance and service quality. Currently, most platforms still employ simple scheduling strategies such as First-In-First-Out (FIFO). The core logic of this strategy is to allocate computing resources and execute inference tasks sequentially according to the order in which requests arrive. Its advantage lies in its simplicity, the absence of complex priority judgment logic, and the ability to quickly complete basic request scheduling. However, as the number of requests connected to the platform continues to increase and the request types become increasingly complex, the limitations of this simple scheduling strategy gradually become apparent. The most prominent problem is the susceptibility to head-of-line blocking. The head-of-queue blocking problem triggers a series of chain reactions, severely impacting the platform's service performance and resource utilization efficiency: When a large request with long inference time and output is at the head of the queue, a large number of subsequent smaller requests with shorter inference time and output will be continuously blocked in the queue, unable to obtain computing resources in time to execute inference tasks, resulting in a significant increase in request queuing latency and a sharp decline in user experience; at the same time, the blocking state prevents the efficient use of computing resources, and some idle resources cannot be allocated to smaller requests that can be completed quickly, leading to a decrease in system throughput and low resource utilization; in the long run, such scheduling defects will also result in insufficient ability of the platform to handle high-concurrency requests, making it unable to adapt to diverse user request scenarios, and limiting the large-scale application and technological upgrades of the Model-as-a-Service platform.

[0021] To address the aforementioned issues, this application provides a scheduling method based on large model output length prediction.

[0022] The following is in conjunction with the appendix Figure 1-3 The embodiments of this application will be described in detail below.

[0023] like Figure 1 As shown, a scheduling method based on large model output length prediction includes: S100: Obtain a large model input request set, wherein the large model input request set includes multiple large model input requests within a time period; Specifically, this step involves collecting all large language model input requests received by the Model as a Service (MaaS) platform within a preset time period, forming a structured large model input request set. This provides foundational data support for subsequent output length prediction and scheduling priority determination. The large model input request set includes multiple large model input requests within a single time period. This time period can be flexibly set according to the platform's concurrent request volume, such as 100ms or 500ms, ensuring that the request set covers a sufficient number of requests for batch scheduling while avoiding request backlog and scheduling delays caused by excessively long time periods.

[0024] Each of the large model input requests may include a request identifier (used to uniquely distinguish each request), request text (such as "summarize a technical document" or "generate a product introduction"), request initiation information (terminal device identifier, application programming interface identifier), and request submission timestamp. Simultaneously, after receiving the request, the platform performs preliminary preprocessing on each input request, including format validation, invalid request filtering (such as blank requests or requests with incorrect formatting), and request deduplication, to ensure the validity and standardization of the input request set. After preprocessing, all valid requests are temporarily sorted by submission timestamp to form a large model input request set, which is stored in the platform's request cache queue, awaiting processing in subsequent steps.

[0025] S200: Determine the corresponding prediction output length based on each input request; Specifically, this step, for each large model input request obtained in S100, accurately predicts the number of characters (i.e., the predicted output length) of the output result after the request is inferred by the large language model through a preset prediction model, providing a core basis for subsequent scheduling priority division.

[0026] The prediction model relied upon in this step, such as length prediction and subsequent scheduling priority determination, is exemplarily the BERT model. BERT stands for Bidirectional Encoder Representations from Transformers, a pre-trained language model based on the Transformer architecture. This embodiment also builds and completes specialized training based on the BERT model. The specific model construction and training process is as follows: First, a multilayer perceptron (MLP) is extended on the pre-trained BERT-base model as a prediction head to output the predicted output length corresponding to the input request. Then, the constructed model is trained using a real dialogue dataset to ensure that the model can accurately capture the correlation between the input request and the output length. Finally, a two-stage fine-tuning strategy is used to optimize the model performance. In the first stage, all parameters of the BERT-base model and the prediction head are updated simultaneously to achieve overall model adaptation. In the second stage, the parameters of the BERT-base model are fixed, and only the parameters of the prediction head are optimized to improve prediction accuracy while reducing the risk of overfitting.

[0027] S300: Based on the predicted output length of each of the above, determine the scheduling priority of each of the above input requests; Specifically, based on the shortest job first strategy, the basic scheduling priority of each input request is determined: the large model input request set obtained in step S100 is sorted in ascending order of the predicted output length (number of words in the predicted output) of each request, forming a sorted request queue; among them, the shorter the predicted output length of the input request, the higher its basic scheduling priority, and it will be given priority to obtain computing resources to execute inference tasks, thereby reducing the risk of head-of-queue blocking, reducing the queuing latency of short requests, and improving the platform throughput. For example, a request with a predicted output length of 100 words has a higher basic priority than a request with a predicted output length of 500 words, and is placed at the front of the queue during sorting.

[0028] Secondly, a dynamic aging mechanism for waiting time is introduced to dynamically adjust the basic scheduling priority and ensure the fairness of long requests: a waiting timer is set for each input request, starting from when the request enters the request cache queue and completes the S200 prediction step, and the waiting time of each request is counted in real time; if the waiting time of a certain input request with a long prediction output exceeds the preset timeout threshold, the scheduling priority of the request is gradually increased until its priority is higher than that of short requests that have not timed out, ensuring that the long request can obtain computing resources in time to execute inference tasks and avoid being blocked for a long time.

[0029] Finally, by combining the basic scheduling priority with the dynamically adjusted priority, the final scheduling priority of each input request is determined. All input requests are then reordered according to the final scheduling priority, and the request cache queue is updated to provide a sorting basis for subsequent S400 scheduling control.

[0030] S400: Based on the scheduling priority, the output order of each input request in the large model input request set is scheduled and controlled.

[0031] Specifically, step S400 is the scheduling execution step, used to control the inference output order of all input requests in the large model input request set according to the final scheduling priority of each input request determined in S300, to reasonably allocate platform computing resources, and to achieve efficient and fair execution of requests. The specific implementation details are as follows: First, the platform server selects input requests to be executed from the request cache queue in descending order of final scheduling priority, prioritizing the highest priority requests for allocating computing resources and sending them to the underlying large language model inference engine for inference task execution. If multiple idle computing resources exist, a batch processing approach can be adopted to select multiple requests with the same or adjacent priorities, allocate computing resources in batches for parallel execution, and improve resource utilization efficiency and scheduling throughput. At the same time, the execution progress of each inference task is monitored in real time, and information such as task execution time and output result status is recorded.

[0032] In addition, during the scheduling and control process, the waiting time of each input request is updated in real time. If a new input request is submitted to the platform, steps S100-S300 will be executed repeatedly to obtain the predicted output length and scheduling priority of the new request and insert it into the corresponding position in the request cache queue to achieve dynamic scheduling. If an exception occurs during the execution of a request (such as inference failure or resource interruption), the request will be put back into the request cache queue and steps S200-S400 will be executed again to ensure that the request can be executed normally.

[0033] Finally, after the input request is executed by the inference engine, the inference output result is formatted according to the interface specification of the request initiator and fed back to the corresponding terminal device or application through a standardized interface. The scheduling execution log of the request (including prediction output length, scheduling priority, waiting time, execution time, etc.) is recorded for subsequent platform performance optimization, scheduling strategy adjustment and prediction model calibration.

[0034] In this embodiment, a scheduling method based on large model output length prediction includes: acquiring a large model input request set, wherein the large model input request set includes multiple large model input requests within a time period; determining the corresponding predicted output length based on each input request; determining the scheduling priority of each input request based on each predicted output length; and controlling the output order of each input request in the large model input request set based on the scheduling priority. This embodiment, by acquiring each input request in the large model input request set within a time period and determining the corresponding predicted output length based on each input request, provides an accurate and reliable basis for determining scheduling priorities, avoiding the blindness of priority division; by determining the scheduling priority of each input request based on the predicted output length, it achieves differentiated management and refined scheduling of input request scheduling, effectively avoiding the head-of-line blocking problem that easily occurs in simple scheduling strategies, while improving the utilization rate of platform computing resources and the overall efficiency of request processing, and ensuring the timeliness and stability of responses to various requests.

[0035] In some embodiments, in step S200, determining the corresponding prediction output length based on each input request includes: S201: Determine a pre-determined template corresponding to each of the input requests, wherein the pre-determined template has a corresponding statistical output length; Specifically, a pre-determined template is determined for each input request. The pre-determined template has a corresponding statistical output length. To improve template matching efficiency, a strategy combining rule matching and string matching is adopted for high-frequency structured requests received by the platform (such as fixed-format query requests, standardized instruction requests, etc.). It is preferred to use a radix tree to achieve longest prefix matching, so that the arriving input request can quickly hit the corresponding pre-determined template and obtain the template ID. The template ID is used to quickly associate the historical output length of the template obtained offline in advance, which greatly shortens the matching time.

[0036] For example, when the input request is "Help me write an essay about the Spring Festival", the system can quickly identify that the request belongs to the essay generation scenario in the text generation category by matching string prefixes and semantic rules. This will match the pre-built essay generation template, whose statistical output length is 500-700 words. Therefore, the initial predicted output length of the input request is 500-700 words. Similarly, if the input request is "Help me write a work summary", the system will match the work summary generation template, whose statistical output length is 1000-1100 words. Therefore, the initial predicted output length of the input request is 1000-1100 words.

[0037] S202: Based on the statistical output length of the pre-determined template and the semantics of the input request, determine the predicted output length corresponding to the input request.

[0038] Specifically, the core purpose of this step is to fine-tune and optimize the predicted output length based on the template statistical output length and the specific semantic differences of the input request. This avoids prediction bias caused by the generality of template statistics and ensures that the prediction results better match the actual needs of a single request. Specific implementation details and examples are as follows: First, obtain the statistical output length corresponding to the pre-determined template hit in step S201. At the same time, use the semantic analysis model to parse the core semantics of the current input request, such as whether the request contains explicit length instructions such as "brief", "detailed", "concise", "expanded", whether it contains more content points (the more points, the longer the output length usually needs to be), and whether there are specific format requirements (such as bullet points, paragraphs, etc., which indirectly affect the output length).

[0039] Example 1: Based on the essay generation template (ID: T001, statistical output length 500-700 words), if the input request is "Help me write an essay about the Spring Festival, requiring it to be concise and refined", the core semantic instruction "concise and refined" can be extracted through semantic analysis. Combined with the template's statistical output length range, it is finely adjusted downwards, and finally the predicted output length corresponding to this input request is determined to be 500-600 words, which not only conforms to the overall statistical pattern of the template, but also fits the specific semantic requirements of the request. If the input request is "Help me write an essay about the Spring Festival, requiring a detailed description of the customs of the Spring Festival and the scene of family reunion, divided into 3 paragraphs", semantic analysis can extract semantic features such as "detailed description", "2 core content points" and "divided into 3 paragraphs". It is determined that the request needs to be slightly longer than the basic length of the template statistics, and it is finely adjusted upwards. Finally, the predicted output length is determined to be 700-800 words.

[0040] Example 2: Based on the work summary template (ID: T002, statistical output length 800-1200 words), if the input request is "Help me write a monthly work summary, only including work content and simple reflection, no expansion required", semantic analysis extracts features such as "monthly", "only includes 2 core contents", and "no expansion required". Combined with the template's statistical length, the predicted output length is determined to be 800-900 words after fine-tuning. If the input request is "Help me write an annual work summary, detailing the year's work achievements, shortcomings, improvement plans, and next year's plans, with each part explained in points", semantic analysis can extract features such as "annual", "4 core contents", "points explained", and "detailed explanation". It is determined that the content of this request is more complex and the length needs to be fine-tuned upwards. Finally, the predicted output length is determined to be 1100-1300 words.

[0041] To further explain, if the input request does not contain explicit semantic instructions related to length, and the content complexity is consistent with the average level of the template's historical statistics, then the statistical output length of the template will be directly used as the predicted output length of the input request. For example, if the input request is "Help me write an argumentative essay about environmental protection", without specifying any additional length requirements, and semantic analysis shows that its content complexity is moderate, then the predicted output length will be directly determined to be the statistical range of 500-700 words for the essay generation template.

[0042] In this embodiment, the accuracy of the predicted output length of the input request is improved by determining a pre-determined template corresponding to the input request and combining the template's statistical output length with the request semantics. Specifically, based on the statistical output length of the pre-determined template, the prediction difficulty is significantly reduced and the prediction efficiency is improved. Fine-tuning by combining the input request semantics effectively avoids the prediction bias caused by a single template, ensuring that the prediction results match the actual needs of each request.

[0043] In some embodiments, in step S201, the pre-determining process of the pre-determined template includes: S2011: Obtain multiple historical input requests for the large model; Specifically, multiple historical input requests for the large model are acquired. These historical input requests need to cover various request scenarios that occur frequently in the daily operation of the platform to ensure that the pre-determined templates generated subsequently have broad adaptability. For example, various historical input requests sent by users within a certain period, such as "Help me write an essay about the Spring Festival", "Help me write a monthly work summary", "Help me write a short product promotion copy", and "Query the inference latency of GPT-4", are acquired. At the same time, the historical output content and historical output length corresponding to each historical input request are acquired simultaneously to provide data support for subsequent feature extraction and statistical analysis.

[0044] S2012: Extract key features for each of the historical input requests; Specifically, key features are extracted for each historical input request. The extracted key features must be able to distinguish different types of requests. The core features include request type features (such as text generation, query consultation, etc.), style / format features (such as essays, work summaries, short copy, fixed format queries, etc.), and core demand features (such as the theme of the essay, the query indicators, the application scenario of the copy, etc.). For example, for the historical input request "Help me write an essay about the Spring Festival", the extracted key features are "text generation type, essay style, and Spring Festival theme".

[0045] S2013: Based on the extracted key features, classify the historical input requests, and summarize multiple statistical features of each type of historical input request into a pre-determined template corresponding to that type; The statistical features include at least the statistical output length; the statistical output length is obtained by statistically analyzing the historical output lengths corresponding to at least a portion of the historical input requests in this class.

[0046] Specifically, based on the extracted key features, the historical input requests are classified. Clustering algorithms (such as K-means clustering) can be used to group historical input requests with similar key features and consistent request types into one category. Then, multiple statistical features of each category of historical input requests are summarized into a pre-determined template corresponding to that category. The statistical features include at least the statistical output length, which is obtained by statistically analyzing the historical output lengths corresponding to at least a portion of the historical input requests in that category. Specifically, offline statistics are performed on the historical output lengths in each category to calculate parameters such as the length mean, standard deviation, and quantiles. The length mean characterizes the overall level of the historical output length for that category, the standard deviation characterizes the dispersion of the historical output length, and the quantiles characterize the distribution range of the historical output length. If the standard deviation of the historical output length for a certain category is large, it indicates that the historical output length of that category fluctuates significantly and has poor stability. In this case, outlier data with large length differences can be removed to weaken the interference of highly dispersed samples, ultimately determining the statistical output length range corresponding to that category and associating it with the pre-determined template. Meanwhile, for cases where the standard deviation of the template statistical output length is large and the reference weight is low, the weight of subsequent semantic analysis will be appropriately increased. This, combined with semantic features, will allow for a more precise adjustment of the length range, avoiding prediction bias. Through this approach, the statistical regularity of the template is combined with the semantic requirements of individual requests, ensuring both prediction efficiency and improving the accuracy of the predicted output length.

[0047] For example, all historical input requests with extracted key features of "text generation type and essay style" (such as "Help me write an essay about the Spring Festival" or "Help me write an argumentative essay about environmental protection") are grouped into one category, and an "essay generation template" is generated. The historical output lengths of all historical input requests in this category are statistically analyzed, and parameters such as the mean length, standard deviation, and quantiles are calculated. The mean length characterizes the overall level of historical output length for this category, the standard deviation characterizes the dispersion of historical output lengths, and the quantiles characterize the distribution range of historical output lengths. If the standard deviation of the historical output length for this category is small, indicating a stable length distribution with minimal fluctuation, the statistical output length range of the essay generation template is directly determined based on the statistical results. If the statistically obtained historical output length mean is 600 characters, the standard deviation is 50 characters, and the length distribution is concentrated between 500 and 700 characters, then the statistical output length of the essay generation template is determined to be between 500 and 700 characters.

[0048] Similarly, all historical input requests for "text generation type, work summary style" are grouped into one category to generate a "work summary generation template". The historical output length is calculated to be in the range of 800-1200 words, which is the statistical output length of the template.

[0049] In this embodiment, by collecting historical input requests from a large model, extracting and classifying key features, and forming pre-determined templates adapted to different request types, and by statistically analyzing the historical output lengths of similar models to obtain the statistical output lengths corresponding to the templates, historical data can be fully utilized to form stable and reliable prior references, making subsequent predicted output lengths more reasonable and credible, while improving the efficiency and stability of the prediction process, and providing solid data support for differentiated scheduling strategies.

[0050] In some embodiments, in step S201, determining the pre-determined template corresponding to each input request includes: S2014: Match the request instruction of each input request with the key features corresponding to the multiple predetermined templates; Specifically, the request instruction of each input request is matched with the key features corresponding to multiple pre-determined templates. The key features corresponding to the pre-determined templates are a summary of the core features of historical input requests belonging to that template category, including request type features (such as text generation, query / consultation), style / format features (such as essays, work summaries, short copy), and instruction prefix features (such as "help me write an article" or "query"). The matching process combines rule-based matching with string prefix matching to quickly compare the fit between the input request instruction and the key features of each template, eliminating the need for deep semantic analysis and balancing matching efficiency and accuracy.

[0051] S2015: In response to determining that the request instruction of the input request matches the key feature corresponding to a predetermined template, the predetermined template is determined as the predetermined template of the input request; Specifically, if the pre-determined templates include an essay generation template (key features: text generation type, essay style, instruction prefix "help me write one") and a work summary generation template (key features: text generation type, work summary style, instruction prefix "help me write one"), when the input request is "help me write an argumentative essay about environmental protection," the "help me write one" in the request instruction matches the instruction prefix feature of the essay generation template, and "argumentative essay" matches the essay style feature. The key features of the two are determined to match, and thus the essay generation template is identified as the pre-determined template corresponding to the input request. Similarly, when the input request is "help me write a quarterly work summary," its key features match the work summary generation template, thus the work summary generation template is identified as the corresponding pre-determined template.

[0052] S2016: In response to the fact that the request instruction of the input request does not match the key features of any of the predetermined templates, a default template is determined as the predetermined template of the input request, wherein the default template has a predetermined statistical output length.

[0053] Specifically, the default template is a pre-configured fallback template used to cover low-frequency, niche requests, requests with unrecorded wording, or requests with unclear key features. This avoids problems such as uncertain predicted output length and interrupted request scheduling due to the lack of a matching template. For example, if the input request is "Help me write an introductory guide to a niche hobby," and the key features of this request (niche hobby, introductory guide) do not match the key features of pre-determined templates such as essay generation, work summary, and product copywriting in the system, then the default template is determined as the pre-determined template for this input request. This default template is pre-configured with a general statistical output length, such as 300-500 words, which can meet the basic length prediction requirements of such unmatched requests, ensuring that all input requests can smoothly enter the prediction process of the subsequent S202 step, and guaranteeing the stability and compatibility of the system operation.

[0054] In this embodiment, by matching the input request with the key features of the pre-determined template, accurate template allocation for the input request is achieved. At the same time, by using the fallback mechanism of the default template, it is ensured that all input requests can be matched with a specific template and obtain statistical output length support, which effectively improves the accuracy and efficiency of template matching, avoids process interruption caused by no matching template, ensures the smooth progress of subsequent output length prediction and request scheduling process, and further enhances the robustness and compatibility of the system.

[0055] In some embodiments, in step S202, determining the predicted output length corresponding to the input request based on the statistical output length of the pre-determined template and the semantics of the input request includes: S2021: Perform semantic parsing on the input request and extract the implicit semantic features corresponding to the input request, wherein the implicit semantic features are implicit features that affect the output length; Specifically, the input request is semantically parsed to extract its corresponding latent semantic features. These latent semantic features are those that affect the output length and cannot be directly read from the request command; they require semantic parsing to uncover deeper correlations. The latent semantic features are then transformed into fixed-dimensional semantic feature vectors, realizing the conversion from abstract descriptions to concrete, computable vectors, thus forming latent semantic feature vectors. This provides an operational foundation for subsequent feature fusion and length prediction.

[0056] For example, if the input request is "Help me write a fourth-grade essay about the Spring Festival," without explicitly mentioning the word count, semantic parsing can extract implicit semantic features such as "suitable for short essays at the fourth-grade level, not too many words, no need for complex rhetoric or in-depth discussion," which are then encoded and aggregated into an implicit semantic feature vector. If the input request is "Help me write a paper about Spring Festival culture," again without specifying the word count, implicit semantic features such as "suitable for longer essays, requires literature support and in-depth analysis, and has a higher word count requirement" can be extracted, which are also converted into an implicit semantic feature vector. The implicit semantic features of the two are significantly different, and the corresponding converted implicit semantic feature vectors are also different, providing targeted support for subsequent accurate length prediction and feature fusion.

[0057] S2022: Based on the statistical output length of the pre-determined template corresponding to the input request and the explicit semantic features corresponding to the input request, determine the first output length corresponding to the input request, wherein the explicit semantic features are explicit features that affect the output length; Specifically, based on the statistical output length of the pre-determined template corresponding to the input request and the explicit semantic features corresponding to the input request, the first output length corresponding to the input request is determined. The explicit semantic features are those that affect the output length and can be directly extracted from the request instruction without deep mining. They mainly include length control words and other features directly related to the output length. Corresponding to the explicit structural path, the processing steps in this step are as follows: First, the ID of the pre-determined template corresponding to the input request is mapped to a learnable template embedding vector. Then, length control word features are extracted. In the offline stage, a set of control words highly correlated with the output length is selected based on relevance criteria such as mutual information. In the online stage, multi-hot features are constructed based on the occurrence of length control words in the input request and compressed through a fully connected layer. Simultaneously, the extracted explicit semantic features are converted into explicit semantic feature vectors. Then, the template embedding vector corresponding to the pre-determined template ID is read, and the explicit semantic feature vector and the template embedding vector are concatenated to finally convert it into a structural feature vector. Among them, explicit semantic features are the core components of the structural feature vector, template statistical prior features are used to inject macroscopic statistical laws, and the structural feature vector formed by the combination of the two and other related features has the same dimension as the semantic feature vector output in step S2021, which is suitable for subsequent splicing and fusion operations.

[0058] For example, if the input request is "Help me write an essay about the Spring Festival, which should be concise and refined", the corresponding essay generation template (statistical output length 500-700 words) extracts the length control word "concise and refined" as an explicit semantic feature, transforms it into an explicit semantic feature vector, and combines it with the template embedding vector mapped by the essay generation template ID to transform it into a structural feature vector, determining the first output length as 500-600 words; if the input request is "Help me write a quarterly work summary, including work content and a brief reflection", the corresponding work summary generation template (statistical output length 800-1200 words) extracts relevant length control words as explicit features, completes feature transformation and concatenation to obtain a structural feature vector, and determines the first output length as 800-900 words.

[0059] S2023: Based on the implicit semantic features and the first output length, determine the predicted output length corresponding to the input request.

[0060] Specifically, the semantic feature vector output from the implicit semantic path and the structural feature vector output from the explicit structural path are first concatenated and fused to form a unified fused feature representation. The concatenation method directly combines the feature dimensions, superimposing the dimensions of the semantic and structural feature vectors to retain all information from both paths (including implicit deep semantic information, explicit features, and template statistical patterns), ensuring that the fused feature can fully support subsequent length prediction. After fusion, the unified fused feature representation is input into a multilayer perceptron (MLP) for nonlinear mapping, precisely fine-tuning the first output length obtained from S2022 to finally obtain a predicted output length that fits the actual requirements of the request.

[0061] It should be noted that this step needs to take into account various special scenarios to improve the flexibility and accuracy of prediction: If no implicit semantic features affecting the output length are extracted after semantic parsing of the input request, then implicit semantic information will not be considered in the subsequent prediction process, and prediction will only be based on explicit semantic features and template statistical output length; If an explicit and definite word count requirement is extracted from the input request (such as "Help me write an 800-word fourth-grade Spring Festival composition" or "Help me write a 3000-word Spring Festival culture paper"), then the explicit word count will be used directly as the predicted output length, without referring to the pre-determined template statistical output length and implicit semantic features, prioritizing the user's explicit needs; If no explicit semantic features are extracted from the input request (no explicit word count, length control words, etc.) and no implicit semantic features are extracted, then the statistical output length of the pre-determined template corresponding to the input request will be directly referred to as the final predicted output length, without additional fine-tuning.

[0062] In this embodiment, by distinguishing between explicit and implicit semantic features of the input request, the first output length is determined by combining the statistical output length of the pre-determined template with the explicit semantic features. Then, the implicit semantic features are combined for precise fine-tuning, effectively integrating explicit needs, implicit potential needs, and template statistical patterns. Explicit semantic features are length-related needs explicitly mentioned by the user in the request instruction and directly extractable (such as specifying the number of characters or length control words). These have the highest priority and do not require deep parsing. Implicit semantic features are deeper needs not explicitly stated by the user but that affect the output length (such as potential length requirements corresponding to a grade level or text style). These need to be mined through semantic parsing and can fill prediction gaps when explicit needs are missing. The template is first combined with explicit semantic features to determine the first output length because the statistical output length of the pre-determined template is based on general patterns obtained from similar historical requests, serving as the baseline for length prediction. Explicit semantic features represent the user's current explicit needs. Combining the two first ensures that the prediction results conform to historical statistical patterns and prioritizes responding to the user's explicit requirements, avoiding a disconnect between the template's general statistics and the user's actual explicit needs. Based on this, further fine-tuning is done by combining implicit semantic features to make up for the deep differences not covered by explicit requirements. Finally, an output length prediction mechanism that takes into account explicit requirements, implicit potential requirements and template statistical regularities is constructed. This effectively makes up for the limitations of single template statistics or single semantic extraction, improves the accuracy and flexibility of output length prediction, ensures that all kinds of input requests can get prediction results that fit their actual needs, and guarantees the reliability and adaptability of the output length prediction process.

[0063] In some embodiments, in step S2021, semantic parsing of the input request and extraction of the implicit semantic features corresponding to the input request includes: S2021a: Perform word segmentation and encoding on the text of the input request to obtain the word sequence corresponding to the input request; Specifically, the input request text is segmented and encoded to obtain the corresponding word sequence. First, a word segmentation tool adapted to Chinese text (such as jieba) is used to accurately segment the natural language text of the input request, splitting it into lexical units related to the output length, including task instructions, topics, styles, grade levels, and constraints, while removing meaningless interjections, auxiliary words, and other redundant information. Then, each segmented lexical unit is input into a pre-trained language encoding model (such as a lightweight version of BERT-base), converting each lexical unit into a fixed-dimensional word vector. Finally, all word vectors are concatenated according to the semantic order of the original text to form a complete word sequence, realizing the transformation of the input request from natural language text to a machine-recognizable and computable word sequence, laying the foundation for subsequent key information extraction and implicit semantic mining. For example, the input request "Help me write a fourth-grade composition about the Spring Festival, requiring simple and easy-to-understand language" is segmented and encoded to obtain a word sequence containing core vocabulary units such as "help me write", "fourth grade", "Spring Festival", "composition" and "simple language"; after segmenting and encoding the input request "Help me write an academic paper about Spring Festival culture, combining relevant literature for analysis" to obtain a word sequence containing vocabulary units such as "help me write", "Spring Festival culture", "academic paper" and "literature analysis".

[0064] S2021b: Based on a preset truncation threshold, target words are truncated from the head and tail of the word sequence, and the truncated target words are concatenated to obtain a concatenated word sequence. Specifically, considering that the input length of the surrogate prediction model has an upper limit, directly inputting a complete word sequence may lead to increased inference overhead and decreased model efficiency. In practice, it has been found that the key information that determines the output length is distributed in a "concentrated at both ends" pattern in the word sequence—the head mainly contains core control information such as task instructions, format constraints, and style / grade requirements, which are directly related to the basic range of the output length; the tail mainly contains auxiliary information such as topic, context, and supplementary requirements, which are used to refine the adaptability of the output length. Therefore, this step adopts a head-and-tail retention strategy to replace the input of a complete sequence. The preset truncation threshold is reasonably set based on the upper limit of the input length of the proxy prediction model and the distribution of the length of historical request lexical units. This ensures that the truncated target lexical units can control the overall length and reduce inference overhead, while also fully preserving the instruction constraint information and contextual auxiliary information that are strongly related to the output length. During truncation, instruction and constraint target lexical units of preset length are truncated from the head of the lexical unit sequence, and topic and context target lexical units of preset length are truncated from the tail. The two truncated target lexical units are then concatenated in the order of "head + tail" to form a concatenated lexical unit sequence, and redundant content with low correlation to the output length is removed. For example, for the word sequence of the above-mentioned fourth-grade composition, the preset truncation threshold is set to 3 words at the beginning and 2 words at the end. The words at the beginning are truncated to be instruction-constraining words such as "help me write", "fourth grade" and "composition", and the words at the end are truncated to be theme-supporting words such as "Spring Festival" and "simple language". After splicing, the corresponding spliced ​​word sequence is obtained. For the word sequence of academic papers on Spring Festival culture, the words at the beginning are truncated to be "help me write", "academic paper" and "literature analysis", and the words at the end are truncated to be "Spring Festival culture" and "related literature". The splicing operation is completed, which controls the length of the sequence and retains the key information.

[0065] S2021c: Based on the concatenated word sequence, determine the implicit semantic features corresponding to the input request.

[0066] Specifically, the concatenated word sequence is input into the pre-trained language model encoder to obtain the latent state representations at each position in the concatenated word sequence. Then, a positional decay-based weighted aggregation method is used to give higher attention to instruction constraint words in the input header, more sensitively capturing the influence of task intent, grade level, and text style on the output length. The latent state representations at each position are then aggregated to finally extract the latent semantic features corresponding to the input request. It is important to clarify that there is a direct correspondence between latent semantic features and semantic feature vectors: latent semantic features are abstract, deep semantic descriptions. After the above aggregation process, they are further transformed into fixed-dimensional latent semantic feature vectors, realizing the transformation from abstract features to computable vectors, providing an operational foundation for subsequent feature fusion and length prediction.

[0067] Continuing with the previous example, from the spliced ​​word sequence of fourth-grade compositions, we extract the implicit semantic features of "suitable for short essays in fourth grade, not too many words, and no need for complex rhetoric" and transform them into implicit semantic feature vectors. From the spliced ​​word sequence of Spring Festival culture papers, we extract the implicit semantic features of "the paper style is suitable for long articles, requires in-depth analysis, and has a high word count requirement" and the corresponding implicit semantic feature vectors. The differences between the two will be directly reflected in the semantic feature vectors, providing targeted support for subsequent accurate predictions.

[0068] In this embodiment, the precise and efficient extraction of implicit semantic features of the input request is achieved through step-by-step processing from S2021a to S2021c. Word segmentation and encoding transform the natural language text of the input request into a computable sequence of tokens, laying a solid foundation for subsequent feature extraction. The truncation strategy effectively preserves key information strongly correlated with the output length in the token sequence while controlling the inference overhead of the proxy prediction model, avoiding interference from redundant information. Extracting implicit semantic features based on the concatenated token sequence ensures that the extraction results match the deep semantic requirements of the input request, while simultaneously realizing the transformation of implicit semantic features into semantic feature vectors. This guarantees the accuracy, relevance, and reliability of the implicit semantic feature extraction process, balancing extraction efficiency and feature quality.

[0069] In some embodiments, in step S300, determining the scheduling priority of each input request based on each predicted output length includes: S301: Arrange all the input requests in order of their corresponding predicted output lengths to form a request queue; Specifically, all input requests are arranged in order of their corresponding predicted output lengths to form a request queue. Specifically, after all input requests have completed the predicted output length calculation in step S202, the predicted output lengths of each input request are collected and arranged in ascending order, with input requests with shorter predicted output lengths placed at the front of the queue and those with longer predicted output lengths placed at the back. If two or more input requests have the same predicted output length, they are arranged according to the order in which they arrive at the server (i.e., first-come, first-served), ensuring the rationality and orderliness of the queue arrangement. The core purpose of this sorting method is to implement a short-task-first strategy. Referring to the input request examples mentioned earlier, fourth-grade essays (predicted output length 300-400 words), beginner's guides to niche hobbies (predicted output length 300-400 words), Spring Festival essays (predicted output length 500-550 words), quarterly work summaries (predicted output length 800-900 words), and Spring Festival cultural essays (predicted output length 3000-5000 words) are arranged in ascending order to form a request queue. This effectively avoids head-of-queue congestion caused by long requests occupying scheduling resources for extended periods, laying the foundation for efficient scheduling later. Simultaneously, the sorting process synchronously associates the identification information and predicted output length of each input request, facilitating subsequent priority determination and dynamic adjustment.

[0070] S302: Based on the order of the request queue, determine the scheduling priority corresponding to the large model input request set; Specifically, the scheduling priority of the large model input request set is determined based on the order of the request queue. The scheduling priority is directly related to the order of the request queue; input requests at the front of the queue have a higher scheduling priority than those at the back. That is, the earlier a request is in the queue, the higher its scheduling priority, and it will be prioritized for entry into the next batch of large model inference execution. Based on the previously arranged request queue, requests with the shortest predicted output length are at the front of the queue and have the highest scheduling priority, thus being scheduled first. Conversely, requests with the longest predicted output length are at the back of the queue and have the lowest scheduling priority, meaning they will be scheduled after the shortest requests at the front are processed. This priority determination method prioritizes requests with shorter predicted output lengths, quickly releases scheduling resources, reduces the average queuing time for all subsequent requests, and improves the overall throughput efficiency of the large model server, aligning with the core requirements of the reference approach: "reducing the risk of head-of-queue blocking and improving system response speed."

[0071] S303: In response to determining that there is a long input request in the request queue whose waiting time exceeds a preset time, the scheduling priority of the long input request is increased, wherein the long input request is an input request whose predicted output length is greater than a preset value.

[0072] Specifically, to prevent long input requests from being suppressed in the queue for extended periods due to the short job priority strategy, resulting in unserved requests, a dynamic aging-up mechanism for waiting time is introduced, balancing scheduling efficiency and fairness. The preset value is reasonably set based on the large model's inference capability, historical request length distribution, and business scenario requirements. For example, considering the request types mentioned earlier, a preset value of 1000 characters would mean that a Spring Festival cultural paper with a predicted output length of 3000-5000 characters would be considered a long input request. The preset duration is set based on system performance and service latency requirements, used to determine whether a long input request is in a "long-term waiting" state. For example, a preset duration of 20 seconds is set. When the system detects a long input request entering the request queue, it begins accumulating its waiting time. If the accumulated waiting time exceeds the preset duration, the scheduling priority of the long input request is automatically increased. The increase can be preset, ensuring it is scheduled before lower-priority short input requests. For example, if a Spring Festival cultural paper (a long input request) waits in the queue for 20 seconds, the system will automatically increase its scheduling priority, placing it before at least some short input requests to ensure that it can be scheduled and processed in a timely manner.

[0073] In this embodiment, through the step-by-step processing of S301 to S303, the scheduling priority based on the predicted output length is accurately determined and dynamically adjusted. S301 sorts the input requests according to the predicted output length to form a request queue, providing an orderly and reasonable basis for priority determination; S302 determines the scheduling priority based on the queue arrangement order, implementing the shortest job first logic, effectively avoiding head-of-queue blocking caused by long requests occupying scheduling resources for a long time; S303, through the mechanism of raising the priority of long input requests waiting for timeout, avoids the phenomenon that long input requests cannot be processed for a long time due to the shortest job first strategy. Overall, the reasonable allocation and dynamic adaptation of scheduling priorities are realized, ensuring that each input request can be reasonably scheduled.

[0074] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0075] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a scheduling device based on large model output length prediction.

[0077] refer to Figure 2 The scheduling device includes: The acquisition module 501 is configured to: acquire a large model input request set, wherein the large model input request set includes multiple large model input requests within a time period; The first control module 502 is configured to: determine the corresponding predicted output length based on each input request; The first control module 503 is configured to: determine the scheduling priority of each of the input requests based on the predicted output lengths; The first control module 504 is configured to: perform scheduling control on the output order of each input request in the large model input request set based on the scheduling priority.

[0078] Furthermore, the first control module 502 is also configured as follows: A predetermined template is determined for each of the input requests, wherein the predetermined template has a corresponding statistical output length; Based on the statistical output length of the pre-determined template and the semantics of the input request, the predicted output length corresponding to the input request is determined.

[0079] Furthermore, the first control module 502 is also configured such that the pre-determining process of the pre-determined template includes: Obtain multiple historical input requests for the large model; Key features are extracted for each of the aforementioned historical input requests; Based on the extracted key features, the historical input requests are classified, and multiple statistical features of each type of historical input request are summarized into a pre-determined template corresponding to that type. The statistical features include at least the statistical output length; the statistical output length is obtained by statistically analyzing the historical output lengths corresponding to at least a portion of the historical input requests in this class.

[0080] Furthermore, the first control module 502 is also configured as follows: The request instruction of each input request is matched with the key features corresponding to the multiple predetermined templates; In response to determining that the request instruction of the input request matches the key feature corresponding to a predetermined template, the predetermined template is determined as the predetermined template of the input request; If the request instruction of the input request does not match the key features of any of the predetermined templates, then the default template is determined as the predetermined template of the input request, wherein the default template has a predetermined statistical output length.

[0081] Furthermore, the first control module 502 is also configured as follows: The input request is semantically parsed to extract the implicit semantic features corresponding to the input request, wherein the implicit semantic features are implicit features that affect the output length; Based on the statistical output length of the pre-determined template corresponding to the input request and the explicit semantic features corresponding to the input request, the first output length corresponding to the input request is determined, wherein the explicit semantic features are explicit features that affect the output length; Based on the implicit semantic features and the first output length, the predicted output length corresponding to the input request is determined.

[0082] Furthermore, the first control module 502 is also configured as follows: The text of the input request is segmented and encoded to obtain the word sequence corresponding to the input request; Based on a preset truncation threshold, target words are truncated from the head and tail of the word sequence, and the truncated target words are concatenated to obtain a concatenated word sequence. Based on the concatenated word sequence, the implicit semantic features corresponding to the input request are determined.

[0083] Furthermore, the first control module 503 is also configured as follows: Arrange all the input requests in order of their corresponding predicted output lengths to form a request queue; Based on the order of the request queue, the scheduling priority corresponding to the large model input request set is determined.

[0084] Furthermore, the first control module 503 is also configured as follows: In response to determining that there is a long input request in the request queue whose waiting time exceeds a preset time, the scheduling priority of the long input request is increased, wherein the long input request is an input request whose predicted output length is greater than a preset value.

[0085] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0086] The apparatus described above is used to implement the corresponding scheduling method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0087] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scheduling method described in any of the above embodiments.

[0088] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0089] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0090] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0091] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0092] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0093] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0094] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0095] The electronic devices described above are used to implement the corresponding scheduling methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0096] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the scheduling method as described in any of the above embodiments.

[0097] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0098] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the scheduling method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0099] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0100] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0101] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0102] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0103] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0104] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0105] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0106] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A scheduling method based on large model output length prediction, characterized in that, include: Obtain a large model input request set, wherein the large model input request set includes multiple large model input requests within a certain time period; Based on each input request, determine the corresponding predicted output length; Based on the predicted output length, the scheduling priority of each input request is determined. Based on the scheduling priority, the output order of each input request in the large model input request set is scheduled and controlled.

2. The scheduling method according to claim 1, characterized in that, Determining the corresponding prediction output length based on each input request includes: A predetermined template is determined for each of the input requests, wherein the predetermined template has a corresponding statistical output length; Based on the statistical output length of the pre-determined template and the semantics of the input request, the predicted output length corresponding to the input request is determined.

3. The scheduling method according to claim 2, characterized in that, The process of determining the predetermined template includes: Obtain multiple historical input requests for the large model; Key features are extracted for each of the aforementioned historical input requests; Based on the extracted key features, the historical input requests are classified, and multiple statistical features of each type of historical input request are summarized into a pre-determined template corresponding to that type. The statistical features include at least the statistical output length; the statistical output length is obtained by statistically analyzing the historical output lengths corresponding to at least a portion of the historical input requests in this class.

4. The scheduling method according to claim 3, characterized in that, The determination of the pre-determined template corresponding to each input request includes: The request instruction of each input request is matched with the key features corresponding to the multiple predetermined templates; In response to determining that the request instruction of the input request matches the key feature corresponding to a predetermined template, the predetermined template is determined as the predetermined template of the input request; If the request instruction of the input request does not match the key features of any of the predetermined templates, then the default template is determined as the predetermined template of the input request, wherein the default template has a predetermined statistical output length.

5. The scheduling method according to claim 2, characterized in that, The step of determining the predicted output length corresponding to the input request based on the statistical output length of the pre-determined template and the semantics of the input request includes: The input request is semantically parsed to extract the implicit semantic features corresponding to the input request, wherein the implicit semantic features are implicit features that affect the output length; Based on the statistical output length of the pre-determined template corresponding to the input request and the explicit semantic features corresponding to the input request, the first output length corresponding to the input request is determined, wherein the explicit semantic features are explicit features that affect the output length; Based on the implicit semantic features and the first output length, the predicted output length corresponding to the input request is determined.

6. The scheduling method according to claim 5, characterized in that, Semantic parsing of the input request and extraction of its corresponding latent semantic features include: The text of the input request is segmented and encoded to obtain the word sequence corresponding to the input request; Based on a preset truncation threshold, target words are truncated from the head and tail of the word sequence, and the truncated target words are concatenated to obtain a concatenated word sequence. Based on the concatenated word sequence, the implicit semantic features corresponding to the input request are determined.

7. The scheduling method according to claim 1, characterized in that, The step of determining the scheduling priority of each input request based on the predicted output length includes: Arrange all the input requests in order of their corresponding predicted output lengths to form a request queue; Based on the order of the request queue, the scheduling priority corresponding to the large model input request set is determined.

8. The scheduling method according to claim 7, characterized in that, The step of determining the scheduling priority of each input request based on the predicted output length further includes: In response to determining that there is a long input request in the request queue whose waiting time exceeds a preset time, the scheduling priority of the long input request is increased, wherein the long input request is an input request whose predicted output length is greater than a preset value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

Images