Model determination method and device and electronic equipment

Abstract

The invention relates to a model determination method and apparatus, and an electronic device. The method comprises the steps of determining a bandwidth parameter of a deployment node where a model in a first model set is located; wherein the model guides to generate a query result based on the prompt information; in response to a user query request, inputting a query text corresponding to the user query request into the routing model to obtain a prediction accuracy rate and a prediction cost of the model for the user query request; and determining a target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a model determination method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, large language model services have been widely applied in multiple fields. In order to improve the model inference efficiency and reduce the running cost, a model routing mechanism is usually adopted to dynamically select among multiple candidate models. However, how to improve the accuracy and resource utilization of the model in the model routing process is a problem that needs to be solved at present. SUMMARY

[0003] The present application provides a model determination method, device, electronic equipment and computer readable storage medium, which can improve the model accuracy and resource utilization of model routing.

[0004] In a first aspect, the embodiments of the present application provide a model determination method, which comprises: determining the bandwidth parameter of the deployment node where the model in the first model set is located; wherein the model generates a query result based on prompt information guidance; in response to a user query request, inputting the query text corresponding to the user query request into the routing model to obtain the prediction accuracy and prediction cost of the model for the user query request; determining the target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost.

[0005] In a second aspect, the embodiments of the present application provide a model determination device, which comprises: a first determination module configured to determine the bandwidth parameter of the deployment node where the model in the first model set is located; wherein the model generates a query result based on prompt information guidance; a second determination module configured to, in response to a user query request, input the query text corresponding to the user query request into the routing model to obtain the prediction accuracy and prediction cost of the model for the user query request; a third determination module configured to determine the target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost.

[0006] In a third aspect, the embodiments of the present application provide an electronic equipment comprising a processor and a memory, wherein the memory stores a computer program capable of running on the processor, and the processor implements the method of the first aspect when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method in the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions. The computer program or instructions are executed by a processor to implement the method in the first aspect.

[0009] The embodiments of the present application provide a model determination method and device, an electronic device and a computer readable storage medium. The bandwidth parameter of a deployment node where a model in a first model set is located is determined. The model generates a query result based on prompt information. In response to a user query request, a query text corresponding to the user query request is input into a routing model to obtain a prediction accuracy and a prediction cost of the model for the user query request. Based on the bandwidth parameter, the prediction accuracy and the prediction cost, a target model for processing the user query request is determined from the first model set. In this way, on the one hand, by introducing the bandwidth parameter as a routing decision basis, resource congestion or service delay problems caused by insufficient model bandwidth can be avoided, and the stability of routing model decision can be improved. On the other hand, by adding a bandwidth constraint condition on the basis of considering the prediction accuracy and the calling cost, the routing decision is closer to the actual deployment environment, so that the resource utilization rate is optimized while the service quality is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 Flowchart of a model determination method provided by an embodiment of the present application Figure 1 ; Figure 2 Flowchart of a model determination method provided by an embodiment of the present application Figure 2 ; Figure 3 Existing model routing framework diagram of a model determination method provided by an embodiment of the present application; Figure 4 Structural diagram of a model determination device provided by an embodiment of the present application; Figure 5 Hardware entity diagram of an electronic device provided by an embodiment of the present application.

[0011] It should be noted that the above-mentioned "first" and "second" are only used to distinguish different schemes, and do not represent the advantages or disadvantages of the schemes or the priority in the implementation process. DETAILED DESCRIPTION

[0012] The above and other advantages and features of the application will become apparent from the following description of the preferred embodiments, when taken in conjunction with the accompanying drawings. The application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these preferred embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the application to those skilled in the art.

[0013] It should be noted that the drawings provided in the following embodiments are only schematic and are intended to provide the basic understanding of the application. In the drawings, the shape, thickness and size of the components may be exaggerated for the purpose of explanation and are not necessarily to scale. The same reference numbers in different drawings represent the same or similar components.

[0014] In the following description, "some embodiments" refer to a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets as each other and can be combined if not in conflict.

[0015] In the following description, the terms "first", "second", "third", etc. are only used to distinguish similar objects and do not represent a specific order or sequence for the objects. It is understood that the "first", "second", "third", etc. can be interchanged in a specific order or sequence as allowed, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.

[0016] In the present embodiment, the term "and / or" is only used to describe the association relationship between the associated objects, which means that there can be three relationships, for example, object A and / or object B, which means that there can be three cases: object A exists alone, object A and object B exist together, and object B exists alone.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification is for the purpose of describing the embodiments of the application only and is not intended to limit the application.

[0018] At present, with the continuous development of artificial intelligence technology, model services have been widely applied in many fields. In order to improve the model inference efficiency and reduce the running cost, a model routing mechanism is usually used to dynamically select among multiple candidate models.

[0019] In the related art, the model routing framework mainly makes decisions based on the trade-off between the model prediction quality and the calling cost, without considering the real-time bandwidth constraints and system load state when deploying the model into the routing logic. For example, some solutions evaluate the performance indicators of each candidate model through preset rules or machine learning models, and decide the routing path of the request accordingly. However, such methods ignore the dynamic changes of network bandwidth in the actual deployment environment and their impact on the concurrent request processing capability.

[0020] The main defects of the related art are: (1) When the concurrent invocation quantity of high-priority models suddenly increases, bandwidth resources are easily exhausted, causing subsequent request blocking, and frequent bandwidth expansion processes need to be started, increasing operation and maintenance costs and service delays; (2) Low-load but low-accuracy models may be preferentially selected, while high-accuracy models are excluded from the routing range due to short-term load peaks, resulting in a decrease in the quality of service (QoS); (3) The routing decision is decoupled from the model deployment state, making it difficult to dynamically adjust the available model pool according to the bandwidth idle condition, and there is a risk of resource mismatch and stability.

[0021] To solve the above problems, the embodiment of the present application provides a model determination method, which determines the bandwidth parameter of the deployment node where the model in the first model set is located; wherein the model generates a query result based on prompt information; in response to a user query request, input the query text corresponding to the user query request into the routing model to obtain the prediction accuracy and prediction cost of the model for the user query request; based on the bandwidth parameter, the prediction accuracy and the prediction cost, determine the target model for processing the user query request from the first model set. In this way, on the one hand, by introducing the bandwidth parameter as the basis for routing decision, the problem of resource blocking or service delay caused by insufficient model bandwidth can be avoided, and the stability of the routing model decision can be improved; on the other hand, by adding the bandwidth constraint condition on the basis of considering the prediction accuracy and the calling cost, the routing decision is closer to the actual deployment environment, so as to optimize the resource utilization rate while ensuring the quality of service. In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application.

[0022] It should be noted that the model determination method provided by each embodiment of the present application can be executed by a routing decision system, which includes but is not limited to a server, a terminal device, a computing node, or a service module deployed on a cloud platform. That is, the model determination method of each embodiment of the present application can be implemented by a server or a terminal device as an execution subject, can also be cooperatively executed by multiple computing nodes, and can also be processed by a service module under a distributed architecture.

[0023] Figure 1 A flowchart of a model determination method provided for an embodiment of the present application Figure 1 As shown in Figure 1 , it can include S101 to S103, wherein: S101, determining the bandwidth parameter of the deployment node where the model in the first model set is located.

[0024] Among them, the model generates query results based on prompt information, and the prompt information is generated based on user query requests.

[0025] It can be understood that the first model set refers to a group of candidate models that generate query results based on prompt information, which can have different performance indicators and resource consumption characteristics to meet different types of query requirements. For example, the first model set can contain multiple large language models (Large Language Model, LLM), lightweight inference models, etc. These large language models and lightweight inference models are deployed on different computing nodes and have different bandwidth usage capabilities.

[0026] It can also be understood that the bandwidth parameter is an important indicator to measure the current running state of the model, which can include but is not limited to residual bandwidth, rated bandwidth, and current concurrency. Among them, the residual bandwidth is used to represent the bandwidth resources not used by the deployment node where the model is located, and the rated bandwidth is the maximum bandwidth capacity that the deployment node where the model is located can provide.

[0027] In some embodiments, when a user initiates a query request, the relevant bandwidth parameters of each model can be extracted from the predefined first model set.

[0028] Illustratively, assuming that a user's query request is an image recognition task, all models used for image recognition can be selected from the first model set, and then the current bandwidth usage of each model is obtained. For example, model A is deployed on a certain cloud server instance, the rated bandwidth of model A is 1Gbps, and the current used bandwidth is 800Mbps, so the residual bandwidth is 200Mbps; while the rated bandwidth of model B is also 1Gbps, but the current used bandwidth is 950Mbps, and the residual bandwidth is 50Mbps.

[0029] It should be noted that in actual implementation, the bandwidth data of each model can be collected in real time to ensure that subsequent routing decisions are always based on the latest load status, thereby avoiding service congestion and user experience degradation caused by model overload.

[0030] S102, in response to the user query request, input the query text corresponding to the user query request into the routing model to obtain the prediction accuracy and prediction cost of the model for the user query request.

[0031] The user query request refers to a query request initiated by a user through a terminal device, and the request content can include but is not limited to natural language text, structured data, picture information, and video stream data, and can express the user's intention.

[0032] It can be understood that the routing model is a machine learning model or algorithm used to evaluate the performance of each model. The routing model can receive the text content or image content of the user query request, and predict the accuracy and calling cost of each model when processing the user query request according to historical data.

[0033] The prediction accuracy refers to the consistency between the prediction result and the actual result of the model in historical tasks, which is usually expressed in percentage; the prediction cost refers to the resource overhead required by the model to execute the user query request, such as CPU time, memory occupation, network transmission, etc.; the lower the cost, the higher the model running efficiency, and it is suitable for large-scale concurrent calling. For example, the prediction accuracy of model A is 92%, the prediction cost is 0.05 seconds / time, the prediction accuracy of model B is 88%, and the prediction cost is 0.03 seconds / time. Therefore, although the accuracy of model B is slightly lower, because the cost of model B is lower, it may be more suitable to be selected in some scenarios.

[0034] In some embodiments, the routing model can adjust the weight distribution of accuracy and cost according to different business needs, so as to optimize the balance between overall service efficiency and quality. In addition, the routing model can also combine user portrait, historical behavior and other information for personalized prediction to improve the recommendation effect and user satisfaction.

[0035] S103, determining a target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost.

[0036] It can be understood that the target model refers to the model finally determined for processing the current user query request after multi-dimensional evaluation of bandwidth, accuracy, cost, etc. after the user query request arrives.

[0037] Next, the embodiments of the present application introduce several implementation ways of determining the target model.

[0038] In one possible implementation, Figure 2 The flowchart of a model determination method provided by the embodiments of the present application Figure 2 For example, Figure 3As shown, the above-mentioned S103 "determining the target model for processing the user query request from the first model set based on bandwidth parameters, prediction accuracy, and prediction cost" may further include the following steps: S1031, determine the bandwidth margin coefficient corresponding to the model based on the remaining bandwidth and the rated bandwidth.

[0039] It is understandable that the remaining bandwidth refers to the unused available bandwidth capacity on the model deployment node, reflecting the real-time resources that the system can schedule; the rated bandwidth refers to the maximum theoretical bandwidth limit of the model deployment node, which is determined by the hardware configuration or cloud service instance specifications.

[0040] It can also be understood that the bandwidth redundancy coefficient is a quantitative indicator used to represent the proportion of the current remaining bandwidth of the deployment node where the model is located to the total bandwidth. The calculation formula can be expressed as: Bandwidth redundancy coefficient = Remaining bandwidth / Rated bandwidth.

[0041] It should be noted that the bandwidth abundance coefficient ranges from 0 to 1. When the bandwidth abundance coefficient is ≥0.2, the model can be considered to be in a bandwidth abundance state, that is, the deployment node where the model is located has sufficient bandwidth resources and can receive more concurrent requests. When the bandwidth abundance coefficient is <0.1, the model can be considered to be in a bandwidth shortage state, that is, the deployment node where the model is located is close to the bandwidth bottleneck.

[0042] S1032. Based on the model's bandwidth surplus coefficient, prediction accuracy, and prediction cost, determine the model's bandwidth score, prediction accuracy score, and cost score.

[0043] It is understandable that before determining the bandwidth score, prediction accuracy score, and cost score, the bandwidth abundance coefficient, model prediction accuracy, and invocation cost need to be uniformly mapped to the same numerical range to avoid the influence of differences in units on the weight calculation. For example, the bandwidth abundance coefficient, model prediction accuracy, and invocation cost can be normalized to 0~1 respectively; the bandwidth abundance coefficient can be mapped to the range [0, 1], with the value closer to 1 when the bandwidth is more abundant; the historical prediction accuracy of the model can be normalized, and its numerical range can be adjusted to [0, 1], with the higher the normalized accuracy value, the closer the value is to 1. The model invocation cost (such as computing resource consumption, execution time, etc.) can be normalized to [0, 1], with the lower the cost, the closer the value is to 1.

[0044] In some embodiments, information from three dimensions—bandwidth parameters, prediction accuracy, and prediction cost—can be comprehensively considered to construct a bandwidth-quality-cost multidimensional decision-making model. First, the weights for accuracy, cost, and bandwidth can be determined. Then, based on these weights, along with the normalized bandwidth abundance coefficient, normalized prediction accuracy, and normalized prediction cost, a scoring calculation model is constructed, thereby obtaining the comprehensive score for each model.

[0045] For example, the formula for calculating the model's overall score can be expressed as: Model Overall Score = (Accuracy Weight × Accuracy) + (Cost Weight × Call Cost) + (Bandwidth Weight × Bandwidth Redundancy Coefficient).

[0046] For example, suppose that after analysis of a user's query request, Model A has a bandwidth surplus coefficient of 0.3, a prediction accuracy of 85%, and a prediction cost of 0.5; Model B has a bandwidth surplus coefficient of 0.2, a prediction accuracy of 90%, and a prediction cost of 0.6. If the weight allocation is: bandwidth weight 0.3, accuracy weight 0.4, and cost weight 0.3, then the overall score of Model A is 0.3×0.3 + 0.4×0.85 + 0.3×0.5 = 0.58; and the overall score of Model B is 0.3×0.2 + 0.4×0.9 + 0.3×0.6 = 0.6. Therefore, Model B will be selected as the target model for processing the user's query request.

[0047] It should be noted that the initial values ​​of each weight can be configured by the user according to business needs (such as the default accuracy weight of 0.4, cost weight of 0.3, and bandwidth weight of 0.3).

[0048] It should also be noted that the weight allocation can be dynamically adjusted based on the current load status. For example, in situations where bandwidth is scarce, the bandwidth weight can be appropriately increased to prioritize models with more idle resources; while in scenarios with high service quality requirements, the accuracy weight can be increased to ensure the accuracy of the output results. S1033, determine the target model based on the bandwidth score, prediction accuracy score, and cost score.

[0049] In some embodiments, the models in the first model set can be sorted according to the comprehensive model score calculated based on the bandwidth score, prediction accuracy score, and cost score, and the model with the highest comprehensive score can be selected as the target model to handle the current user request.

[0050] In this way, by using a weighted calculation model to calculate the overall score, dynamic ranking and selection of models can be achieved, making reasonable use of available bandwidth resources while meeting service quality requirements and cost control conditions. The weighted calculation model supports flexible adjustment of weight allocation according to actual scenarios, thereby adapting to different business needs and system load states.

[0051] In another possible implementation, the above-mentioned S103 "determining the target model for processing the user query request from the first model set based on bandwidth parameters, prediction accuracy, and prediction cost" may further include the following steps: The semantic features of the user query, real-time network bandwidth, and performance metrics (prediction accuracy, prediction cost) of each candidate model can be used as input features and fed into a pre-trained neural network model. This model outputs a decision vector through forward inference, based on which it dynamically determines and calls the target model best suited for this specific scenario from a first set of models to process the user query request.

[0052] This application provides a model determination method. On the one hand, by introducing bandwidth parameters as the basis for routing decisions, resource blocking or service delays caused by insufficient model bandwidth can be avoided, thereby improving the stability of routing model decisions. On the other hand, by adding bandwidth constraints based on prediction accuracy and call costs, the routing decisions are made closer to the actual deployment environment, thereby optimizing resource utilization while ensuring service quality.

[0053] In some embodiments, the above-described S1033 "determining the target model based on the bandwidth score, prediction accuracy score, and cost score" may further include the following steps: S201, when the bandwidth parameter meets the bandwidth amplification condition, the bandwidth score of the model is adjusted based on the bandwidth parameter to obtain the adjusted bandwidth score; S202, determine the target model based on the adjusted bandwidth score, prediction accuracy score, and cost score.

[0054] The bandwidth amplification condition refers to the condition that when the remaining bandwidth of a model deployment node reaches a certain threshold, it is determined that the node has additional processing capabilities and the routing priority of the target model can be appropriately increased.

[0055] For example, the bandwidth amplification condition can be that the bandwidth margin coefficient of the model is greater than or equal to a preset threshold. The preset threshold can be 0.2 or other values. The specific value can be user-defined according to different scenarios. This application embodiment does not limit this.

[0056] In some embodiments, the bandwidth score of the model can be adjusted based on the bandwidth abundance coefficient of the model. The bandwidth abundance coefficient can be linearly amplified based on the first coefficient to obtain a first value, and the adjusted bandwidth score can be determined by multiplying the first value with the original bandwidth score.

[0057] For example, the adjusted bandwidth score calculation formula can be expressed as: Adjusted bandwidth score = Original bandwidth score × (1 + Bandwidth margin coefficient × 0.5).

[0058] For example, the overall score of model A is 0.3×0.3+0.4×0.85+0.3×0.5=0.58; the bandwidth score is 0.09; the adjusted bandwidth score is 0.09×(1+0.3×0.5)=0.1035. Furthermore, the overall score of model A can be determined as 0.1035+0.4×0.85+0.3×0.5=0.5935.

[0059] For example, if a model has a bandwidth margin coefficient of 0.3 and an original bandwidth score of 0.8, then after skewing (i.e. after adjustment), the score is 0.8×(1+0.3×0.5)=0.92, which is preferred over the unskewed model.

[0060] In this embodiment of the application, the bandwidth surplus is monitored to determine whether the bandwidth amplification condition is met. When there is a model with surplus bandwidth, its comprehensive score is weighted and amplified to increase its priority in routing decisions. This can guide traffic to models with lower load and sufficient processing capacity, thereby improving resource utilization and reducing expansion needs.

[0061] In some embodiments, the above-described S103 "determining the target model for processing the user query request from the first model set based on bandwidth parameters, prediction accuracy, and prediction cost" may further include the following steps: S301, add the models in the first model set whose bandwidth parameters meet the preset bandwidth conditions to the second model set.

[0062] It is understandable that the preset bandwidth condition can be that the bandwidth surplus coefficient of the model is greater than a preset threshold. The preset threshold can be 0, 0.1, 0.2, etc., and the specific value can be set according to the specific scenario. If the preset threshold is 0, it means that the bandwidth surplus coefficient is greater than 0, that is, there is spare bandwidth, and the model is determined to be a usable model, while models with 100% bandwidth utilization (i.e., no spare bandwidth) are eliminated.

[0063] In some embodiments, if the bandwidth parameters of the deployment node where the model is located in the first model set meet the preset bandwidth conditions, the model is determined to be an available model; and a second model set is determined based on all available models in the first model set.

[0064] In some embodiments, the second model set can be updated based on a preset update timing. For example, the preset update timing can be automatically triggered before each round of routing decision (i.e., after the user query list is entered and before the model is reordered), or triggered by immediate filtering and updating when the model bandwidth abundance coefficient changes from a first condition to a second condition. The first condition is that the bandwidth abundance coefficient is greater than a first threshold; the second condition is that the bandwidth abundance coefficient is less than a second threshold; the first threshold is greater than or equal to the second threshold; for example, the first threshold can be 0.2 and the second threshold can be 0.1.

[0065] It can also be understood that the models in the second model set have already passed the initial filtering of bandwidth resources and possess a certain degree of concurrent processing capability. Based on all available models in the first model set, the second model set can be determined, narrowing the original model set into a model pool that is more focused on performance and resource balance, thereby helping to improve the accuracy and efficiency of routing decisions.

[0066] S302, based on bandwidth parameters, prediction accuracy, and prediction cost, determines the target model from the second model set to handle user query requests.

[0067] Based on the second set of models, a multi-dimensional evaluation system is constructed according to key indicators such as bandwidth parameters, prediction accuracy, and call cost of the deployment node where the model is located. By normalizing the three indicators of prediction accuracy, call cost, and bandwidth parameters and assigning corresponding weights, a comprehensive score for each model is calculated. Finally, the model with the highest score from the second set of models is selected as the target model to handle the current user's query request.

[0068] In this embodiment, models that meet preset bandwidth conditions can be selected as available models first, and then routing decisions can be made within this subset. This approach, on the one hand, eliminates models with no available bandwidth through a pre-screening mechanism, preventing invalid calls and resource waste; on the other hand, it narrows the decision-making scope, improves routing efficiency, and reduces redundant computation.

[0069] In some embodiments, after determining the second set of models, the following steps may also be included: S401, determine the candidate models that meet the first preset conditions from the first model set.

[0070] S402, add the candidate model to the second model set.

[0071] The first preset condition is that the model's historical prediction accuracy is greater than the preset accuracy.

[0072] Historical prediction accuracy refers to the accuracy with which a model predicts user query requests over a predetermined historical period, reflecting the model's long-term predictive stability. For example, the historical prediction accuracy can be determined by statistically analyzing the match between the model's predictions on the test set and the true labels. For instance, if model A processed 1000 user query requests in the past week, and correctly predicted 950, then model A's historical prediction accuracy would be 95%.

[0073] It can also be understood that the preset accuracy rate is a benchmark value set by the user or the system to filter out models that meet a certain performance standard. The preset accuracy rate can be flexibly configured according to business needs, for example, set to 85% or 90%, to ensure that only high-accuracy models can enter the next routing decision process.

[0074] In some embodiments, models with historical prediction accuracy greater than a preset accuracy can be identified from the first model set as candidate models, and the model status information of each candidate model can be recorded; and each candidate model can be added to the second model set to obtain an updated second model set; wherein, the model status information may include, but is not limited to, the model's basic prediction accuracy, maximum concurrency, and the model's deployment node information.

[0075] It should be noted that the actual call accuracy of the model can be determined within a preset configuration period (e.g., every 24 hours), and the second model set can be updated again based on the actual call accuracy data. Models with actual call accuracy below the preset threshold for three consecutive times are removed. At the same time, models that meet the first preset condition can be identified from the first model set and added back to the second model set to supplement the new qualified models. In this way, the real-time performance of the second model set can be guaranteed, thereby improving the accuracy and real-time performance of model routing.

[0076] It is understandable that S401 and S402 above are for updating the second model set. The second model can be updated only if the models in the second model set meet certain update conditions.

[0077] In some embodiments, the above-mentioned step S401, "determining candidate models that satisfy the first preset condition from the first model set," includes the following steps: In response to the fact that the prediction accuracy of each model in the second model set is lower than the target threshold, a candidate model that meets the first preset condition is determined from the first model set. The bandwidth parameter of the deployment node where the candidate model is located does not meet the preset bandwidth condition.

[0078] The target threshold can be a preset accuracy threshold; for example, the target threshold can be 85%, that is, when the prediction accuracy of all models in the second model set is less than 85%, the second model set is updated by adding the high-accuracy models in the first model set that meet the first preset condition to the second model set, and the second model set is updated.

[0079] It is understandable that when the prediction accuracy of all models in the second model set fails to reach the set target threshold, it indicates that the current available model pool lacks high-accuracy models, and the overall quality is low, which may not meet the business's service quality requirements. In this case, instead of relying on the low-accuracy models in the second model set, high-accuracy models are added to the second model set from the first model set.

[0080] It can also be understood that the bandwidth parameters of the deployment node where the candidate model is located do not meet the preset bandwidth conditions. In other words, even if the bandwidth parameters of the deployment node where a high-accuracy candidate model is located do not meet the preset bandwidth conditions, it will still be added to the second model set first. Since the bandwidth usage of the deployment node is close to saturation (i.e., it does not meet the preset bandwidth conditions), it cannot be called immediately. The candidate models can be treated as potential schedulable resources to facilitate the gradual activation of the calling capability of these models that meet the first preset conditions through mechanisms such as waiting and scaling.

[0081] In this embodiment, when the prediction accuracy falls below a target threshold (i.e., when the current candidate model is of poor quality), a candidate model from the first model set is added (proactively introducing historically high-performing models as supplements), and filtering is performed in conjunction with bandwidth constraints. This prevents low-accuracy models from dominating routing decisions and thus reducing overall service quality; and it also ensures that high-quality models are still available in extreme cases through a fallback mechanism, enhancing system robustness.

[0082] In some embodiments, after adding a candidate model to the second model set, the following steps may also be included: S501, within a preset waiting time, determine one or more candidate models from the second model set whose bandwidth parameters meet the second preset conditions.

[0083] The preset waiting time, also known as the maximum waiting time, refers to the time window reserved for routing decisions. The preset waiting time allows the system to complete the screening and evaluation of models in the available second model set within a specified time after receiving a user query request, ensuring timely response. For example, the preset waiting time can be 50 milliseconds, 5 seconds, 6 seconds, etc., and can be customized according to different business scenarios; this embodiment does not impose any limitations on this.

[0084] The second preset condition can be used to evaluate whether the bandwidth status of the candidate model is stable and can handle additional requests within a specific time window. For example, the first preset condition may be that the bandwidth surplus coefficient in the bandwidth parameters is greater than a preset threshold, the bandwidth utilization rate is less than a preset threshold, or the remaining bandwidth is greater than a preset threshold. For example, the first preset condition may be that the bandwidth surplus coefficient is greater than 0.1, or the bandwidth utilization rate is less than 90%, or the remaining bandwidth is greater than 10MB / s.

[0085] In this way, by setting reasonable second preset conditions, the bandwidth status assessment method can avoid excessive consumption of bandwidth resources while ensuring service quality.

[0086] S502 determines the target model from one or more candidate models based on bandwidth parameters, prediction accuracy, and prediction cost.

[0087] In some embodiments, based on the candidate models in the second model set determined in S501 above, the bandwidth parameters, prediction accuracy and call cost of the deployment node where each candidate model is located can be further comprehensively considered to calculate the comprehensive score of each candidate model, and sort them according to the score. Finally, the candidate model with the highest score in the second model set is selected as the target model.

[0088] Under the high-accuracy model (i.e., candidate model) activation mechanism, if the accuracy of all models in the second model set is lower than a preset threshold, a priority enhancement strategy will be initiated. This temporarily relaxes bandwidth restrictions and adds models from the first model set with accuracy higher than the preset threshold, thereby ensuring service quality. Simultaneously, if a candidate model with accuracy higher than the preset threshold is experiencing bandwidth constraints, a waiting mechanism will be activated until bandwidth is released, at which point it will be immediately invoked.

[0089] In this embodiment, when the bandwidth of a candidate model is limited, a waiting window is set. If bandwidth is released during this period, the model is immediately enabled. This avoids resource waste caused by direct capacity expansion and improves resource utilization efficiency. Furthermore, it reduces service interruption time and ensures a continuous user experience.

[0090] In some embodiments, after adding a candidate model to the second model set, the following steps may also be included: S601: If, within a preset waiting period, the bandwidth parameters of the deployment nodes where one or more candidate models in the second model set are located meet the expansion conditions, an expansion request is sent to the operation and maintenance management system.

[0091] Among them, the expansion conditions can be that the bandwidth surplus coefficient in the bandwidth parameters is less than a preset threshold, the bandwidth utilization rate is greater than a preset threshold, or the remaining bandwidth is less than a preset threshold. For example, the expansion conditions can be that the bandwidth surplus coefficient is less than 0.1, or the bandwidth utilization rate is greater than 90%, or the remaining bandwidth is less than 10MB / s.

[0092] It is understandable that when the bandwidth parameters of the deployment nodes where one or more candidate models in the second model set are located meet the expansion conditions (e.g., bandwidth surplus coefficient < 0.1), it is determined that one or more models in the second model set are in a state of bandwidth shortage. At this time, the operation and maintenance management system will trigger the expansion mechanism to increase the processing capacity of one or more models in the second model set.

[0093] The operation and maintenance management system can be used to allocate bandwidth resources to the deployment nodes where the model is located.

[0094] It can also be understood that an operations and maintenance management system is a system module that centrally manages computing and network resources. The functions of an operations and maintenance management system may include, but are not limited to, resource allocation, capacity planning, monitoring, and automatic scaling control. Operations and maintenance management systems are typically deployed in data centers or cloud platforms and have the ability to interact with underlying infrastructure (such as virtual machines, containers, load balancers, etc.).

[0095] It can also be understood that a capacity expansion request is a resource adjustment instruction initiated by the routing decision system. The content of the capacity expansion request may include, but is not limited to, the target model identifier, the amount of bandwidth to be increased, the expected usage period, the current concurrency, and other information.

[0096] It should be noted that scaling requests can be submitted to the operations and maintenance management system through standardized interfaces (such as REST API, message queues, RPC, etc.).

[0097] In some embodiments, after receiving a capacity expansion request from the routing decision system, the operation and maintenance management system will dynamically allocate additional bandwidth resources to the specified model based on the current availability of the resource pool, so as to meet the concurrent call requirements of the specified model.

[0098] In other embodiments, the operation and maintenance management system can automatically make decisions based on the bandwidth status of one or more candidate models in the second model set reported in real time by the load monitoring module, determine whether to initiate expansion operations, and coordinate the underlying infrastructure (such as cloud servers and container clusters) to adjust resources.

[0099] For example, in a high-concurrency business scenario, when the bandwidth utilization of one or more models in the second model set reaches 98%, the operation and maintenance management system can automatically allocate additional CPU or network resources to one or more models in the second model set, thereby improving the concurrent processing capability of one or more models in the second model set.

[0100] In this way, by sending a capacity expansion request to the operation and maintenance management system, the system can respond promptly to sudden bandwidth demands during model operation. It can dynamically adjust the availability of one or more models in the second model set without interrupting service. The bandwidth expansion mechanism can improve the elasticity and stability of the overall system, thereby ensuring the continuity and stability of model services, and ultimately improving the overall service quality and reducing operation and maintenance costs.

[0101] S602, upon receiving the expansion completion instruction, determines the target model from one or more candidate models based on bandwidth parameters, prediction accuracy, and prediction cost.

[0102] The expansion completion command is a notification signal issued by the operations and maintenance management system after successfully completing bandwidth expansion. This command indicates that the system has adjusted resources according to the expansion request and that the target model now has sufficient bandwidth to handle new requests. The expansion completion command may include, but is not limited to, key information such as the expanded bandwidth value, effective time, and resource allocation status, for the routing decision system to reference and record.

[0103] It's understandable that after scaling up the candidate models, the bandwidth parameters of the deployment nodes hosting those models will change; for example, the bandwidth margin coefficient of a candidate model might increase from 0.05 to over 0.2. In other words, after scaling up the candidate models, the model parameters of the models in the second model set need to be updated. Models with changed bandwidth parameters or whose bandwidth parameters exceed a preset threshold will participate in the subsequent model ranking and selection logic. At this point, the routing framework will re-evaluate the overall score of the candidate models, including factors such as bandwidth margin, prediction accuracy, and call cost, and select the optimal model as the target model for processing user requests.

[0104] It should be noted that after the routing decision system receives the expansion completion instruction, it can update the bandwidth status information of the candidate models and mark them as routable. By associating the status information of the candidate models with the resource readiness status, the system ensures that a model is only included in the routing logic when the resources are truly ready, avoiding false expansion or incomplete readiness, thereby guaranteeing the reliability and consistency of the call requests.

[0105] For example, suppose a machine learning model experiences a surge in concurrent calls during peak hours, causing its bandwidth to approach its limit. Upon detecting this, the routing decision system sends a request to the operations and maintenance management system to increase bandwidth. The management system then allocates more bandwidth resources. After confirming the increase is complete, the routing decision system reactivates candidate models that were previously suspended due to insufficient bandwidth and sets them as the target model on the routing path, thus ensuring the normal operation of the service.

[0106] In this embodiment, when the waiting period cannot meet the demand, the expansion process can be automatically triggered to ensure that the model restores its service capacity as soon as possible. This achieves two goals: firstly, by reducing manual intervention through an automated expansion mechanism, the system's response speed is improved; secondly, by combining bandwidth monitoring and scheduling logic, dynamic matching of resource supply and business demand is achieved, thereby improving overall operational efficiency.

[0107] In other words, in scenarios where low-accuracy models dominate, service quality can be guaranteed by activating high-accuracy models, while the "wait-expansion" two-step method can be used to balance load pressure and service timeliness.

[0108] The following describes the application of the model determination method provided in the embodiments of this application in a real-world scenario.

[0109] With the continuous development of artificial intelligence technology, model services have been widely used in many fields. In order to improve model inference efficiency and reduce operating costs, a model routing mechanism is usually adopted to dynamically select from multiple candidate models.

[0110] Existing model routing frameworks only make tradeoff decisions based on "model prediction quality - call cost", without incorporating real-time bandwidth constraints and dynamic load status of model deployment into the routing logic.

[0111] First, this application's embodiments introduce existing model routing frameworks, exemplarily, Figure 3 This is a schematic diagram of an existing model routing framework for a model determination method provided in an embodiment of this application. Figure 4 As shown in the diagram, the routing model framework mainly includes three modules: device-side routing module 301, model routing service module 302, and cloud-based model pool module 303.

[0112] The device-side routing module 301 includes a user request unit 3011, a device-side routing decision unit 3012, and a device-side model unit 3013. The user request unit 3011 (i.e., User Query) receives and initiates user query requests. The device-side routing decision unit 3012 (i.e., On-device Model router) performs initial routing decisions based on local policies (such as latency and privacy), determining whether to send the request to the device-side model unit 3013 or forward it to the cloud routing module 302. The device-side model unit 3013 (i.e., On-device LLMs) consists of lightweight models deployed on the terminal device, used to handle user requests with extremely high latency and privacy requirements.

[0113] The model routing service module 302 includes a model routing unit 3021 and a model connection adaptation unit 3022.

[0114] The model routing service module 302 is the core of cloud-based routing decisions. It receives requests from the device-side router module 301 or directly from the gateway and employs more complex strategies to select the optimal model. For example, the model routing service module 302 may include three routing strategies: (1) Device / Cloud Routing: Based on the model's capabilities and computing resources, automatically select whether to use a local LLM / Agent or a cloud-based model.

[0115] (2) Quality first: without considering cost, only select the model that can provide the best performance for the current query (e.g., the most powerful GPT-4 class model).

[0116] (3) Quality-cost trade-off: Under the premise of ensuring a certain level of quality, choose a lower-cost model (e.g., use GPT-3.5 instead of GPT-4) to reduce API call costs.

[0117] The core technology of the model routing service module 302 is to achieve intelligent decision-making using an "LLM representation + small model" approach. The LLM representation vectorizes or semantically understands the characteristics of user queries and available models (such as programming proficiency, long text understanding, and cost). The small model refers to a specially trained small classification or decision-making model that quickly predicts the most suitable model based on the query's "representation." This approach is more efficient and less costly than using a large model for decision-making each time.

[0118] The Model Router (3021) receives final instructions (e.g., "Use Model X") from the Model Routing Service and distributes queries to the corresponding specific model endpoints. The Model Connector (3022) is an adaptation layer responsible for handling the actual communication with APIs of various models. It converts the unified internal request format into the specific API format required by different model providers (such as OpenAI, Anthropic, and local private models) and processes the returned results. This ensures the system's scalability and allows for easy integration of new models.

[0119] The cloud model pool module 303 aggregates all callable cloud model resources, including public model units 3031 and private model units 3032. Public model units 3031 (i.e., Public LLMs) refer to commercial large language models provided by third-party service providers and accessed via API. Private model units 3032 (i.e., Private LLMs) refer to proprietary large language models built by enterprises themselves or deployed in private environments, used to meet data security and customization needs.

[0120] However, the existing routing model described above has the following shortcomings: 1. When the number of concurrent calls to high-priority models suddenly increases, it can easily trigger the exhaustion of bandwidth resources, causing subsequent requests to be blocked. This requires frequent initiation of the bandwidth expansion process, increasing operation and maintenance costs and service latency. 2. Low-load but low-accuracy models may be preferred, while high-accuracy models may be excluded from the routing range due to short-term load peaks, resulting in a decrease in quality of service (QoS). 3. The routing decision is disconnected from the model deployment status, making it impossible to dynamically adjust the candidate model pool based on bandwidth availability, which poses risks of resource mismatch and service stability.

[0121] Therefore, this application proposes a model determination method. By adding a real-time large model load monitoring module to the model routing framework and constructing a multi-dimensional routing decision system of "bandwidth-load-quality-cost", it addresses the shortcomings of existing technologies. Specifically, it includes three parts: module architecture, dynamic adjustment process, and control rules. In some embodiments, the bandwidth parameters of the deployment node where the model is located can be monitored through the load monitoring module.

[0122] 1. The load monitoring module includes: Bandwidth data acquisition unit: Deployed on each model deployment node (such as containerized clusters, cloud server instances) through lightweight probes to collect the current concurrency of model calls, used bandwidth, remaining bandwidth, and bandwidth utilization rate in real time (calculation formula: bandwidth utilization rate = used bandwidth / rated bandwidth of deployment node × 100%). The acquisition frequency is set to 100ms / time to ensure data real-time performance. Load status analysis unit: preprocesses the collected bandwidth data, calculates the absolute value of spare bandwidth (remaining bandwidth) and bandwidth abundance coefficient (calculation formula: bandwidth abundance coefficient = remaining bandwidth / rated bandwidth, value range 0~1, coefficient ≥0.2 is defined as "bandwidth abundance", coefficient <0.1 is defined as "bandwidth shortage"). Data synchronization unit: The "model identifier-free bandwidth-bandwidth surplus coefficient" triple is synchronized to the model reordering framework in real time via a message queue (such as Kafka), with a synchronization latency of ≤500ms, ensuring that routing decisions are based on the latest load status.

[0123] 2. The load monitoring module, which interacts with the routing framework, acts as a front-end data input unit. Before the "quality-cost tradeoff" decision-making process, it dynamically filters the candidate routing model library. Filtering rules: Only retain models with "bandwidth redundancy coefficient ≥ 0" (i.e., there is spare bandwidth), and remove models with 100% bandwidth utilization (no spare bandwidth) to form a "available model subset" (that is, the second model set in the above embodiment). Filtering trigger timing: Automatically triggered before each round of routing decision (i.e. after the user query list is entered and before the model is reordered), or triggered when the bandwidth surplus coefficient of a certain model drops from ≥0.1 to <0.1, triggering an immediate filter update.

[0124] In some embodiments, in the existing re-ranking model based on "quality (prediction accuracy) - cost (call cost)," a bandwidth sufficiency coefficient can be added as a core decision parameter, with the specific optimization as follows: 1. Parameter normalization: The bandwidth margin coefficient (0~1), model prediction accuracy (0~1), and call cost (normalized to 0~1, the lower the cost, the closer the value is to 1) are uniformly mapped to the same numerical range to avoid the difference in units affecting the weight calculation. 2. Multi-dimensional weight allocation model: Constructing the weight calculation formula: Model overall score = (accuracy weight × accuracy) + (cost weight × normalized cost) + (bandwidth weight × bandwidth redundancy coefficient) The initial values ​​of each weight can be configured by the user according to business needs (such as a default accuracy weight of 0.4, cost weight of 0.3, and bandwidth weight of 0.3), and dynamic adjustment is supported (such as increasing the bandwidth weight to 0.4 in bandwidth-constrained scenarios). 3. Dynamic update of candidate model pool: The comprehensive score is calculated and ranked only based on the "available model subset", ensuring that routing decisions are made only among models with spare bandwidth, thus avoiding bandwidth resource exhaustion from the source.

[0125] In some embodiments, the bandwidth score can be weighted and amplified through the weight skew mechanism of the bandwidth abundance model.

[0126] 1. Weight skew triggering condition: When a model with a "bandwidth margin coefficient ≥ 0.2" exists in the routing candidate model library, weight skew will be automatically initiated; 2. Tilt Calculation Method: The overall score of the bandwidth-rich model is weighted and amplified. The calculation formula is as follows: The post-tilt composite score = original composite score × (1 + bandwidth redundancy coefficient × 0.5) (For example: if a model has a bandwidth redundancy coefficient of 0.3 and an original composite score of 0.8, then the score after skewing is 0.8×(1+0.3×0.5)=0.92, which is preferred over the unskewed model.) In this way, by tilting the weights, the routing prioritizes models with sufficient bandwidth, reducing the call pressure on high-load models, avoiding bandwidth expansion triggered by excessive concurrency, and reducing infrastructure costs.

[0127] In some embodiments, the real-time performance and accuracy of routing decisions can be ensured through the activation and fallback mechanism of a high-accuracy model.

[0128] 1. Maintenance of the list of high-accuracy models List composition: Models with a pre-screened historical prediction accuracy of ≥90% (configurable threshold) are recorded, along with their "basic accuracy, maximum concurrency, and deployment node information". Dynamic updates: Every 24 hours (configurable period), the accuracy data based on the actual use of the model is re-filtered, models with accuracy below the threshold for three consecutive times are removed, and new qualified models are added.

[0129] 2. Activation trigger condition: When the real-time accuracy of all models in the "available model subset" of the current route candidate is less than the preset threshold (e.g., 85%), the activation mechanism is automatically triggered.

[0130] 3. Post-activation operation process Step 1: Break the load bottleneck—allow routes to include high-accuracy models in the candidate pool, even if their bandwidth surplus coefficient is <0.1 (bandwidth constraints). Step 2: Call waiting mechanism—start a queue to wait for call requests to the high-accuracy model, set the maximum waiting time (e.g., 5 seconds, configurable), and if the bandwidth is released within the waiting time (sufficiency coefficient ≥ 0.1), then execute the call immediately; Step 3: Deployment Expansion Trigger — If no bandwidth is released after the waiting time expires, an expansion request will be automatically sent to the operation and maintenance management system. The request content includes "target model identifier, current concurrency, and required expansion bandwidth quota". After the expansion is completed (bandwidth surplus coefficient ≥ 0.1), the parameters of each model in the route candidate pool will be updated. This ensures that even in scenarios where low-accuracy models dominate, service quality can still be guaranteed by activating high-accuracy models. At the same time, the "wait-expansion" two-step method balances load pressure and service timeliness.

[0131] Based on the above embodiments, this application also provides a model determining device. Figure 4 This is a schematic diagram of the composition structure of a model determination device provided in an embodiment of this application, as shown below. Figure 5 As shown, the model determining device 400 includes a first determining module 401, a second determining module 402, and a third determining module 403, wherein: The first determining module 401 is used to determine the bandwidth parameters of the deployment node where the model in the first model set is located; wherein, the model is a model that generates query results based on prompt information. The second determining module 402 is used to respond to a user query request by inputting the query text corresponding to the user query request into the routing model to obtain the prediction accuracy and prediction cost of the model for the user query request. The third determination module 403 is used to determine the target model for processing user query requests from the first model set based on bandwidth parameters, prediction accuracy, and prediction cost.

[0132] In some embodiments of this application, the bandwidth parameters include the remaining bandwidth and the rated bandwidth. The third determining module 403 is further configured to determine the bandwidth margin coefficient corresponding to the model based on the remaining bandwidth and the rated bandwidth; determine the bandwidth score, prediction accuracy score and cost score of the model based on the bandwidth margin coefficient, prediction accuracy and prediction cost; and determine the target model based on the bandwidth score, prediction accuracy score and cost score.

[0133] In some embodiments of this application, the third determining module 403 is further configured to adjust the bandwidth score of the model based on the bandwidth parameters when the bandwidth parameters meet the bandwidth amplification conditions, and obtain the adjusted bandwidth score; and determine the target model based on the adjusted bandwidth score, prediction accuracy score and cost score.

[0134] In some embodiments of this application, the third determining module 403 is further configured to add models in the first model set whose bandwidth parameters meet preset bandwidth conditions to the second model set; and determine the target model for processing user query requests from the second model set based on bandwidth parameters, prediction accuracy, and prediction cost.

[0135] In some embodiments of this application, the above-described model determining device further includes a fourth determining module and an adding module, wherein: The fourth determination module is used to determine candidate models that meet the first preset condition from the first model set; the first preset condition is that the historical prediction accuracy of the model is greater than the preset accuracy. Add a module to add candidate models to the second model set. In some embodiments of this application, the fourth determining module is further configured to, in response to the fact that the prediction accuracy of each model in the second model set is lower than the target threshold, determine a candidate model from the first model set that meets the first preset condition, wherein the bandwidth parameter of the deployment node where the candidate model is located does not meet the preset bandwidth condition. In some embodiments of this application, the above-described model determining device further includes a fifth determining module and a sixth determining module, wherein: The fifth determining module is used to determine one or more candidate models whose bandwidth parameters meet the second preset conditions from the second model set within a preset waiting time. The sixth determination module is used to determine the target model from one or more candidate models based on bandwidth parameters, prediction accuracy, and prediction cost. In some embodiments of this application, the above-described model determining device further includes a sending module and a seventh determining module, wherein: The sending module is used to send a capacity expansion request to the operation and maintenance management system if the bandwidth parameters of the deployment nodes where one or more candidate models in the second model set are located meet the capacity expansion conditions within a preset waiting time; the operation and maintenance management system is used to allocate bandwidth resources for the models. The seventh determination module is used to determine the target model from one or more candidate models based on bandwidth parameters, prediction accuracy, and prediction cost when receiving the expansion completion instruction.

[0136] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0137] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the above-described method.

[0138] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. The computer-readable storage medium can be transient or non-transient.

[0139] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof.

[0140] In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0141] It should be noted that, Figure 5 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application, such as... ​ As shown, the hardware entity of the electronic device 500 includes: a processor 501, a communication interface 502, and a memory 503, wherein: Processor 501 typically controls the overall operation of electronic device 500.

[0142] Communication interface 502 enables electronic device 500 to communicate with other terminals or servers via a network.

[0143] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 501 and various modules in the electronic device 500. It can be implemented using flash memory or RAM. Data transfer between the processor 501, the communication interface 502, and the memory 503 can be performed via bus 504.

[0144] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0145] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0148] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0150] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory, magnetic disks, or optical disks.

[0151] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0152] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A model determination method, the method comprising: determining a bandwidth parameter of a deployment node where a model in a first model set is located; wherein the model is used to guide generation of a query result based on prompt information; in response to a user query request, inputting query text corresponding to the user query request into a routing model to obtain a prediction accuracy and a prediction cost of the model for the user query request; based on the bandwidth parameter, the prediction accuracy and the prediction cost, determining a target model for processing the user query request from the first model set.

2. The method of claim 1, wherein the bandwidth parameter comprises a residual bandwidth and a rated bandwidth; and determining the target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost comprises: determining a bandwidth margin coefficient corresponding to the model according to the residual bandwidth and the rated bandwidth; determining a bandwidth score, a prediction accuracy score and a cost score of the model according to the bandwidth margin coefficient, the prediction accuracy and the prediction cost of the model; determining the target model according to the bandwidth score, the prediction accuracy score and the cost score.

3. The method of claim 2, wherein determining the target model according to the bandwidth score, the prediction accuracy score and the cost score comprises: in a case where the bandwidth parameter satisfies a bandwidth amplification condition, adjusting the bandwidth score of the model based on the bandwidth parameter to obtain an adjusted bandwidth score; and determining the target model according to the adjusted bandwidth score, the prediction accuracy score and the cost score.

4. The method of claim 1, wherein determining the target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy and the prediction cost comprises: adding a model in the first model set, for which the bandwidth parameter satisfies a preset bandwidth condition, to a second model set; and determining the target model for processing the user query request from the second model set based on the bandwidth parameter, the prediction accuracy and the prediction cost.

5. The method of claim 4, further comprising: determining a candidate model satisfying a first preset condition from the first model set; the first preset condition is that a historical prediction accuracy of a model is greater than a preset accuracy; and adding the candidate model to the second model set.

6. The method of claim 5, wherein determining the candidate model satisfying the first preset condition from the first model set comprises: in response to the prediction accuracy of each model in the second model set being lower than a target threshold, determining the candidate model satisfying the first preset condition from the first model set, wherein a bandwidth parameter of a deployment node where the candidate model is located does not satisfy the preset bandwidth condition.

7. The method of claim 5, further comprising: ​ ​ determining one or more candidate models from the second model set that satisfy a second preset condition on the bandwidth parameter within the preset waiting duration; determining the target model from the one or more candidate models based on the bandwidth parameter, the prediction accuracy, and the prediction cost.

8. The method of claim 7, further comprising: if the bandwidth parameter of the deployment node where one or more candidate models in the second model set are located satisfies a capacity expansion condition within the preset waiting duration, sending a capacity expansion request to an operation and maintenance management system, the operation and maintenance management system being configured to allocate bandwidth resources of the deployment node where the model is located; in a case where a capacity expansion completion instruction is received, determining the target model from the one or more candidate models based on the bandwidth parameter, the prediction accuracy, and the prediction cost.

9. A model determination apparatus, comprising: a first determination module configured to determine a bandwidth parameter of a deployment node where a model in a first model set is located, wherein the model is configured to generate a query result based on prompt information; a second determination module configured to, in response to a user query request, input query text corresponding to the user query request into a routing model to obtain a prediction accuracy and a prediction cost of the model for the user query request; a third determination module configured to determine a target model for processing the user query request from the first model set based on the bandwidth parameter, the prediction accuracy, and the prediction cost.

10. An electronic device comprising a processor and a memory, the memory storing a computer program operable on the processor, characterized in that, The processor implements the steps in the method of any one of claims 1 to 8 when executing the computer program.

Images