Intelligent model calling method and intelligent model calling system
By constructing a multi-level model degradation sequence and a local backup model, and dynamically adjusting the priority of intelligent models, the problem of quality degradation after intelligent model service degradation is solved, achieving high availability and continuity, and improving processing quality and system stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, the service quality of intelligent model services is easily significantly reduced after degradation, mainly due to the mismatch of capabilities and cascading failures caused by the fixed priority in the primary/backup switchover scheme.
A multi-level model degradation sequence is constructed, and the priority of the intelligent model is dynamically adjusted according to the task type. Local backup models are used, and priority is sorted by health score and capability matching degree to ensure that the model automatically switches to a suitable backup model when the main model fails, until the task is successful or the failure threshold is reached.
It improves the high availability and continuity of intelligent model services, reduces the decline in task quality caused by capability mismatch, enhances the processing quality after degradation, maximizes system availability, and reduces the risk of service interruption.
Smart Images
Figure CN122363992A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent models, specifically to an intelligent model invocation method and an intelligent model invocation system. Background Technology
[0002] In the field of multi-intelligent model invocation systems, especially in service architectures that integrate multiple large language model or multimodal model providers, to ensure the high availability of intelligent model services, when the primary model fails, a service degradation mechanism is typically needed to switch requests to a backup model to guarantee the continuity of intelligent model services. How to effectively maintain the service quality after degradation has become a pressing technical problem to be solved in this field.
[0003] The commonly used degradation scheme in related technologies is a simple primary-backup switchover scheme. The primary-backup switchover scheme usually pre-configures a list of models with fixed priorities. When the primary model service is detected to have failed, the next available backup model is selected in a fixed order to process the request. The above mechanism can easily lead to a significant reduction in the service quality after degradation. Summary of the Invention
[0004] In view of this, the embodiments of this application aim to provide an intelligent model invocation method and an intelligent model invocation system to solve the technical problem that existing service degradation mechanisms can easily lead to a significant reduction in the quality of services after degradation.
[0005] The first aspect of this application provides a method for invoking an intelligent model, including: Get the current task request; Obtain the corresponding task type of the current task request; Based on the corresponding task type of the current task request, obtain the corresponding model degradation sequence. The model degradation sequence includes at least two intelligent models arranged in order, and the end of the model degradation sequence is a local backup model. The task request is executed using the smart model that is ranked first in the model degradation sequence, and the execution result of the smart model is obtained. If the intelligent model fails to execute, obtain the next intelligent model in the model degradation sequence; The task request is executed using the next intelligent model in the model degradation sequence. If the intelligent model fails to execute the task, the next intelligent model in the model degradation sequence is obtained until the termination condition is met. The termination condition includes either the intelligent model successfully executing the current task request or the local backup model failing to execute the current task request.
[0006] In one embodiment of this application, the step of obtaining the corresponding model degradation sequence based on the corresponding task type of the current task request includes: Based on the identified task type corresponding to the current task request, the corresponding model degradation sequence is obtained from the pre-stored task type and model degradation sequence mapping database.
[0007] In one embodiment of this application, the model degradation sequence is configured to arrange the intelligent models in descending order based on at least one of the following: the capability matching degree with the task type, the processing capability of the intelligent model, and the unit processing cost of the intelligent model.
[0008] In one embodiment of this application, the step of obtaining the corresponding task type of the current task request includes: Based on the task type identifier in the current task request, obtain the corresponding task type of the current task request.
[0009] In one embodiment of this application, the step of obtaining the corresponding task type of the current task request includes: Based on the request characteristics in the current task request, the corresponding task type of the current task request is inferred.
[0010] In one embodiment of this application, the step of obtaining the next intelligent model in the model degradation sequence when the intelligent model fails to execute includes: If the intelligent model fails to execute, the intelligent models in the model degradation sequence are checked in turn until the first intelligent model that meets the preset working conditions or the local backup model is output.
[0011] In one embodiment of this application, the preset working conditions include the health score of the intelligent model meeting a preset threshold.
[0012] In one embodiment of this application, the preset working conditions include the working state of the intelligent model being normal, and the working state includes normal state, circuit breaker state, and half-open state; The intelligent model invocation method also includes: If the intelligent model in normal condition reaches the preset circuit breaker condition, the current working state of the intelligent model will be switched to the circuit breaker state. If the waiting time of the intelligent model in the circuit-breaker state meets the preset half-open condition, the current working state of the intelligent model will be switched to the half-open state; If the smart model in the half-open state meets the preset recovery conditions, the state of the smart model will be converted to the normal state.
[0013] In one embodiment of this application, the preset circuit breaker condition includes the number of consecutive failures of the task request reaching a set failure threshold; The intelligent model invocation method further includes: if the result of the intelligent model executing the current task request is successful, resetting the consecutive failure count of the intelligent model; And / or, the preset circuit breaker condition includes the failure rate of the task request reaching a set failure rate threshold; And / or, the preset half-open condition includes a waiting time that meets the waiting interval duration; And / or, the preset recovery condition includes the result of successfully executing a preset number of consecutive probe requests.
[0014] In one embodiment of this application, the termination condition further includes the number of failures of the failed smart model reaching a set retry threshold.
[0015] In one embodiment of this application, the intelligent model invocation method further includes: outputting an execution exception if all intelligent models in the model degradation sequence fail to execute.
[0016] In one embodiment of this application, the intelligent model invocation method further includes: outputting a preset default response when all intelligent models in the model degradation sequence fail to execute.
[0017] Another aspect of this application provides an intelligent model invocation system, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement an intelligent model invocation method.
[0018] The intelligent model invocation method of this application embodiment can ensure the high availability of intelligent model services by constructing a multi-level model degradation sequence. When one intelligent model fails, it can automatically switch to a subsequent intelligent model to ensure the continuity of intelligent model services. Furthermore, by establishing a correspondence between task types and model degradation sequences, it can ensure that the degraded model has the ability to match the current task request, thus avoiding a significant drop in task quality due to capability mismatch and effectively improving the processing quality of returned results. In addition, the intelligent model invocation method of this application embodiment can maximize system availability and reduce the risk of complete interruption of intelligent model services by setting up a local backup model. Attached Figure Description
[0019] It should be understood that the following figures only illustrate certain embodiments of this application and should not be construed as limiting the scope.
[0020] It should be understood that the same or similar reference numerals are used in the accompanying drawings to denote the same or similar elements.
[0021] It should be understood that the accompanying drawings are only schematic, and the dimensions and scales of the elements in the drawings are not necessarily precise.
[0022] Figure 1 This is a schematic diagram illustrating the steps of the intelligent model invocation method in an embodiment of this application.
[0023] Figure 2 This is a schematic diagram of another step in the intelligent model invocation method of this application embodiment.
[0024] Figure 3 This is a schematic diagram of another step in the intelligent model invocation method of this application embodiment.
[0025] Figure 4 This is a schematic diagram of another step in the intelligent model invocation method of this application embodiment.
[0026] Figure 5 This is a schematic diagram of another step in the intelligent model invocation method of this application embodiment.
[0027] Figure 6 This is a schematic diagram of another step in the intelligent model invocation method of this application embodiment. Detailed Implementation
[0028] Numerous specific details are set forth below to provide an understanding of the structure, function, and use of the embodiments described and illustrated in the specification and figures. It is to be understood that the embodiments described and illustrated herein are non-limiting examples, and thus it will be appreciated that the particular structural and functional details disclosed herein are representative and exemplary. Variations and changes may be made to these embodiments without departing from the scope of the claims.
[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0030] This application provides a method for invoking intelligent models. This method is used to invoke multiple intelligent models, and can be used for cloud-based multi-intelligent model API services or locally deployed multi-intelligent model systems. It includes: Get the current task request; Get the corresponding task type for the current task request; Based on the corresponding task type of the current task request, obtain the corresponding model degradation sequence. The model degradation sequence includes at least two intelligent models arranged in order, and the end of the model degradation sequence is the local backup model. The task request is executed using the smart model that is ranked first in the model degradation sequence, and the execution result of the smart model is obtained. If the intelligent model fails to execute, obtain the next intelligent model in the model degradation sequence; The task request is executed using the next intelligent model in the model degradation sequence. If the intelligent model fails to execute, the next intelligent model in the model degradation sequence is obtained until the termination condition is met. The termination condition includes either the intelligent model successfully executing the current task request or the local backup model failing to execute the current task request.
[0031] That is, Figure 1 As shown, the intelligent model invocation method in this application embodiment includes: S101: Get the current task request.
[0032] The current task request is a set of instructions sent by the client to the intelligent model calling system to execute a specific artificial intelligence task. It includes the request content and may also include necessary data such as request identifier and user identity information.
[0033] S102: Get the corresponding task type of the current task request; Task types are standardized categories based on the functional and capability requirements of artificial intelligence tasks, used to distinguish the differentiated requirements of different tasks for model capabilities.
[0034] Specifically, in specific embodiments, task types can be categorized into code-based, reasoning-based, dialogue-based, embedding-based, and multimodal tasks. Among these, code-based tasks may include code generation, code analysis, and code review; reasoning-based tasks may include reasoning, planning, decision, and problem solving; dialogue-based tasks may include chat, qa, instruction, and dialogue; embedding-based tasks may include embedding, reranking, and retrieval; and multimodal tasks may include image understanding, video analysis, and OCR.
[0035] S103: Based on the corresponding task type of the current task request, obtain the corresponding model degradation sequence.
[0036] The model degradation sequence includes at least two sequentially arranged intelligent models, with a local backup model at the end of the sequence. Specifically, the model degradation sequence is an ordered list of intelligent models used to try subsequent models in turn when an intelligent model fails to execute a task, thus ensuring the continuity of the current task request execution.
[0037] The model degradation sequence typically differs for different task types. For example, in a specific implementation, the code class can prioritize CodeLlama-34B (professional code model), Qwen-Coder (code capability model), and GPT-4 (general large model) as the third priority intelligent model. In the reasoning class, Qwen3-30B-Thinking (thinking chain model) can be prioritized as the first priority intelligent model, Claude-3-Opus (strong reasoning ability) as the second priority intelligent model, and llama3.2:3b as a local backup model. The local backup model is an intelligent model deployed on the system's local server or edge computing device, unaffected by external network failures or third-party service interruptions, and can serve as the final availability guarantee for the intelligent model calling system.
[0038] S104: Execute the task request using the smart model that is first in the model degradation sequence and obtain the execution result of the smart model; if the execution result of the smart model fails, proceed to S105; if the execution result of the smart model succeeds, the smart model invocation method of this application embodiment ends.
[0039] As can be seen, the execution result of the intelligent model can be either successful or unsuccessful. Success usually means that the intelligent model returns a response that conforms to the expected format within the preset timeout period. Failure usually means that the intelligent model fails to complete the task request, which may include the returned result not conforming to the format requirements, the request processing time exceeding the preset timeout period, the intelligent model returning a response code that displays an error, or the request being unable to reach the intelligent model due to network or service unavailability, etc.
[0040] S105: Obtain the next intelligent model in the model degradation sequence.
[0041] In other words, if the execution result of the intelligent model fails, the next intelligent model in the model degradation sequence can be obtained, and subsequent models can be tried in turn to ensure the continuity of the current task request execution.
[0042] S106: Execute the task request using the next intelligent model in the model degradation sequence. If the execution result of the intelligent model fails, execute S105. If the termination condition is met, the intelligent model invocation method of this application embodiment ends.
[0043] It is understood that the termination condition is used to determine the termination of the intelligent model invocation method in this application embodiment, and is used to control the execution of the degradation process. In the intelligent model invocation method in this application embodiment, the termination condition may include the result that the intelligent model executes the current task request successfully, or the result that the local backup model executes the current task request fails.
[0044] Specifically, the intelligent model invocation method of this application embodiment can be applied to an enterprise-level multi-model invocation system. This system simultaneously accesses multiple cloud-based intelligent models and locally deployed intelligent models, and provides a unified model invocation service. During the execution of the intelligent model invocation method of this application embodiment, a current task request is first received, and then the system obtains the task type corresponding to the current task request. Based on the identified task type, the system obtains a model degradation sequence corresponding to that task type. This model degradation sequence contains multiple intelligent models sorted according to preset rules, and the last model in the sequence is a local backup model deployed on a local server.
[0045] The system first calls the highest-ranking intelligent model in the model degradation sequence, forwards the current task request to that model for processing, and waits for the execution result. If the intelligent model executes successfully, the system returns the execution result to the business system that initiated the request, records the relevant log information for this call, and the process ends.
[0046] If the intelligent model fails to execute, the system records the failure information and retrieves the next intelligent model in the model degradation sequence. The system forwards the current task request to the next intelligent model for processing and waits for the execution result. If the next intelligent model executes successfully, the system returns the execution result to the business system that initiated the request, marks this call as a degradation call in the return result, records relevant log information, and the process ends. If the next intelligent model fails to execute, the system repeats the above steps to retrieve the next intelligent model in the model degradation sequence and attempt to execute it. The above process continues until one of the following two situations occurs: an intelligent model successfully executes the current task request, the system returns the execution result, and the process ends; or all intelligent models in the model degradation sequence fail to execute, including the last local backup model, at which point the system ends the process.
[0047] The intelligent model invocation method of this application embodiment can ensure the high availability of intelligent model services by constructing a multi-level model degradation sequence. When one intelligent model fails, it can automatically switch to a subsequent intelligent model to ensure the continuity of intelligent model services. Furthermore, by establishing a correspondence between task types and model degradation sequences, it can ensure that the degraded model has the ability to match the current task request, thus avoiding a significant drop in task quality due to capability mismatch and effectively improving the processing quality of returned results. In addition, the intelligent model invocation method of this application embodiment can maximize system availability and reduce the risk of complete interruption of intelligent model services by setting up a local backup model.
[0048] Furthermore, such as Figure 2As shown, in one embodiment of this application, S103: the step of obtaining the corresponding model degradation sequence according to the corresponding task type of the current task request includes: S203: Based on the identified task type, retrieve the corresponding model degradation sequence from the pre-stored task type and model degradation sequence mapping database according to the corresponding task type of the current task request.
[0049] In the intelligent model invocation method of this application embodiment, a task type and model degradation sequence mapping database can be pre-stored. The task type and model degradation sequence mapping database contains the mapping relationship between all supported task types and corresponding model degradation sequences; and the corresponding model degradation sequence can be obtained from the task type and model degradation sequence mapping database.
[0050] Understandably, if no model degradation sequence corresponding to the task type is found in the task type and model degradation sequence mapping database, the pre-configured global default model degradation sequence can be used as the degradation sequence for the current task request.
[0051] The intelligent model invocation method in this application embodiment can quickly obtain the model degradation sequence corresponding to the task type by pre-stored task type and model degradation sequence mapping database, effectively improving the speed of degradation response. By setting a global default degradation sequence, it can ensure that requests without pre-configured task types can also obtain degradation protection, thereby improving the availability of the system.
[0052] like Figure 3 As shown, in one embodiment of this application, the step of obtaining the corresponding model degradation sequence according to the corresponding task type of the current task request in the intelligent model invocation method of this application further includes: S303: Based on preset indicators, adjust and rearrange the order of intelligent models in the model degradation sequence.
[0053] The preset metrics include the health score of the intelligent model and / or the execution success rate of the intelligent model.
[0054] Understandably, preset metrics are a set of quantitative parameters used to dynamically assess the real-time operational status of the intelligent model and adjust its degradation priority, reflecting the current availability and execution capability of the intelligent model. The health score is a numerical indicator that comprehensively evaluates the operational status and availability of the intelligent model, and can be calculated from metrics such as the intelligent model's historical success rate, response time, and failure rate. The execution success rate is the ratio of the number of times the intelligent model successfully executes task requests to the total number of executions within a set time window, used to quantify the model's reliability.
[0055] The intelligent model invocation method of this application embodiment can obtain the corresponding model degradation sequence from the pre-stored task type and model degradation sequence mapping database, and then adjust and rearrange the order of intelligent models in the model degradation sequence based on preset indicators. This dynamically modifies the order of each intelligent model in the model degradation sequence, generating a dynamically adjusted model degradation sequence. This ensures that the model degradation sequence reflects the actual operating status of each intelligent model in real time, optimizes the rationality of degradation selection, and avoids the problem of static sequences failing to adapt to model performance fluctuations. The intelligent model invocation method of this application embodiment can prioritize models with better current health and higher execution success rates, significantly improving the first-time success rate of degradation requests and shortening the average task response time. Furthermore, it can enhance the adaptability to complex operating environments and improve the stability of intelligent model services without changing the basic configuration architecture.
[0056] It is understandable that, before and after the order of the intelligent models in the model degradation sequence is adjusted and rearranged, the local backup model always remains at the last position in the model degradation sequence and is not affected by the dynamic rearrangement rules.
[0057] In one embodiment of this application, the model degradation sequence is configured to arrange the intelligent models in descending order based on at least one of the following: capability matching degree with task type, processing capability of intelligent model, and unit processing cost of intelligent model.
[0058] Among them, capability matching degree refers to the degree to which the professional capabilities of the intelligent model match the capabilities required for a specific task type. The higher the capability matching degree, the better the quality of the intelligent model in handling that type of task. Processing capacity refers to the overall performance level of the intelligent model in handling a specific type of task, which can usually be measured by indicators such as processing speed, output accuracy, and contextual understanding ability. Unit processing cost is the cost incurred by the intelligent model in processing a unit number of task requests, calculated in a set number of tokens.
[0059] In specific embodiments, when obtaining the model degradation sequence, the intelligent model invocation method of this application can prioritize the matching degree between the intelligent model and the task type, placing models with higher matching degrees at the top of the sequence. For models with the same matching degree, the processing power of the intelligent model can be prioritized, placing models with stronger processing power at the top. For models with both the same matching degree and processing power, the unit processing cost of the intelligent model can be prioritized, placing models with lower unit processing costs at the top. In another embodiment of this application, for models with the same matching degree, the unit processing cost of the intelligent model can also be prioritized, placing models with lower unit processing costs at the top.
[0060] It is understood that the last position in the model degradation sequence is fixed as the local backup model, unaffected by the aforementioned sorting rules. The intelligent model invocation method of this application embodiment can select one or more factors as the sorting criteria according to actual needs, and can also adjust the weights of the three priorities as needed.
[0061] The intelligent model invocation method in this application prioritizes intelligent models suitable for handling certain tasks based on capability matching, ensuring the highest possible task execution quality. By introducing processing capacity and unit processing cost as sorting factors, it effectively improves response speed and operating costs while maintaining task quality. Furthermore, the sorting rules can be adjusted as needed to meet the differentiated requirements of different business scenarios regarding quality, speed, and cost. In a specific embodiment, the quality retention rate of the code generation task after degradation can be increased from 58% in the primary / backup switchover scheme to 92% in this application embodiment, an improvement of 59%, and the service quality fluctuation is reduced by 40%, effectively improving user experience.
[0062] like Figure 4 As shown, in one embodiment of this application, S102: Obtaining the corresponding task type of the current task request. The step of obtaining the corresponding task type of the current task request includes: S402: Based on the task type identifier in the current task request, obtain the corresponding task type of the current task request.
[0063] As can be seen, the task type identifier is a standardized identifier that is explicitly carried in the request parameters by the user's client when sending a task request, and is used to specify the task type. It can directly determine the type of the task request.
[0064] The intelligent model invocation method of this application embodiment, upon receiving a current task request, first checks whether the request parameters contain a task type identifier. If the request parameters contain a valid task type identifier, the task type corresponding to that identifier is directly used as the task type of the current task request, without requiring additional inference operations. The identified task type is then directly used in the subsequent model degradation sequence acquisition steps.
[0065] The intelligent model invocation method of this application embodiment can accurately and quickly determine the task type by explicitly identifying the task type, avoiding identification errors caused by automatic inference, effectively reducing computational overhead, improving response speed, and making it easier for the client to explicitly specify the task type according to its own business needs, thereby improving controllability.
[0066] like Figure 5 As shown, in another embodiment of this application, step S102: obtaining the corresponding task type of the current task request includes: S502: Based on the request characteristics in the current task request, infer the corresponding task type of the current task request.
[0067] Request characteristics are the feature information contained in the task request content that can reflect the task type, and may include keywords, data format, content structure, attachment type, etc. in the request content.
[0068] In the intelligent model invocation method of this application embodiment, after receiving the current task request, if the request parameters do not contain a task type identifier, the task type can be inferred based on the request features in the request content. Specifically, text information and attachment information in the request content can be extracted and the features contained therein analyzed; if the request content contains code blocks or programming-related keywords, the task request is inferred to be a code generation task; if the request content contains image attachments, the task request is inferred to be a multimodal understanding task; if the request content is an embedding vector generation request, the task request is inferred to be a text embedding task. For task requests that cannot be clearly inferred from the above features, they can be classified as general dialogue tasks. The inferred task type can be used in the subsequent model degradation sequence acquisition step.
[0069] Specifically, the intelligent model invocation method in this application embodiment can also employ a machine learning-based classifier, that is, a lightweight classification model is pre-trained, which can automatically identify the task type based on the request content; and a vector similarity-based matching method, that is, by vectorizing the request content and matching it with the predefined task type vector to obtain the corresponding task type.
[0070] Therefore, the intelligent model invocation method of this application embodiment can support requests without explicit task type identifiers, improve compatibility and ease of use, automatically identify common task types, effectively reduce the threshold for use, cover more use scenarios, and handle various forms of task requests.
[0071] In one embodiment of this application, the step of obtaining the next intelligent model in the model degradation sequence when the intelligent model fails to execute includes: checking the intelligent models in the model degradation sequence in turn until the first intelligent model that meets the preset working conditions or the local backup model is output.
[0072] That is, Figure 6 As shown, in one embodiment of this application, step S105: obtaining the next intelligent model in the model degradation sequence includes: S605: Check the intelligent models in the model degradation sequence one by one until the first intelligent model or local backup model that meets the preset working conditions is output.
[0073] The preset working conditions are used to determine whether the intelligent model can process task requests normally and stably. They can be used to filter out unhealthy or unusable models and prevent them from being downgraded to models that cannot work properly.
[0074] Specifically, the intelligent model invocation method in this embodiment can, after a certain intelligent model fails to execute a task request, start from the next position of the failed model in the degradation sequence and sequentially check whether each subsequent intelligent model meets the preset working conditions. For each intelligent model to be checked, the current state information of the model is obtained, and it is determined whether it meets the preset working conditions. If the model meets the preset working conditions, the model is selected as the next intelligent model to execute the task. If the model does not meet the preset working conditions, the model is skipped, and the next intelligent model in the degradation sequence is checked until the first intelligent model that meets the preset working conditions is found, or the last model in the degradation sequence, i.e., the local backup model, is checked. Regardless of whether the local backup model meets the preset working conditions, the model will be selected as the next intelligent model to execute the task.
[0075] The intelligent model invocation method in this application embodiment checks the working conditions of the model before degradation, avoiding forwarding task requests to unhealthy or unavailable models. This effectively prevents cascading failures in degradation, improves the success rate of degradation, reduces the average response time of task requests, and ensures system stability, preventing system avalanches caused by a single model failure. In a specific embodiment, the degradation secondary failure rate can be reduced from 15% in existing solutions to 2%, a reduction of 87%.
[0076] In one embodiment of this application, the preset working conditions include the intelligent model's health score meeting a preset threshold. The health score is a numerical indicator that comprehensively evaluates the intelligent model's operating status and availability, and can be calculated in real time based on indicators such as the model's historical call success rate, response time, and failure rate. The preset threshold is a pre-set standard for judging the health score; when the intelligent model's health score is lower than this threshold, the model is in an unhealthy state and is not suitable for processing task requests.
[0077] Specifically, the intelligent model invocation method in this application embodiment can monitor the running status of each intelligent model in real time, and calculate and update the health score of each model based on the historical invocation data of the intelligent models. When it is necessary to check whether an intelligent model meets the preset working conditions, the current health score of the intelligent model can be obtained first and compared with a preset threshold. If the health score of the model is greater than or equal to the preset threshold, the model meets the preset working conditions and can be used to process task requests. If the health score of the model is less than the preset threshold, the model is in an unhealthy state and is not suitable for processing task requests. The model needs to be skipped and the next intelligent model needs to be checked.
[0078] The intelligent model invocation method in this application embodiment quantifies and evaluates the running status of the model through health scoring. It can accurately identify models with degraded or unstable performance, filter out unhealthy models in advance, reduce the probability of downgrade failure, and also reflect the status changes of the intelligent model in a timely manner by updating the health score in real time, thereby improving the accuracy of downgrade decisions.
[0079] In one embodiment of this application, the health score is configured to be obtained based on the execution success rate of the smart model and / or the execution latency of the smart model within a set time window.
[0080] Setting a time window as a continuous period of time for statistical intelligent model performance metrics can be achieved using a sliding window mechanism, effectively smoothing short-term data fluctuations and accurately reflecting the recent operational status of the intelligent model. Execution latency is the time interval from the start of a task request to the return of the execution result, which can be used to quantify the model's response speed and load pressure.
[0081] Specifically, the intelligent model invocation method in this application embodiment can statistically analyze the operational metrics of each intelligent model, record the results and time consumption information of each intelligent model's task request execution, and periodically calculate the health score of each intelligent model based on the statistical data within the sliding time window. In the calculation of the health score, the execution success rate can be used alone as the calculation basis, the execution latency can be used alone as the calculation basis, or the execution success rate and execution latency can be combined for a comprehensive calculation.
[0082] The intelligent model invocation method of this application embodiment can statistically analyze operating indicators through a sliding time window mechanism, effectively filter the impact of a single abnormal request on the health score, improve the accuracy of health status assessment, and comprehensively and objectively reflect the availability and performance status of the intelligent model by combining two dimensions of indicators: execution success rate and execution latency. It can also periodically update the health score, promptly detect changes in the status of the intelligent model, provide reliable real-time basis for degradation decisions, and further reduce the probability of degradation failure.
[0083] In one embodiment of this application, the preset working conditions include the intelligent model's working state being a normal state, which includes a normal state, a circuit breaker state, and a half-open state. The intelligent model invocation method further includes: when the intelligent model in the normal state reaches the preset circuit breaker condition, switching the current working state of the intelligent model to the circuit breaker state; when the waiting time of the intelligent model in the circuit breaker state meets the preset half-open condition, switching the current working state of the intelligent model to the half-open state; and when the intelligent model in the half-open state meets the preset recovery condition, switching the state of the intelligent model back to the normal state.
[0084] The working status is a status identifier maintained for each intelligent model, used to control the request receiving permissions of the intelligent model and prevent faulty models from receiving too many requests, which could lead to the spread of the fault. The normal state is when the intelligent model can normally receive and process all task requests. The circuit breaker state is when the intelligent model is temporarily prohibited from receiving all requests. The half-open state is a transitional state during the recovery process of the intelligent model from the circuit breaker state.
[0085] In this embodiment of the intelligent model invocation method, when checking whether the intelligent model meets the preset working conditions, the system first checks the working state of the model. If the working state of the model is normal, the system considers the model to meet the preset working conditions. If the working state of the model is in a circuit breaker state or a half-open state, the system considers the model not to meet the preset working conditions.
[0086] Furthermore, the intelligent model can switch between normal, circuit breaker, and half-open states. Specifically, when an intelligent model in the normal state reaches a preset circuit breaker condition, its operating state is switched to the circuit breaker state, and any task requests are refused to be forwarded to that model. When the model is in the circuit breaker state for the time specified by a preset half-open condition, its operating state is switched to the half-open state, allowing a limited number of probe requests to be forwarded to that model. When the intelligent model in the half-open state successfully processes all probe requests and meets the preset recovery condition, its operating state is switched back to the normal state, and it resumes receiving all task requests. If a probe request fails to execute in the half-open state, the model's operating state is switched back to the circuit breaker state.
[0087] The intelligent model invocation method of this application embodiment can effectively prevent the faulty model from receiving too many requests through the circuit breaker mechanism, avoid fault propagation and system avalanche, realize automatic detection and isolation of intelligent model faults, improve system stability, and realize automatic recovery of intelligent models through the half-open state detection mechanism without manual intervention.
[0088] In one embodiment of this application, the preset circuit breaker condition includes the number of consecutive failures in executing a task request reaching a set failure threshold. The intelligent model invocation method in this embodiment further includes: resetting the consecutive failure count of the intelligent model if the intelligent model successfully executes the current task request. The consecutive failure count is the cumulative number of times the intelligent model has consecutively failed to process the task request. The set failure threshold is a pre-set upper limit for the number of consecutive failures that triggers the circuit breaker.
[0089] In the intelligent model invocation method of this application embodiment, when the intelligent model fails to execute a task request, the number of consecutive failures of the model can be increased. When the number of consecutive failures reaches a set failure threshold, a circuit breaker can be triggered, and the working state of the intelligent model is changed to a circuit breaker state. When the intelligent model successfully executes a task request, the number of consecutive failures of the model can be reset to zero.
[0090] In one embodiment of this application, the preset circuit breaker condition includes the failure rate of the task request reaching a set failure rate threshold; the failure rate is the ratio of the number of failed task requests processed by the intelligent model within a specified time window to the total number of requests; the set failure rate threshold is a preset upper limit of the failure rate that triggers the circuit breaker.
[0091] That is, the intelligent model invocation method in this application embodiment can trigger the circuit breaker of the intelligent model if the failure rate of the intelligent model reaches a set failure rate threshold within a specified statistical time window, based on the failure rate circuit breaker condition.
[0092] In one embodiment of this application, the preset half-open condition includes a waiting time that meets the waiting interval duration; that is, the waiting interval duration is the duration during which the intelligent model is in the circuit breaker state, and it automatically enters the half-open state after reaching this duration. In the intelligent model invocation method of this application embodiment, when the intelligent model enters the circuit breaker state, when the duration of the circuit breaker state reaches the set waiting interval duration, the working state of the model can be converted to the half-open state.
[0093] In one embodiment of this application, the preset recovery condition includes the success of executing a preset number of probe requests consecutively. In the half-open state, a preset number of probe requests can be sent to the intelligent model. If all probe requests are executed successfully, the model has recovered and its operating state can be switched back to the normal state. If any probe request fails, the model's operating state can be switched back to the circuit breaker state, and the timing can be restarted.
[0094] Therefore, the intelligent model invocation method of this application embodiment can provide multiple circuit breaker triggering conditions, flexibly adapt to different fault scenarios, improve the accuracy of circuit breaker triggering by dual judgment of consecutive failure count and failure rate, realize the automated operation of the circuit breaker through clear state transition conditions, and safely verify whether the intelligent model has recovered to normal through the detection request mechanism, avoiding secondary faults caused by directly restoring a large number of requests.
[0095] It is understood that in another embodiment of this application, the health status of the model can be actively detected by periodically sending probe requests to the intelligent model. The process is similar and will not be described in detail here.
[0096] In one embodiment of this application, the termination condition further includes the number of failures of the failed intelligent model reaching a set retry threshold. The set retry threshold is a pre-defined maximum number of failures of the intelligent model allowed to be attempted during the degradation process of a single task request, and can be used to limit the duration of the degradation process.
[0097] The intelligent model invocation method of this application can increment the failure count when an intelligent model fails to execute a task request, and check whether the failure count has reached a set retry threshold before each attempt to the next intelligent model. If the failure count counter has not reached the set retry threshold, the next intelligent model is tried; if the failure count counter has reached the set retry threshold, the degradation process is terminated and no further attempts are made to the subsequent intelligent models.
[0098] The intelligent model invocation method in this application embodiment can avoid excessive task request response time caused by trying too many models by limiting the number of retries during the degradation process, thus ensuring system response efficiency and improving user experience; and preventing system performance degradation caused by a single task request consuming too many system resources.
[0099] In one embodiment of this application, the intelligent model invocation method further includes: If all smart models fail to execute in the model degradation sequence, an execution exception will be output.
[0100] An execution exception is a standardized error response returned when all intelligent models are unable to complete the task request. It includes an error code, error description, and a unique identifier for the request.
[0101] The intelligent model invocation method in this application embodiment can output an execution exception when all intelligent models in the model degradation sequence fail, i.e., the last local backup model also fails. The execution exception response includes a standardized error code to indicate the error type, a detailed error description to explain the reason for the failure, and may also include a unique identifier for this task request for subsequent troubleshooting and log analysis. After returning the generated execution exception response to the requesting client, detailed log information for this failure can also be recorded.
[0102] Therefore, the intelligent model invocation method in this application embodiment can clearly inform the client of the result and reason for the task execution failure, which facilitates the client to perform subsequent error handling, provides a standardized error response format, facilitates the client to perform unified exception handling, and facilitates system administrators to troubleshoot and locate faults by requesting a unique identifier.
[0103] In one embodiment of this application, the smart model invocation method further includes: If all smart models fail in the model degradation sequence, a preset default response is output. The preset default response is a standardized response content pre-configured for different task types, returned to the client when all smart models are unable to complete the task request, to ensure that there is always a response output.
[0104] The intelligent model invocation method of this application embodiment can pre-configure a corresponding preset default response for each supported task type. When all intelligent models in the model degradation sequence fail to execute, the corresponding preset default response is obtained according to the task type of the current task request, and the obtained preset default response is returned to the client that initiated the request, while simultaneously recording relevant log information. The content of the preset default response can be different for different task types. For example, for dialogue tasks, the preset default response is a message indicating that the service is temporarily unavailable; for text embedding tasks, the preset default response is a zero vector of a specified dimension.
[0105] The intelligent model invocation method in this application embodiment can prevent the client from seeing the original error information, improve the user experience, ensure that there is always a response output, avoid the client from encountering anomalies due to no response, and make the default response more in line with business needs by configuring different default responses for different task types, thereby reducing the impact on the client's business logic.
[0106] To verify the technical effects of the embodiments of this application, comparative experiments were conducted: The experimental environment is as follows: Access models: 4 cloud-based API services, 2 local deployment models Task type distribution: Code generation (30%), general dialogue (50%), multimodal understanding (20%) Test duration: 7 days, total of 1,200,000 requests processed. Comparison Solution: Simple Primary / Backup Switchover (Fixed Priority Degradation) Experimental results show that the embodiments of this application have significant improvements in core indicators such as quality maintenance during downgrades, downgrade success rate, and service stability.
[0107] This application also provides an intelligent model invocation system, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned intelligent model invocation method.
[0108] The intelligent model invocation system can be a computing gateway deployed on the client side or a gateway server located on the server side. The device includes memory and a processor at the hardware level. The memory stores a software system containing computer programs that encode all the logical steps implementing the aforementioned intelligent model invocation method.
[0109] It should be noted that the elements described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately.
[0110] It should be understood that multiple components and / or parts can be provided by a single integrated component or part. Alternatively, a single integrated component or part can be divided into multiple separate components and / or parts. The use of the public designation "a" or "an" to describe a component or part does not exclude other components or parts.
[0111] It should be understood that while terms such as "first" or "second" may be used in this application to describe various elements, these elements are not limited by these terms; these terms are merely used to distinguish one element from another. The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms "a," "the," and "the" as used in one or more embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.
[0112] In this document, terms such as "upper," "lower," "front," "back," "left," and "right" are used only to indicate the relative positional relationship between related parts, and not to limit the absolute position of these related parts. In this document, terms such as "equal" and "same" are not strict mathematical and / or geometric limitations, but also include errors that are understandable to those skilled in the art and permissible in manufacturing or use. Unless otherwise stated, the numerical ranges in this document include not only the entire range within its two endpoints, but also several sub-ranges contained therein. The basic principles of this application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this application. Furthermore, the specific details disclosed above are only for illustrative and facilitative purposes, and are not limitations. The above details do not limit the application to the necessity of adopting the above specific details for implementation.
[0113] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for invoking an intelligent model, characterized in that, include: Get the current task request; Obtain the corresponding task type of the current task request; Based on the corresponding task type of the current task request, obtain the corresponding model degradation sequence. The model degradation sequence includes at least two intelligent models arranged in order, and the end of the model degradation sequence is a local backup model. The task request is executed using the smart model that is ranked first in the model degradation sequence, and the execution result of the smart model is obtained. If the intelligent model fails to execute, obtain the next intelligent model in the model degradation sequence; The task request is executed using the next intelligent model in the model degradation sequence. If the intelligent model fails to execute the task, the next intelligent model in the model degradation sequence is obtained until the termination condition is met. The termination condition includes either the intelligent model successfully executing the current task request or the local backup model failing to execute the current task request.
2. The intelligent model invocation method according to claim 1, characterized in that, The step of obtaining the corresponding model degradation sequence based on the corresponding task type of the current task request includes: Based on the identified task type corresponding to the current task request, the corresponding model degradation sequence is obtained from the pre-stored task type and model degradation sequence mapping database.
3. The intelligent model invocation method according to claim 2, characterized in that, The model degradation sequence is configured to arrange the intelligent models in descending order based on at least one of the following: the ability matching degree with the task type, the processing capacity of the intelligent model, and the unit processing cost of the intelligent model.
4. The intelligent model invocation method according to claim 2, characterized in that, The step of obtaining the corresponding model degradation sequence based on the corresponding task type of the current task request further includes: Based on preset indicators, the order of intelligent models in the model degradation sequence is adjusted and rearranged. The preset indicators include the health score of the intelligent model and / or the execution success rate of the intelligent model.
5. The intelligent model invocation method according to claim 1, characterized in that, The steps for obtaining the corresponding task type of the current task request include: Based on the task type identifier in the current task request, obtain the corresponding task type of the current task request.
6. The intelligent model invocation method according to claim 1, characterized in that, The steps for obtaining the corresponding task type of the current task request include: Based on the request characteristics in the current task request, the corresponding task type of the current task request is inferred.
7. The intelligent model invocation method according to claim 1, characterized in that, In the event that the intelligent model fails to execute, the step of obtaining the next intelligent model in the model degradation sequence includes: If the intelligent model fails to execute, the intelligent models in the model degradation sequence are checked in turn until the first intelligent model that meets the preset working conditions or the local backup model is output.
8. The intelligent model invocation method according to claim 7, characterized in that, The preset working conditions include the health score of the intelligent model meeting a preset threshold.
9. The intelligent model invocation method according to claim 8, characterized in that, The health score is configured to be obtained based on the execution success rate of the intelligent model and / or the execution latency of the intelligent model within a set time window.
10. The intelligent model invocation method according to claim 7, characterized in that, The preset working conditions include the working state of the intelligent model being normal, and the working state includes normal state, circuit breaker state, and half-open state. The intelligent model invocation method also includes: If the intelligent model in normal condition reaches the preset circuit breaker condition, the current working state of the intelligent model will be switched to the circuit breaker state. If the waiting time of the intelligent model in the circuit-breaker state meets the preset half-open condition, the current working state of the intelligent model will be switched to the half-open state. If the smart model in the half-open state meets the preset recovery conditions, the state of the smart model will be converted to the normal state.
11. The intelligent model invocation method according to claim 10, characterized in that, The preset circuit breaker condition includes the number of consecutive failures of the task request reaching a set failure threshold; The intelligent model invocation method further includes: if the result of the intelligent model executing the current task request is successful, resetting the consecutive failure count of the intelligent model; And / or, the preset circuit breaker condition includes the failure rate of the task request reaching a set failure rate threshold; And / or, the preset half-open condition includes a waiting time that meets the waiting interval duration; And / or, the preset recovery condition includes the result of successfully executing a preset number of consecutive probe requests.
12. The intelligent model invocation method according to any one of claims 1 to 11, characterized in that, The termination condition also includes the number of failed attempts of the intelligent model reaching a set retry threshold.
13. The intelligent model invocation method according to any one of claims 1 to 11, characterized in that, The intelligent model invocation method further includes: outputting an execution exception if all intelligent models in the model degradation sequence fail to execute.
14. The intelligent model invocation method according to any one of claims 1 to 11, characterized in that, The intelligent model invocation method further includes: outputting a preset default response when all intelligent models fail to execute in the model degradation sequence.
15. An intelligent model invocation system, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the intelligent model invocation method according to any one of claims 1 to 14.