Method and device for selecting optimal service provider large model of API (Application Program Interface) system

Abstract

The invention provides an optimal service provider large model selection method and device for an API system, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the fusing filtering, concurrent hard current limiting filtering and weighted random routing filtering of an initial service provider large model set according to model identification data and state data, and obtaining a preferred service provider large model; calling the first-choice service provider large model to a server of the API system, and obtaining a calling state of the first-choice service provider large model; when the calling state is calling failure, rejecting the preferred service provider large model from the initial service provider large model set, constructing a degraded service provider large model set based on the remaining service provider large models, and performing fusing filtering, concurrent hard current limiting filtering and weighted random routing filtering on the degraded service provider large model set to obtain a degraded service provider large model set; re-screening a first-choice service provider large model; when the calling state is calling success, taking the preferred service provider large model as an optimal service provider large model; and generating an answer text of the question text through the optimal service provider large model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for selecting the optimal service provider model for an API system. Background Technology

[0002] With the rapid development of artificial intelligence technology, numerous large models with different functions and performance have emerged in the market. When developing API systems, enterprises or developers often need to connect to multiple heterogeneous service provider large models to meet diverse business needs. However, the existing methods for selecting service provider large models have many problems that need to be solved, which makes it impossible for API systems to select the optimal service provider large model, ultimately resulting in inaccurate generated answer text.

[0003] Existing methods for selecting large service provider models lack effective screening mechanisms, leading to significant differences in the stability of these models. Some models are prone to high-frequency failures and concurrency overload, directly impacting the continuity of downstream business. Furthermore, existing methods lack efficient fault degradation mechanisms, often resorting to blind retries or manual switching after call failures. This wastes resources and fails to achieve smooth degradation, resulting in a noticeable user interruption and insufficient fault tolerance for API systems in multi-service provider large model collaboration scenarios. Summary of the Invention

[0004] To address the aforementioned problems, in a first aspect, this invention provides a method for selecting the optimal service provider model for an API system, comprising: After receiving a client's request, the API system's gateway obtains the model identifier data from the request, retrieves the initial service provider's large model set from the API system's configuration center based on the model identifier data, and obtains the status data of the initial service provider's large model set through the API system's status monitor. Based on the model identification data and status data, circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are performed on the initial service provider large model set to obtain the preferred service provider large model; The preferred service provider's large model is invoked to the API system's server to obtain the invocation status. If the invocation status indicates failure, the preferred service provider's large model is removed from the initial set of service provider large models. A set of degraded service provider large models is constructed based on the remaining service provider large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the degraded service provider large model set to re-select the preferred service provider's large model. If the invocation status indicates success, the preferred service provider's large model is selected as the optimal service provider's large model. Obtain the client's question text and generate the answer text using the optimal service provider's large model.

[0005] Optionally, the model identification data includes static weights W(P) i ), preset maximum concurrency M (P) i ) and demotion order O(P) i ); Status data includes the current concurrency C(P) i ), consecutive failure count F(P) i ) and the final failure time T L (P i ); where i represents the service provider's large model number, P i Let i represent the large model of the i-th service provider.

[0006] Optionally, the process of applying circuit breaker filtering to the initial large set of service providers includes: Get the current time T now Circuit breaker recovery time T R and the maximum consecutive failure threshold F max ; If the current time T now With service provider large model P i The final failure time T L (P i The difference is less than the circuit breaker recovery time T. R And the service provider's large model P i The consecutive failure count F(P) i () is greater than or equal to the maximum consecutive failure threshold F max Then determine the service provider's large model P i In the circuit breaker state, the service provider's large model P will be activated. i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

[0007] Optionally, the process of performing concurrent hard rate limiting filtering on the initial large set of service providers includes: Obtain the large model P of the service provider i The preset maximum concurrency M (P) i ) and the current concurrency C(P) i ); If the current concurrency level C(P) i ) is greater than or equal to the preset maximum concurrency M(P) i If the service provider's large model P is determined, then... i In a congested state, the service provider's large model P i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

[0008] Optionally, the step of performing weighted random route filtering on the initial large set of service provider models to obtain the preferred large set of service provider models includes: Obtain the static weights of the large models from each service provider, and sum the static weights of the large models from each service provider to obtain the total weight W. total From 0 to the total weight W total Choose a random integer R from between; The static weights are accumulated sequentially according to the downgrade order of each service provider's large model to obtain the accumulated static weight S; When the accumulated static weight S is greater than the random integer R, the current service provider's large model is selected as the preferred service provider's large model.

[0009] Optionally, when the call status is "call failed," the preferred service provider's large model is removed from the initial service provider's large model set, and a degraded service provider's large model set is constructed based on the remaining service provider's large models, including: When the call status is "call failed", retrieve the response exception type; If the response exception type is a business layer error or content security block, an error message is sent to the client and the process ends; If the response exception type is a network layer error, server error, or streaming interruption, the preferred service provider large model is moved from the initial service provider large model set to the exclusion list. The remaining service provider large models in the initial service provider large model set are sorted in ascending order according to the degradation order. The remaining service provider large models after the ascending order are used to form the degraded service provider large model set.

[0010] Optionally, the step of generating the answer text from the question text using the optimal service provider's large model includes: Obtain the API protocol format of the optimal service provider's large model, convert the question text into the API protocol format through the API system's protocol adaptation layer to obtain the optimized question text, input the optimized question text into the optimal service provider's large model to generate the answer text, and obtain the answer text.

[0011] Secondly, the present invention provides an optimal service provider large-scale model selection device for an API system, used to implement the optimal service provider large-scale model selection method for the API system, the device comprising: The data acquisition module is used by the API system's gateway to obtain the model identification data in the request after receiving a client's request. Based on the model identification data, it obtains the initial service provider's large model set from the API system's configuration center and obtains the status data of the initial service provider's large model set through the API system's status monitor. The preferred service provider large model acquisition module is used to perform circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering on the initial service provider large model set based on model identification data and status data, so as to obtain the preferred service provider large model. The optimal service provider large model acquisition module is used to call the preferred service provider large model to the API system's server and obtain the call status of the preferred service provider large model. When the call status is "call failed," the preferred service provider large model is removed from the initial service provider large model set. A fallback service provider large model set is constructed based on the remaining service provider large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the fallback service provider large model set to re-select the preferred service provider large model. When the call status is "call successful," the preferred service provider large model is taken as the optimal service provider large model. The answer text acquisition module is used to acquire the client's question text and generate the answer text for the question text using the optimal service provider's big data model.

[0012] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the optimal service provider big model selection method of the API system.

[0013] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the optimal service provider large model selection method of the API system.

[0014] The present invention has the following beneficial effects: 1. By filtering out high-frequency service provider models through circuit breaking and shielding congested service provider models through concurrent hard rate limiting, requests are avoided from being sent to unstable nodes at the source, reducing call failures and timeouts, and significantly improving service reliability. Load balancing is achieved through weighted random routing filtering. Weighted random routing filtering combined with degradation order allows high-weight service provider models to get more call opportunities, while retaining randomness to avoid single point overload. When the first-choice service provider model fails, a set of degraded service provider models is automatically built to re-select the first-choice service provider model, achieving smooth degradation of service provider model failures, ensuring the continuity of service provider model operation, improving the fault tolerance of service provider models, and reducing the user's perceived interruption.

[0015] 2. Business layer errors and content security interceptions are classified as unrecoverable errors, while network layer errors, server-side errors, and streaming interruptions are classified as degradable errors. By distinguishing between unrecoverable and degradable errors, blind retries for invalid scenarios are avoided, reducing resource waste. Degradable errors automatically switch to alternative service provider models without manual intervention, ensuring process continuity. Failed service provider models are moved to an exclusion list to avoid repeated calls that increase their load in a short period of time, while reducing the consumption of invalid system requests. A set is built according to the degradation order, prioritizing highly available alternatives to reduce the risk of secondary failures. For degradable errors, the background automatically switches to service provider models, requiring no manual retries from users, resulting in almost imperceptible interruptions and achieving smooth failure degradation of service provider models.

[0016] 3. By shielding heterogeneous characteristics through the protocol adaptation layer, adding or replacing service provider models only requires configuring protocol information, significantly reducing system integration and expansion costs, breaking down heterogeneous protocol barriers, and enabling plug-and-play functionality for service provider models. Clients only need to submit request text according to a unified standard, without needing to concern themselves with the protocol details of the underlying service provider models, reducing repetitive development work for adapting multiple service provider models. Subsequent protocol rule optimizations only require modifying the adaptation layer configuration, without affecting client logic, improving development and maintenance efficiency. The adaptation layer converts the original results returned by the service provider models into a unified system format, eliminating the need for clients to handle differences in results from different service provider models, thus improving the accuracy of the service provider models' response text. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0020] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0021] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0022] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Reference Figure 1 This invention provides a method for selecting the optimal service provider model for an API system, comprising: After receiving a request from a client, the gateway of the S100API system obtains the model identification data in the request, retrieves the initial service provider's large model set from the API system's configuration center based on the model identification data, and obtains the status data of the initial service provider's large model set through the API system's status monitor.

[0024] In some embodiments, the model identification data includes static weights W(P) i ), preset maximum concurrency M (P) i ) and demotion order O(P) i ); Status data includes the current concurrency C(P) i ), consecutive failure count F(P) i ) and the final failure time T L (P i ); where i represents the service provider's large model number, P i Let i represent the large model of the i-th service provider.

[0025] In some embodiments, monitoring data is obtained through a gateway deployed between the client and the server. Upon receiving a request from the client, the gateway parses the model identifier in the request and reads the initial service provider's large model set P={P1, P2, ..., P...} corresponding to that model identifier from the configuration center. nThe system atomically reads monitoring data from all current service provider models, including configuration and status data, through the status monitor.

[0026] The configuration data consists of predefined static parameters, set by the system administrator based on the service provider's large-scale model capabilities, cooperation agreements, etc., and remain valid indefinitely after being entered into the system configuration center.

[0027] Static weights W(P) i The value range for ) is 1-10, with higher weights indicating higher priority for the service provider's large model. The criteria for setting this weight include: the service provider's large model's response speed, accuracy, service fees, and cooperation level. For example: Service provider large model P1 has fast response and high accuracy, so its weight is set to 10; service provider large model P2 has medium performance, so its weight is set to 6; service provider large model P3 meets basic standards, so its weight is set to 3.

[0028] Preset maximum concurrency M(P) i The parameters are set based on the hardware resources, interface capacity, and cooperation agreement of the service provider's large-scale model. For example, if service provider large-scale model P1 deploys 10 high-performance servers and supports a maximum concurrency of 500, then M(P1) = 500; service provider large-scale model P2 has limited resources and supports a maximum concurrency of 100, then M(P2) = 100. The system will use this parameter as a threshold for concurrency control to prevent the service provider's large-scale model from refusing service due to overload.

[0029] Demotion order O(P) i The value of O(P1) is a consecutive positive integer, with smaller numbers indicating higher priority for degradation. The settings are based on the service provider's historical stability data and service availability commitments. For example: P1 has a 30-day failure rate of 0.5%, O(P1) = 1; P2 has a failure rate of 3%, O(P2) = 2; P3 has a failure rate of 5%, O(P3) = 3.

[0030] The status data consists of parameters collected in real time and synchronized once per second by the system monitoring module, reflecting the current operating status of the service provider's large model.

[0031] Current concurrency C(P) i The system uses a request counter to track requests in real time. For each new request received by the service provider's large model, the counter increments by 1; after a request is completed, the counter decrements by 1. For example, if service provider large model P1 is currently processing 320 requests, then C(P1) = 320. If a new request arrives, the counter is updated to 321; if 5 requests complete simultaneously, the counter is updated to 316. The system uses this parameter to determine whether the service provider's large model is in a congested state.

[0032] Continuous failure count F(P) iThe initial value of F(P1) is 0. It is reset to 0 on a successful call and automatically incremented by 1 on a failed call. For example, if P1 fails to call 3 times consecutively, F(P1) = 3; if the 4th call succeeds, F(P1) is reset to 0; if it fails again, F(P1) starts accumulating from 1 again. This parameter is used to trigger the circuit breaker mechanism.

[0033] Last failure time T L (P i The system uses a timestamp format to record the system time of the most recent failed call to the service provider's large model. If subsequent calls succeed, this parameter remains unchanged until the next failure. The system uses this parameter to calculate the difference between the current time and the previous time to determine whether the circuit breaker recovery conditions are met.

[0034] S200 performs circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering on the initial large set of service provider models based on model identification data and status data to obtain the preferred service provider large model.

[0035] In some embodiments, the process of applying circuit breaker filtering to the initial service provider's large model set includes: Get the current time T now Circuit breaker recovery time T R and the maximum consecutive failure threshold F max ; If the current time T now With service provider large model P i The final failure time T L (P i The difference is less than the circuit breaker recovery time T. R And the service provider's large model P i The consecutive failure count F(P) i () is greater than or equal to the maximum consecutive failure threshold F max Then determine the service provider's large model P i In the circuit breaker state, the service provider's large model P will be activated. i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

[0036] In some embodiments, the core of circuit breaker filtering is to avoid continuously sending requests to unstable service providers by judging the failure frequency and recovery time in a two-dimensional manner.

[0037] Circuit breaker recovery time T R and the maximum consecutive failure threshold F max Configured by the system administrator according to business needs, stored in the global configuration center, and supports dynamic adjustment. Among them, T... R The value ranges from 10 seconds to 5 minutes, and is used to define the cooling time after a service provider's large model experiences a circuit breaker; F maxThe value ranges from 3 to 10 times and is used to define the threshold for frequent failures.

[0038] The system monitoring module synchronizes two key data points every second: ① Current time T now ② Service Provider Large Model P i Status data, including consecutive failure count F(P) i ) and the final failure time T L (P i After collection, a data verification mechanism is used to ensure that there are no empty values ​​and that the timestamps are valid.

[0039] If the service provider's large model P i If there are no failure records, F(Pi)=0 and TL(Pi) is empty, it is directly judged as a normal state and retained in the candidate service provider large model set; if there are failure records, first calculate the time difference ΔT=T now -T L (P i Then compare the two conditions; when ΔT <T R And F(P) i )≥F max Then it is determined to be in a circuit breaker state, and P is immediately activated. i Remove them from the initial large set of service provider models and mark them with a circuit breaker flag; otherwise, determine that they are in a serviceable state and retain them in the large set of candidate service provider models. Also, if ΔT ≥ T... R Automatically F(P) i Reset to 0.

[0040] The large model P of the service provider whose circuit breaker was suspended i When ΔT≥T R The circuit breaker flag will then be automatically removed, and the service provider will be rejoined in the initial large model set to participate in the next round of selection. i Reset to 0, T L (P i Preserve the time of the last failure in history.

[0041] In some embodiments, the process of performing concurrent hard rate limiting filtering on the initial service provider's large model set includes: Obtain the large model P of the service provider i The preset maximum concurrency M (P) i ) and the current concurrency C(P) i ); If the current concurrency level C(P) i ) is greater than or equal to the preset maximum concurrency M(P) i If the service provider's large model P is determined, then... i In a congested state, the service provider's large model P i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

[0042] In some embodiments, the core of concurrent hard rate limiting filtering is to accurately determine the load status of the service provider's large model by using a preset threshold and real-time concurrency count, so as to avoid service avalanche caused by request overload.

[0043] Iterate through the initial service provider's large model set, for each P i Obtain M(P) respectively i ) and real-time C(P) i If C(Pi) ≥ M(Pi), then it is determined to be a congested state, and P is immediately... i Remove the congested service provider from the initial large model set and mark it as temporarily unavailable; subsequent requests will not be routed to this service provider's large model for the time being. The system checks the C(P) of the congested service provider's large model every 10 seconds. i When C(P) i ) <M(P i When C(P) × 80%, the temporarily unavailable flag is automatically removed and the element is re-added to the set. i ) <M(P i If the condition is met, it is determined to be in an available state and retained in the candidate service provider large model set, while the current concurrency percentage C(P) is recorded. i ) / M(P i This is for subsequent optimization and analysis.

[0044] If the service provider's large model P i To adjust capacity due to expansion or reduction, administrators can modify M(P) through the configuration center. i The system synchronizes new thresholds and recalculates concurrency ratios in real time without requiring a service restart. For example, if P2 is expanded to support 200 concurrent users and M(P2) is updated to 200, and the current C(P2) = 150 (150 < 200), then P2 will be automatically added back to the candidate set.

[0045] In some embodiments, the step of performing weighted random route filtering on the initial large set of service provider models to obtain the preferred large set of service provider models includes: Obtain the static weights of the large models from each service provider, and sum the static weights of the large models from each service provider to obtain the total weight W. total From 0 to the total weight W total Choose a random integer R from between; The static weights are accumulated sequentially according to the downgrade order of each service provider's large model to obtain the accumulated static weight S; When the accumulated static weight S is greater than the random integer R, the current service provider's large model is selected as the preferred service provider's large model.

[0046] In some embodiments, the core of weighted random routing filtering is weight probability allocation plus degradation order constraint, which ensures that high-priority service providers and large models get more calling opportunities while also taking into account the orderliness of the filtering.

[0047] Extract all P from the candidate service provider large model set from the system configuration center. i static weights W(P) i ); Execute the total weight accumulation logic W total =W(P1)+W(P2)+...+W(P n ), where n is the number of candidate service provider models. If the candidate set is empty, the degradation process is triggered directly; if W total =0, by default, the service provider with the largest model ranked first is selected as the first choice according to the downgrade order.

[0048] The system employs a built-in encrypted pseudo-random number generator to generate random numbers in the closed interval (0, W). total Select a unique random integer R within the range to ensure that the probability of each value being selected is evenly distributed; record the R value after generation. If generation fails, the default value is R=0, and directly match the first service provider large model whose cumulative sum exceeds 0.

[0049] The candidate service provider model is ranked in descending order of O(P) i Sort in ascending order, O(P) i The smaller the value of ), the earlier it appears, ensuring priority verification of the high-availability service provider's large model. After sorting, the set order is kept fixed. Initialize the accumulated static weight S=0, and traverse the service provider's large model one by one in the sorted order. Traverse to the current service provider's large model P. i hour, Initially, S=0, and the S of the first service provider's large model is directly equal to its own weight; if S>R, the traversal is terminated immediately, and the current service provider's large model is marked as the preferred service provider's large model; if S≤R, the traversal of the next service provider's large model continues, and the accumulation and comparison logic is repeated.

[0050] In determining the preferred service provider model P target Then, perform the following operations to ensure state consistency: Pre-deduction: Atomically deducting P target The concurrent counter increments by -, C(P) target )←C(P target )+1.

[0051] Request execution: Forward the request to the protocol adaptation layer and call P. target API.

[0052] Post-processing: If the call is successful, C(P) is executed atomically. target )←C(P target )—1, and reset F(P) target)←0; If the call fails, C(P) is executed atomically. target )←C(P target )—1, increasing F(P) target )←F(P target +1, update T L (P target ) ←T now And immediately trigger a downgrade retry.

[0053] S300 calls the preferred service provider's large model to the API system's server to obtain the call status of the preferred service provider's large model. When the call status is "call failed," the preferred service provider's large model is removed from the initial set of service provider's large models. A set of degraded service provider's large models is constructed based on the remaining service provider's large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the set of degraded service provider's large models to re-select the preferred service provider's large model. When the call status is "call successful," the preferred service provider's large model is taken as the optimal service provider's large model.

[0054] In some embodiments, when the call status is "call failed," removing the preferred service provider's large model from the initial service provider's large model set and constructing a degraded service provider's large model set based on the remaining service provider's large models includes: When the call status is "call failed", retrieve the response exception type; If the response exception type is a business layer error or content security block, an error message is sent to the client and the process ends; If the response exception type is a network layer error, server error, or streaming interruption, the preferred service provider large model is moved from the initial service provider large model set to the exclusion list. The remaining service provider large models in the initial service provider large model set are sorted in ascending order according to the degradation order. The remaining service provider large models after the ascending order are used to form the degraded service provider large model set.

[0055] In some embodiments, five types of exceptions and judgment criteria are preset, and they are automatically identified by parsing the status code and error information fields of the call response.

[0056] The response codes for business layer errors are 400 and 404, and the error messages contain keywords such as invalid parameters and business rule restrictions, which are caused by the client request not meeting the service provider's large model business requirements; The response code for content security blocking is 403. The error message contains keywords such as content violation and security blocking, which is caused by the request text triggering the service provider's large-scale content review rules. The response code for a network layer error is 503. The error message may include "no response after timeout" or other network error messages such as "connection timeout" or "DNS resolution failure". The server-side error response codes are 500 and 502, which are caused by a system failure in the service provider's large model itself. A streaming interruption occurs when data transmission is interrupted or data packets are lost during the streaming response process, and transmission is not resumed within 1 second.

[0057] If the error is identified as a business layer error or content security interception, it is determined that an unrecoverable error has occurred. The system automatically generates a structured error message, including the error type, specific cause, and rectification suggestions. The error message is returned to the client via the HTTP response body, and the error log is recorded. The optimal service provider model selection process is terminated directly, and no further downgrade operations are performed.

[0058] If the error is identified as a network layer error, server error, or streaming interruption, it is determined that a recoverable error has occurred, and the fault self-healing degradation process is executed.

[0059] Step 1: Eliminate the failed service provider model, and select the preferred service provider model P that is currently failing to be called. i Removed from the initial large set of service providers' models and moved to a temporary exclusion list. The list's validity period is equal to the circuit breaker recovery time T. R To avoid repeated calls within a short period of time, and to update P at the same time. i Status data, consecutive failure count F(P) i +1, final failure time T L (P i = Current timestamp.

[0060] Step 2: Filter the remaining available service provider large models. Iterate through the remaining service provider large models in the initial service provider large model set, remove service provider large models that are in a circuit breaker or congestion state, and retain service provider large models that still meet the availability conditions.

[0061] Step 3: Sort by downgrade order. For the remaining available service providers in the large model, sort them according to their downgrade order O(P). i Sort in ascending order, O(P) i The smaller the value, the higher it appears, ensuring that the high-availability alternative service provider model is called first.

[0062] Step 4: Construct a fallback set. The sorted list of service provider large models is the fallback service provider large model set. If the set is empty, return the message "No service provider large models available, please try again later" to the client and terminate the process; otherwise, use this set as the new initial service provider large model set, return to the starting point of the filtering process, and re-execute circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering to filter new preferred service provider large models.

[0063] The S400 acquires the client's question text and generates the answer text using the optimal service provider's large model.

[0064] In some embodiments, the generation of the answer text from the question text using the optimal service provider's large model includes: Obtain the API protocol format of the optimal service provider's large model, convert the question text into the API protocol format through the API system's protocol adaptation layer to obtain the optimized question text, input the optimized question text into the optimal service provider's large model to generate the answer text, and obtain the answer text.

[0065] In some embodiments, to shield the interface differences between large models from different service providers and achieve hot-swappable models transparent to upper-layer business logic, this system introduces a protocol adaptation layer. The protocol adaptation layer extracts the pre-configured protocol information of the large model from the system configuration center, including the request method, data format, field mapping rules, and request header requirements. For example, the API requirements for large model P1 from service provider P1 are: POST method, JSON format, field mappings of prompt→content and timeout→expire, and a required parameter including api_key.

[0066] The client submits a request using a standardized system template, containing fixed fields in JSON format. The system automatically verifies the completeness of the fields and the validity of the data types in the request text; it also verifies the validity of the service provider's large-scale API protocol format. If the verification fails, it returns a "Request format error" message and terminates the process.

[0067] The adaptation layer replaces the fields in the problem text with the field names required by the service provider's large model API according to pre-configured field mapping rules, and also supplements the additional parameters that are required by the service provider's large model. For example, it converts the standard text "user_query" into the "content" required by the optimal service provider's large model.

[0068] If the service provider's large model API requires a non-JSON format, the adaptation layer will automatically convert the transformed field data to the target format, ensuring that the Content-Type matches the format, and adjust the data according to the parameter constraints of the service provider's large model API.

[0069] The adaptation layer verifies whether the converted optimal request text fully conforms to the service provider's large model API specification. If the verification passes, it proceeds to the next step; if it fails, it triggers a retry; if it still fails, it returns a "protocol adaptation failed" message.

[0070] Send requests to the API interface of the heterogeneous service provider's large model via HTTP / HTTPS protocol, set a timeout, and enable connection pooling for reuse. Receive the raw response returned by the service provider's large model; if it is a streaming response, receive and concatenate it in real time according to the streaming rhythm; if it is a normal response, obtain the complete result at once.

[0071] The adaptation layer converts the original result into a system-standard format and returns the standardized response to the client. If the request fails to send, the reason for the failure is recorded and a "Service call timed out" message is returned; if the returned result format is abnormal, "Result format is abnormal, please retry" is returned.

[0072] Reference Figure 2 This application provides an optimal service provider large model selection device 20 for an API system, used to implement the optimal service provider large model selection method for an API system. The device includes: The data acquisition module 21 is used by the API system gateway to obtain the model identification data in the request after receiving the client's request, obtain the initial service provider large model set from the API system configuration center based on the model identification data, and obtain the status data of the initial service provider large model set through the API system status monitor. The preferred service provider large model acquisition module 22 is used to perform circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering on the initial service provider large model set based on model identification data and status data to obtain the preferred service provider large model. The optimal service provider large model acquisition module 23 is used to call the preferred service provider large model to the API system's server and obtain the call status of the preferred service provider large model. When the call status is "call failed," the preferred service provider large model is removed from the initial service provider large model set. A set of degraded service provider large models is constructed based on the remaining service provider large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the degraded service provider large model set to re-select the preferred service provider large model. When the call status is "call successful," the preferred service provider large model is taken as the optimal service provider large model. The answer text acquisition module 24 is used to acquire the client's question text and generate the answer text of the question text through the optimal service provider's big model.

[0073] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the optimal service provider large model selection method of the API system of any of the above schemes.

[0074] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0075] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0076] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the optimal service provider big model selection method for an API system based on any of the above-described schemes. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the methods described in the embodiments of this application.

[0077] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0078] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for selecting the optimal service provider model for an API system, characterized in that, include: After receiving a client's request, the API system's gateway obtains the model identifier data from the request, retrieves the initial service provider's large model set from the API system's configuration center based on the model identifier data, and obtains the status data of the initial service provider's large model set through the API system's status monitor. Based on the model identification data and status data, circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are performed on the initial service provider large model set to obtain the preferred service provider large model; The preferred service provider's large model is invoked to the API system's server to obtain the invocation status. If the invocation status indicates failure, the preferred service provider's large model is removed from the initial set of service provider large models. A set of degraded service provider large models is constructed based on the remaining service provider large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the degraded service provider large model set to re-select the preferred service provider's large model. If the invocation status indicates success, the preferred service provider's large model is selected as the optimal service provider's large model. Obtain the client's question text and generate the answer text using the optimal service provider's large model.

2. The optimal service provider large-scale model selection method for the API system according to claim 1, characterized in that, The model identification data includes static weights W(P) i ), preset maximum concurrency M (P) i ) and demotion order O(P) i ); Status data includes the current concurrency C(P) i ), consecutive failure count F(P) i ) and the final failure time T L (P i ); where i represents the service provider's large model number, P i Let i represent the large model of the i-th service provider.

3. The optimal service provider large-scale model selection method for the API system according to claim 2, characterized in that, The process of applying circuit breaker filtering to the initial large set of service providers includes: Get the current time T now Circuit breaker recovery time T R and the maximum consecutive failure threshold F max ; If the current time T now With service provider large model P i The final failure time T L (P i The difference is less than the circuit breaker recovery time T. R And the service provider's large model P i The consecutive failure count F(P) i () is greater than or equal to the maximum consecutive failure threshold F max Then determine the service provider's large model P i In the circuit breaker state, the service provider's large model P will be activated. i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

4. The optimal service provider large-scale model selection method for the API system according to claim 2, characterized in that, The process of performing concurrent hard rate limiting filtering on the initial large set of service providers includes: Obtain the large model P of the service provider i The preset maximum concurrency M (P) i ) and the current concurrency C(P) i ); If the current concurrency level C(P) i ) is greater than or equal to the preset maximum concurrency M(P) i If the service provider's large model P is determined, then... i In a congested state, the service provider's large model P i Remove from the initial large set of service provider models; otherwise, remove the large set of service provider models P. i reserve.

5. The optimal service provider large-scale model selection method for the API system according to claim 2, characterized in that, The step of performing weighted random route filtering on the initial large set of service provider models to obtain the preferred large set of service provider models includes: Obtain the static weights of the large models from each service provider, and sum the static weights of the large models from each service provider to obtain the total weight W. total From 0 to the total weight W total Choose a random integer R from between; The static weights are accumulated sequentially according to the downgrade order of each service provider's large model to obtain the accumulated static weight S; When the accumulated static weight S is greater than the random integer R, the current service provider's large model is selected as the preferred service provider's large model.

6. The optimal service provider large-scale model selection method for an API system according to claim 2, characterized in that, When the call status is "call failed," the preferred service provider model is removed from the initial service provider model set, and a degraded service provider model set is constructed based on the remaining service provider models, including: When the call status is "call failed", retrieve the response exception type; If the response exception type is a business layer error or content security block, an error message is sent to the client and the process ends; If the response exception type is a network layer error, server error, or streaming interruption, the preferred service provider large model is moved from the initial service provider large model set to the exclusion list. The remaining service provider large models in the initial service provider large model set are sorted in ascending order according to the degradation order. The remaining service provider large models after the ascending order are used to form the degraded service provider large model set.

7. The optimal service provider large-scale model selection method for the API system according to claim 1, characterized in that, The answer text generated by the optimal service provider's large model includes: Obtain the API protocol format of the optimal service provider's large model, convert the question text into the API protocol format through the API system's protocol adaptation layer to obtain the optimized question text, input the optimized question text into the optimal service provider's large model to generate the answer text, and obtain the answer text.

8. An optimal service provider large-scale model selection device for an API system, used to implement the optimal service provider large-scale model selection method for an API system as described in any one of claims 1 to 7, characterized in that, The device includes: The data acquisition module is used by the API system's gateway to obtain the model identification data in the request after receiving a client's request. Based on the model identification data, it obtains the initial service provider's large model set from the API system's configuration center and obtains the status data of the initial service provider's large model set through the API system's status monitor. The preferred service provider large model acquisition module is used to perform circuit breaker filtering, concurrent hard rate limiting filtering, and weighted random routing filtering on the initial service provider large model set based on model identification data and status data, so as to obtain the preferred service provider large model. The optimal service provider large model acquisition module is used to call the preferred service provider large model to the API system's server and obtain the call status of the preferred service provider large model. When the call status is "call failed," the preferred service provider large model is removed from the initial service provider large model set. A fallback service provider large model set is constructed based on the remaining service provider large models. Circuit breaking filtering, concurrent hard rate limiting filtering, and weighted random routing filtering are applied to the fallback service provider large model set to re-select the preferred service provider large model. When the call status is "call successful," the preferred service provider large model is taken as the optimal service provider large model. The answer text acquisition module is used to acquire the client's question text and generate the answer text for the question text using the optimal service provider's big data model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the optimal service provider large model selection method for the API system as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the optimal service provider big model selection method for the API system as described in any one of claims 1 to 7.

Images