Semantic-aware multi-model dynamic routing methods, devices, equipment, and media

Abstract

This invention relates to the field of artificial intelligence technology, providing a semantically aware multi-model dynamic routing method, apparatus, device, and medium. On one hand, it is applied to a multi-model dynamic routing system deployed as an independent gateway service, providing an open AI-compatible application programming interface protocol, enabling zero-code access. On the other hand, it detects whether reuse conditions are met based on query results to reduce processing overhead and improve routing decision efficiency. Furthermore, it employs a business utility mechanism to configure the weight coefficient matrix, improving the dynamic adaptability of weights. Finally, it uses a delay penalty mechanism to perform three-dimensional joint weighting processing on the target semantic feature vector, real-time word unit price matrix, and model capability score matrix based on the weight coefficient matrix, obtaining the candidate model with the highest joint score as the routing target, enabling fine-grained multi-model routing cost optimization based on multi-dimensional data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-model dynamic routing method, apparatus, device, and medium based on semantic awareness. Background Technology

[0002] With the rapid commercialization and widespread adoption of Large Language Model (LLM) services, enterprises face complex environments with multiple vendors and models when invoking LLM services. For example, enterprise gateways for large models often need to select the appropriate model from multiple vendors, and to meet business requirements, more than one model typically needs to provide services simultaneously. Different models vary significantly in terms of inference capabilities, output quality, cost per token, and response latency. Therefore, enterprise users need to minimize invocation costs while meeting business quality requirements.

[0003] Existing multi-model routing solutions mainly rely on the following technical paths: rule-based static routing, simple routing based on request length, and fixed strategies based on historical experience.

[0004] However, the above solution still has the following shortcomings: (1) Rule-based static routing: Predefined routing rules (such as task type keyword matching) are used to assign requests of a specific category to a specified model. This approach has high configuration costs, poor flexibility, and cannot perceive the semantic complexity of the requests; (2) Simple routing based on request length: Short requests are assigned to low-cost models based on the number of input tokens. This approach ignores the semantic differences between requests of the same length. For example, a short code debugging request and a short content summary request have drastically different requirements for model capabilities. (3) Fixed strategy based on historical experience: The mapping table between each business scenario and the model is manually configured, which mainly relies on expert experience and is difficult to adapt to the rapid changes in the model ecosystem and cost fluctuations. Summary of the Invention

[0005] In view of the above, it is necessary to provide a semantically aware multi-model dynamic routing method, device, equipment and medium, which aims to solve the problem that the single dimension of model routing decision-making cannot capture the semantic features of the request, resulting in limited optimization effect in balancing cost and quality.

[0006] A semantically aware multi-model dynamic routing method is applied to a multi-model dynamic routing system deployed as an independent gateway service, providing an open AI-compatible application programming interface protocol. The semantically aware multi-model dynamic routing method includes: In response to a multi-model dynamic routing instruction triggered based on a target request, the target semantic feature vector of the target request is extracted; The target semantic feature vector is used to perform a query in the semantic vector cache, and the reuse condition is detected based on the query result; When it is detected that the reuse condition is not met, the real-time word unit price matrix and the model capability score matrix are obtained. The weight coefficient matrix is ​​configured using a business utility mechanism; A delayed penalty mechanism is adopted, and the target semantic feature vector, the real-time word unit price matrix and the model capability score matrix are subjected to three-dimensional joint weighting based on the weight coefficient matrix to obtain the joint score of each candidate model; The candidate model with the highest joint score is selected as the routing target, and a call is initiated to the routing target.

[0007] A semantically aware multi-model dynamic routing device operates within a multi-model dynamic routing system deployed as an independent gateway service, providing an open AI-compatible application programming interface protocol. The semantically aware multi-model dynamic routing device includes: The extraction unit is used to extract the target semantic feature vector of the target request in response to a multi-model dynamic routing instruction triggered based on the target request; The query unit is used to query the semantic vector cache using the target semantic feature vector, and to detect whether the reuse condition is met based on the query result; The construction unit is used to obtain the constructed real-time word unit price matrix and the constructed model capability score matrix when it is detected that the reuse condition is not met. Configuration unit, used to configure the weight coefficient matrix using a business utility mechanism; The processing unit is used to perform three-dimensional joint weighting processing on the target semantic feature vector, the real-time word unit price matrix and the model capability scoring matrix according to the weight coefficient matrix using a delay penalty mechanism to obtain the joint score of each candidate model. The calling unit is used to obtain the candidate model with the highest joint score as the routing target and to initiate a call to the routing target.

[0008] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the semantically aware multi-model dynamic routing method.

[0009] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the semantically aware multi-model dynamic routing method.

[0010] As can be seen from the above technical solutions, on the one hand, the multi-model dynamic routing system, deployed as an independent gateway service and providing an open AI-compatible application programming interface protocol, can achieve zero-code modification for access; on the other hand, it detects whether the reuse conditions are met based on the query results to reduce processing overhead and improve routing decision efficiency; on the other hand, it adopts a business utility mechanism to configure the weight coefficient matrix, improving the dynamic adaptability of the weights; and on the other hand, it adopts a delay penalty mechanism to perform three-dimensional joint weighting processing on the target semantic feature vector, the real-time word unit price matrix, and the model capability score matrix based on the weight coefficient matrix, and obtains the candidate model with the highest joint score as the routing target, which can achieve fine-grained multi-model routing cost optimization based on multi-dimensional data. Attached Figure Description

[0011] Figure 1 This is a flowchart of a preferred embodiment of the semantically aware multi-model dynamic routing method of the present invention; Figure 2 This is a functional block diagram of a preferred embodiment of the semantically aware multi-model dynamic routing device of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the semantically aware multi-model dynamic routing method of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the semantically aware multi-model dynamic routing method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0014] The semantically aware multi-model dynamic routing method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0015] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0016] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0017] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0018] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0020] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0021] The semantically aware multi-model dynamic routing method described in this embodiment is applied to a multi-model dynamic routing system that is deployed as an independent gateway service and provides an Open Artificial Intelligence (OpenAI) compatible Application Programming Interface (API) protocol.

[0022] The semantically aware multi-model dynamic routing method can also be embedded in existing applications as a Sidecar container or SDK (Software Development Kit), and maintains OpenAI-compatible API protocols for upper-layer services.

[0023] Unlike existing routing solutions that often require business users to label requests according to preset tag formats or modify call code to trigger specific routing branches, which imposes significant transformation costs and compatibility burdens on existing applications, this embodiment, through the multi-model dynamic routing system, allows business users to enjoy semantic routing benefits without any modification to their call code. Furthermore, it enables frictionless integration of existing applications and can be rapidly deployed in complex scenarios where multiple existing LLM (Large Language Model) applications are mixed within an enterprise, achieving zero-code modification integration—a completely transparent and non-intrusive integration.

[0024] Specifically, the semantically aware multi-model dynamic routing method includes: S10, in response to the multi-model dynamic routing instruction triggered based on the target request, extract the target semantic feature vector of the target request.

[0025] In this embodiment, the target request can be a user request from an upper-layer business system received through an LLM inference gateway.

[0026] In this embodiment, before extracting the target semantic feature vector of the target request, in order to ensure data quality, the target request can be preprocessed for normalization, including: parsing the Prompt text, System instructions, Function Calling definitions, and additional parameters (such as temperature parameter, max_Tokens maximum generation length, etc.) in the request; performing noise reduction, length truncation (the default upper limit can be 8192 tokens), and sensitive information anonymization on the Prompt text; and extracting request metadata (such as source application ID, user tenant ID, and business scenario tags).

[0027] Current routing schemes primarily rely on request length, keywords, or manually preset rules, utilizing only a shallow representation of the request. This fails to capture deep semantic features such as request complexity, task type, and domain attributes. Consequently, in real-world business scenarios with highly diverse natural language requests, routing accuracy is low. Complex tasks are often incorrectly assigned to underpowered, low-cost models, leading to quality degradation, while simple tasks are over-assigned to high-cost flagship models, resulting in resource waste. This demonstrates a serious problem of a single dimension in routing decision-making. Therefore, this embodiment first extracts the semantic information of the target request.

[0028] Specifically, extracting the target semantic feature vector of the target request includes: The lightweight semantic coding model's word segmenter is invoked to segment the target request into a word sequence; The word sequence is input into the embedding model for forward computation to obtain the initial vector; The initial vector is normalized using the L2 norm to obtain the intermediate vector; The intermediate vector is dimensionally aligned by calling a linear projection layer pre-trained according to the capability vector dimension of each candidate model to obtain the target semantic feature vector.

[0029] The lightweight semantic coding model may include open-source lightweight models such as BGE-small (BAAI General Embedding-small, BGE-small embedding model) and E5-base (Embedding Encoder-base, E5-base embedding model), or self-trained domain-specific coding models.

[0030] Using lightweight embedding models such as BGE-small and E5-base (inference latency <5ms) for semantic vector extraction results in a latency two orders of magnitude lower than semantic judgment schemes based on LLM inference. Therefore, the low latency of lightweight semantic encoding in this embodiment will not become a throughput bottleneck for high-concurrency gateway scenarios using large models in enterprises.

[0031] The dimension of the target semantic feature vector can be from 128 to 4096, preferably 768, to balance computational overhead and representational capability.

[0032] In this process, after inputting the lexical sequence into the embedding model for forward computation, the hidden state or the pooled mean vector of the last layer [CLS] Token (Classification Token) can be extracted as the initial vector.

[0033] In this process, after performing L2 norm normalization on the initial vector, all vectors can be located on the unit hypersphere, which facilitates subsequent similarity calculation.

[0034] The problem of inconsistent capability vector dimensions among different candidate models can be solved by calling a linear projection layer pre-trained according to the capability vector dimension of each candidate model to align the dimensions of the intermediate vector.

[0035] In the above embodiments, compared with traditional rule matching schemes that use request length or keywords as a single-dimensional routing basis, this embodiment can accurately capture deep features such as semantic complexity, task type, and domain attributes of requests by extracting semantic feature vectors. It has deep semantic awareness routing capabilities and avoids the blind spot of treating the same length the same as in traditional schemes.

[0036] S11, use the target semantic feature vector to query the semantic vector cache, and check whether the reuse condition is met based on the query result.

[0037] In this embodiment, the step of querying the semantic vector in the semantic vector cache using the target semantic feature vector and detecting whether the reuse condition is met based on the query result includes: Each candidate key-value pair is retrieved from the semantic vector cache, which is constructed based on the Least Recently Used (LRU) eviction policy. The key of each candidate key-value pair is the request vector hash value, and the value of each candidate key-value pair consists of the semantic feature vector, the last routing decision, and the time to live (TTL). Calculate the similarity between the target semantic feature vector and the semantic feature vector in each candidate key-value pair; When a target candidate key-value pair is detected whose similarity is greater than the similarity threshold, it is determined that the reuse condition is met; or When no candidate key-value pair is detected whose similarity is greater than the similarity threshold, it is determined that the reuse condition is not met. Specifically, when the reuse condition is detected, the previous routing decision of the target candidate key-value pair is reused.

[0038] The survival time is used to limit the lifespan of each candidate key-value pair in the semantic vector cache, so as to avoid selecting it after it has lost its reference value, which would cause incorrect routing.

[0039] The capacity of the semantic vector cache can be configured to 100,000 entries.

[0040] The survival time can be configured to be 1 hour.

[0041] For example, when a request arrives, the system first searches the semantic vector cache for historical vectors with a cosine similarity higher than a threshold (e.g., 0.95). If a match is found, the corresponding historical routing decision is directly reused, significantly reducing the processing overhead for high-frequency, repetitive requests.

[0042] In the above embodiments, for frequently repeated or highly similar requests, historical routing decisions can be directly reused based on semantic vector caching, skipping the complete joint calculation process. In hit scenarios, processing latency can be reduced from a typical 10ms-50ms to 1ms-3ms, significantly improving the overall gateway throughput and system performance.

[0043] S12, when it is detected that the reuse condition is not met, obtain the constructed real-time word unit price matrix and the constructed model capability score matrix.

[0044] In this embodiment, before obtaining the constructed real-time lexical unit price matrix, the method further includes: At preset time intervals, and / or when a unit price change event is detected, obtain the input unit price and output unit price of each candidate model; The real-time lexical unit price matrix is ​​constructed with each candidate model as a row and the input lexical unit price and output lexical unit price of each candidate model as columns.

[0045] The preset time interval can be configured to be 600 seconds, meaning the unit price is refreshed every 600 seconds.

[0046] The unit price change event may include receiving a pricing change notification from the supplier. In this case, a unit price refresh can be triggered immediately.

[0047] Specifically, the input lexicon unit price and output lexicon unit price of each candidate model can be obtained in real time from the official APIs, pricing pages, or third-party price aggregation services of each model supplier.

[0048] The real-time lexical unit price matrix can be primarily stored in memory using key-value pairs (KV) storage, supplemented by disk persistence, thereby ensuring high availability.

[0049] Through the above embodiments, the real-time lexical unit price matrix can be updated in real time, enabling the routing strategy to automatically respond to market price fluctuations and dynamically adapt to changes in the model ecosystem. When a model supplier adjusts its prices, a new model is launched, or a model's capabilities are upgraded, the routing decision can adapt to the latest state without any manual intervention, resulting in near-zero operational costs.

[0050] In this embodiment, the construction of the model capability scoring matrix includes: Acquire multi-source scoring data, including publicly available benchmark scores, enterprise historical call quality scores, and capability scores published by model vendors; Obtain multiple capability dimensions of the configuration; For each candidate model, sub-scores corresponding to each capability dimension are obtained from the multi-source scoring data, and the sub-scores corresponding to each capability dimension are fused according to adjustable weights to obtain the quantitative score of each capability dimension. The model capability scoring matrix is ​​constructed with each candidate model as a row and the quantified score of each capability dimension corresponding to each candidate model as a column.

[0051] The publicly available benchmark scores may include, but are not limited to, standardized scores from MMLU (Massive Multitask Language Understanding), HumanEval (Human-level Evaluation), MT-Bench (LLM Multi-turn Dialogue Benchmark), and GSM8K (Grade School Math 8K), and may be expressed on a percentage scale.

[0052] The enterprise historical call quality score can be the actual quality statistics in the enterprise's historical call data, such as manual scoring, user feedback, downstream task pass rate, etc., and can be in percentage form.

[0053] The capability score can be a capability declaration issued by the model provider. The capability dimensions of the capability score may include, but are not limited to: code understanding and generation, mathematical reasoning, multilingual ability, long context processing, instruction compliance, security compliance, and domain knowledge (such as knowledge in finance, healthcare, and law). The scores for the above capability dimensions can all be based on a 100-point scale. It should be noted that if a speed dimension is involved, a specific speed value can be directly used, such as 130 Tokens / s.

[0054] Among the adjustable weights, the sub-scores derived from the publicly available benchmark scores typically have the highest weight, the sub-scores derived from the enterprise's historical call quality scores have constantly changing weights, and the sub-scores derived from the capability scores published by the model vendors support custom adjustments.

[0055] Specifically, for each capability dimension, when there are sub-scores from multiple sources, the weighted sum of the sub-scores from each source is calculated according to the adjustable weights to obtain the quantified score for that capability dimension; when there is only one sub-score from a single source, that sub-score is directly used as the quantified score (equivalent to other source sub-scores being 0).

[0056] In the above embodiments, the capability scoring matrix integrates multiple sources of data, including publicly available benchmark tests, actual quality statistics of enterprise historical call data, and capability declarations released by model vendors. This avoids the one-sidedness of a single source, and the multi-dimensional capability characterization enables routing decisions to more accurately match the needs of different tasks for different capability dimensions.

[0057] S13 uses a business utility mechanism to configure the weight coefficient matrix.

[0058] In this embodiment, configuring the weight coefficient matrix using a business utility mechanism includes: When the multi-model dynamic routing system is deployed for the first time, the initial weights for cold start are obtained to construct the weight coefficient matrix; or When the multi-model dynamic routing system is not being deployed for the first time, it acquires continuously collected historical decision data to construct a historical sample set. When the sample size of the historical sample set is greater than or equal to a sample size threshold, and the time since the last weight recommendation is greater than or equal to the minimum recommendation period, it calls a business utility function to calculate the business utility value of each historical sample in the historical sample set. Following a descending order, it acquires the historical samples with the highest business utility values ​​from the historical sample set to construct a target sample set. It calculates the decision consistency of each group of candidate weights on the target sample set in the weight search space. Finally, it acquires the group of candidate weights with the highest decision consistency to construct the weight coefficient matrix. The business utility function is constructed based on a weighted sum of quality, cost, and latency.

[0059] The values ​​of the initial cold start weight include: (1) General default configuration (e.g., quality weight α=0.7, cost weight β=0.3, equilibrium weight γ=0.0); (2) Industry templates pre-set according to business scenarios (such as customer service scenarios, code scenarios, etc.); (3) Reference values ​​based on existing deployment experience in the same industry.

[0060] During the cold start period, historical call samples can be continuously collected, and exploratory traffic can be enabled at a configurable ratio (in 5%). This means that a small number of requests are processed using a weight combination that deviates from the initial weight of the cold start, thereby enriching the sample distribution coverage and avoiding the influence of sample bias on subsequent recommendations.

[0061] The historical decision data may include decision samples and their actual business results for each call, such as request semantic features, candidate model set, normalized value, historical actual selected model, as well as actual quality, actual cost, actual latency, etc.

[0062] The sample size threshold can be configured to 5000, and the minimum recommendation period can be configured to 7 days.

[0063] If the sample size threshold and any condition corresponding to the minimum recommendation period are not met, the current weight will be maintained and samples will continue to be collected.

[0064] The business utility function can be expressed as follows: U = u_q × Q - u_c × C - u_l × L; Where U represents the business utility value of the current call; Q represents the actual quality of the current call; C represents the actual cost of the current call; L represents the actual latency of the current call; and u_q, u_c, and u_l represent the utility parameters configured according to the scenario.

[0065] Specifically, the top 20 historical samples can be selected according to their business utility values ​​from high to low to construct a more excellent and typical target sample set.

[0066] After constructing the weight coefficient matrix, it can be pushed to the administrator via the RESTful (Representational State Transfer) management API exposed by the multi-model dynamic routing system, along with metadata such as expected utility improvement and confidence level. When the administrator triggers one-click adoption, the weight coefficient matrix takes effect immediately through atomic hot update, with zero interruption to online requests and no need to restart the gateway service; when the administrator triggers canary adoption, it is verified on 5%-30% of the traffic before taking full effect; when the administrator triggers rejection with explanation, the weight coefficient matrix is ​​archived and does not take effect.

[0067] All of the above decision data are written to the log.

[0068] Furthermore, it also allows administrators to adjust the weight coefficient matrix in real time without restarting the gateway service via the management API. For example, α can be temporarily increased during peak business periods to prioritize quality, while α can be decreased during nighttime maintenance periods to prioritize cost, thereby achieving differentiated scenario management.

[0069] In the above embodiments, by adaptively adjusting the weights, it is possible to finely balance quality and cost objectives in different business scenarios and flexibly switch between various strategies such as quality priority, cost priority, and balance.

[0070] S14. A delayed penalty mechanism is adopted, and the target semantic feature vector, the real-time word unit price matrix and the model capability scoring matrix are subjected to three-dimensional joint weighting processing according to the weight coefficient matrix to obtain the joint score of each candidate model.

[0071] In this embodiment, before performing three-dimensional joint weighting processing on the target semantic feature vector, the real-time lexical unit price matrix, and the model capability score matrix according to the weight coefficient matrix, candidate models that do not meet hard constraints such as context length, data compliance requirements, and geographical availability can be eliminated first. This two-stage decision architecture of first constraining and then optimizing ensures that even in industries with high compliance requirements (such as finance and healthcare), routing decisions always comply with business rules.

[0072] In a service environment where multiple models coexist, the token price of different models can vary by 5 to 100 times, and their performance differences on different tasks are not simply proportional to their price. Existing solutions lack the technical means to incorporate semantic complexity, model capabilities, and token price into a unified computing framework, making it difficult to achieve fine-grained cost optimization while ensuring quality. Balancing cost and quality is challenging, resulting in persistently high LLM call costs for enterprises.

[0073] Furthermore, the models, capabilities, and pricing in the LLM field are constantly and rapidly iterating (new models are launched, pricing is adjusted, and capabilities are upgraded every month). The existing static routing tables based on manual configuration cannot respond to such ecosystem changes in a timely manner. The routing strategy quickly becomes decoupled from the latest model cost-quality ratio, requiring frequent manual maintenance costs, which leads to the rigidity of strategies under the dynamic changes of the model ecosystem.

[0074] To address the aforementioned issues, in this embodiment, a delayed penalty mechanism is employed. Based on the weight coefficient matrix, a three-dimensional joint weighting process is performed on the target semantic feature vector, the real-time lexical unit price matrix, and the model capability scoring matrix to obtain the joint score for each candidate model, including: For each candidate model, the model capability vector is constructed by obtaining the quantified score of each capability dimension from the model capability scoring matrix, and the price vector is constructed by obtaining the corresponding input word unit price and output word unit price from the real-time word unit price matrix. Calculate the similarity between the target semantic feature vector and the model capability vector; Obtain the first-byte latency and output throughput of the candidate model, and predict the expected number of output tokens based on the target semantic feature vector; Calculate the quotient of the expected number of output tokens multiplied by a factor of 1000 and the output throughput, and sum the quotient with the first byte delay to obtain the total delay; Calculate the weighted sum of the first byte delay and the total delay to obtain the normalized delay penalty term; The negative values ​​of the similarity, the price vector, and the normalized delay penalty term are weighted according to the weight coefficient matrix to obtain the joint score of the candidate model.

[0075] The cosine similarity algorithm can be used to calculate the similarity between the target semantic feature vector and the model capability vector, which reflects the degree to which the model fits the semantic requirements of the current request.

[0076] In constructing the price vector, logarithmic normalization or Min-Max (deviation standardization) normalization can be performed on the real-time word unit price matrix to eliminate the price dimension differences between different models.

[0077] The formula for calculating the joint score is as follows: Score(m_i)=α×Sim(V_req,C_cap_i)-β×P_norm_i-γ×Latency_norm_i; Where Score(m_i) represents the joint score of the i-th candidate model; Sim(V_req,C_cap_i) represents the similarity between the target semantic feature vector V_req and the model capability vector C_cap_i of the i-th candidate model; P_norm_i represents the price vector of the i-th candidate model; and Latency_norm_i represents the normalized delay penalty term of the i-th candidate model.

[0078] The formula for calculating the normalized delay penalty term is as follows: Latency_norm_i=0.3×TTFT_norm_i+0.7×(TTFT_norm_i+L_pred / TPS_norm_i×1000); Where TTFT_norm_i represents the first-byte delay of the i-th candidate model, in milliseconds; L_pred represents the expected number of output tokens, which can be inferred from the output length tendency feature extracted from the target semantic feature vector; TPS_norm_i represents the output throughput of the i-th candidate model, in tokens / second.

[0079] The smaller the Latency_norm_i, the higher the joint score.

[0080] S15, obtain the candidate model with the highest joint score as the routing target, and initiate a call to the routing target.

[0081] In this embodiment, the candidate model with the highest joint score is selected as the routing target. Based on the three-dimensional joint weighted calculation framework of semantic complexity, model capability, and token unit price, the optimal model with the highest weighted score can be found under each strategy.

[0082] In this embodiment, after obtaining the candidate model with the highest joint score as the routing target, a request body (including path, Header, and Body format conversion) adapted to the API protocol of the routing target can be constructed, and a call can be initiated to the routing target, and the response can be forwarded to the upper-layer business system.

[0083] In this embodiment, all relevant data from the above steps can be recorded to a log so that the intermediate data of each routing decision (such as semantic feature vectors, similarity scores of each candidate model, normalized price, and final joint score) can be verified and visualized. Operations and business parties can clearly understand why this model is chosen instead of that model. This interpretability is far superior to black-box decision-making, which facilitates optimization, compliance verification, and the establishment of trust among business parties.

[0084] The recorded logs also allow for the execution of the following tasks: (1) Identify which task types are over-configured; (2) Discover new capability dimension requirements (such as the emergence of highly complex request patterns that are not characterized by existing dimensions); (3) As an objective basis for model supplier selection and negotiation, the routing system is upgraded from a cost reduction tool to an LLM application governance infrastructure.

[0085] In this embodiment, the model level can be selected first, and then token-level or context-level optimization can be performed. The overall cost reduction can be even greater after the synergistic combination.

[0086] In this embodiment, the functional units corresponding to each execution step can collaborate through an internal message bus and provide a unified gateway entry point to the outside world, so as to achieve accurate, automatic, and low-intrusion optimal model selection for each request, thereby achieving a fine-grained balance between cost and quality in a multi-model LLM service environment.

[0087] As can be seen from the above technical solutions, on the one hand, the multi-model dynamic routing system, deployed as an independent gateway service and providing an open AI-compatible application programming interface protocol, can achieve zero-code modification for access; on the other hand, it detects whether the reuse conditions are met based on the query results to reduce processing overhead and improve routing decision efficiency; on the other hand, it adopts a business utility mechanism to configure the weight coefficient matrix, improving the dynamic adaptability of the weights; and on the other hand, it adopts a delay penalty mechanism to perform three-dimensional joint weighting processing on the target semantic feature vector, the real-time word unit price matrix, and the model capability score matrix based on the weight coefficient matrix, and obtains the candidate model with the highest joint score as the routing target, which can achieve fine-grained multi-model routing cost optimization based on multi-dimensional data.

[0088] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the semantically aware multi-model dynamic routing device of the present invention. The semantically aware multi-model dynamic routing device 11 operates as an independent gateway service, providing an open AI-compatible application programming interface protocol to the outside world. Specifically, it includes an extraction unit 110, a query unit 111, a construction unit 112, a configuration unit 113, a processing unit 114, and a calling unit 115. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0089] The extraction unit 110 is used to extract the target semantic feature vector of the target request in response to a multi-model dynamic routing instruction triggered based on the target request. The query unit 111 is used to query the semantic vector cache using the target semantic feature vector, and to detect whether the reuse condition is met based on the query result. The construction unit 112 is used to obtain the constructed real-time word unit price matrix and the constructed model capability score matrix when it is detected that the reuse condition is not met. The configuration unit 113 is used to configure the weight coefficient matrix using a business utility mechanism; The processing unit 114 is used to employ a delay penalty mechanism to perform three-dimensional joint weighting processing on the target semantic feature vector, the real-time word unit price matrix, and the model capability scoring matrix according to the weight coefficient matrix, so as to obtain the joint score of each candidate model. The calling unit 115 is used to obtain the candidate model with the highest joint score as the routing target and to initiate a call to the routing target.

[0090] As can be seen from the above technical solutions, on the one hand, the multi-model dynamic routing system, deployed as an independent gateway service and providing an open AI-compatible application programming interface protocol, can achieve zero-code modification for access; on the other hand, it detects whether the reuse conditions are met based on the query results to reduce processing overhead and improve routing decision efficiency; on the other hand, it adopts a business utility mechanism to configure the weight coefficient matrix, improving the dynamic adaptability of the weights; and on the other hand, it adopts a delay penalty mechanism to perform three-dimensional joint weighting processing on the target semantic feature vector, the real-time word unit price matrix, and the model capability score matrix based on the weight coefficient matrix, and obtains the candidate model with the highest joint score as the routing target, which can achieve fine-grained multi-model routing cost optimization based on multi-dimensional data.

[0091] like Figure 3 The diagram shown is a structural schematic of a computer device that implements a preferred embodiment of the semantically aware multi-model dynamic routing method of the present invention.

[0092] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a semantically aware multi-model dynamic routing program.

[0093] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0094] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0095] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a semantically aware multi-model dynamic routing program, but also to temporarily store data that has been output or will be output.

[0096] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing semantically aware multi-model dynamic routing programs) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0097] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various semantically aware multi-model dynamic routing method embodiments described above, for example... Figure 1 The steps are shown.

[0098] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an extraction unit 110, a query unit 111, a construction unit 112, a configuration unit 113, a processing unit 114, and a calling unit 115.

[0099] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the semantically aware multi-model dynamic routing method described in the various embodiments of this invention.

[0100] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0101] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0102] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0103] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0104] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0105] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0106] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the computer device 1 and other computer devices.

[0107] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0108] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0109] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0110] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a semantically aware multi-model dynamic routing method, and the processor 13 can execute the multiple instructions to achieve: In response to a multi-model dynamic routing instruction triggered based on a target request, the target semantic feature vector of the target request is extracted; The target semantic feature vector is used to perform a query in the semantic vector cache, and the reuse condition is detected based on the query result; When it is detected that the reuse condition is not met, the real-time word unit price matrix and the model capability score matrix are obtained. The weight coefficient matrix is ​​configured using a business utility mechanism; A delayed penalty mechanism is adopted, and the target semantic feature vector, the real-time word unit price matrix and the model capability score matrix are subjected to three-dimensional joint weighting based on the weight coefficient matrix to obtain the joint score of each candidate model; The candidate model with the highest joint score is selected as the routing target, and a call is initiated to the routing target.

[0111] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0112] It should be noted that all the data involved in this case was legally obtained.

[0113] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0114] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0115] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0118] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0119] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0120] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A semantically aware multi-model dynamic routing method, characterized in that, The semantically aware multi-model dynamic routing method, applied to a multi-model dynamic routing system deployed as an independent gateway service and providing an open AI-compatible application programming interface protocol, includes: In response to a multi-model dynamic routing instruction triggered based on a target request, the target semantic feature vector of the target request is extracted; The target semantic feature vector is used to perform a query in the semantic vector cache, and the reuse condition is detected based on the query result; When it is detected that the reuse condition is not met, the real-time word unit price matrix and the model capability score matrix are obtained. The weight coefficient matrix is ​​configured using a business utility mechanism; A delayed penalty mechanism is adopted, and the target semantic feature vector, the real-time word unit price matrix and the model capability score matrix are subjected to three-dimensional joint weighting based on the weight coefficient matrix to obtain the joint score of each candidate model; The candidate model with the highest joint score is selected as the routing target, and a call is initiated to the routing target.

2. The semantically aware multi-model dynamic routing method as described in claim 1, characterized in that, The extraction of the target semantic feature vector of the target request includes: The lightweight semantic coding model's word segmenter is invoked to segment the target request into a word sequence; The word sequence is input into the embedding model for forward computation to obtain the initial vector; The initial vector is normalized using the L2 norm to obtain the intermediate vector; The intermediate vector is dimensionally aligned by calling a linear projection layer pre-trained according to the capability vector dimension of each candidate model to obtain the target semantic feature vector.

3. The semantically aware multi-model dynamic routing method as described in claim 1, characterized in that, The step of querying the semantic vector in the semantic vector cache using the target semantic feature vector and detecting whether the reuse condition is met based on the query result includes: Each candidate key-value pair is retrieved from the semantic vector cache; wherein the semantic vector cache is constructed based on the least recently used eviction policy; the key of each candidate key-value pair is the request vector hash value, and the value of each candidate key-value pair consists of the semantic feature vector, the last routing decision, and the time to live; Calculate the similarity between the target semantic feature vector and the semantic feature vector in each candidate key-value pair; When a target candidate key-value pair is detected whose similarity is greater than the similarity threshold, it is determined that the reuse condition is met; or When no candidate key-value pair is detected whose similarity is greater than the similarity threshold, it is determined that the reuse condition is not met. Specifically, when the reuse condition is detected, the previous routing decision of the target candidate key-value pair is reused.

4. The semantically aware multi-model dynamic routing method as described in claim 1, characterized in that, Before obtaining the constructed real-time lexical unit price matrix, the method further includes: At preset time intervals, and / or when a unit price change event is detected, obtain the input unit price and output unit price of each candidate model; The real-time lexical unit price matrix is ​​constructed with each candidate model as a row and the input lexical unit price and output lexical unit price of each candidate model as columns.

5. The semantically aware multi-model dynamic routing method as described in claim 4, characterized in that, The model construction capability scoring matrix includes: Acquire multi-source scoring data, including publicly available benchmark scores, enterprise historical call quality scores, and capability scores published by model vendors; Obtain multiple capability dimensions of the configuration; For each candidate model, sub-scores corresponding to each capability dimension are obtained from the multi-source scoring data, and the sub-scores corresponding to each capability dimension are fused according to adjustable weights to obtain the quantitative score of each capability dimension. The model capability scoring matrix is ​​constructed with each candidate model as a row and the quantified score of each capability dimension corresponding to each candidate model as a column.

6. The semantically aware multi-model dynamic routing method as described in claim 1, characterized in that, The configuration of the weight coefficient matrix using the business utility mechanism includes: When the multi-model dynamic routing system is deployed for the first time, the initial weights for cold start are obtained to construct the weight coefficient matrix; or When the multi-model dynamic routing system is not being deployed for the first time, it acquires continuously collected historical decision data to construct a historical sample set. When the sample size of the historical sample set is greater than or equal to a sample size threshold, and the time since the last weight recommendation is greater than or equal to the minimum recommendation period, it calls a business utility function to calculate the business utility value of each historical sample in the historical sample set. Following a descending order, it acquires the historical samples with the highest business utility values ​​from the historical sample set to construct a target sample set. It calculates the decision consistency of each group of candidate weights on the target sample set in the weight search space. Finally, it acquires the group of candidate weights with the highest decision consistency to construct the weight coefficient matrix. The business utility function is constructed based on a weighted sum of quality, cost, and latency.

7. The semantically aware multi-model dynamic routing method as described in claim 5, characterized in that, The method employs a delayed penalty mechanism, which performs a three-dimensional joint weighting process on the target semantic feature vector, the real-time lexical unit price matrix, and the model capability scoring matrix based on the weight coefficient matrix, to obtain the joint score for each candidate model, including: For each candidate model, the model capability vector is constructed by obtaining the quantified score of each capability dimension from the model capability scoring matrix, and the price vector is constructed by obtaining the corresponding input word unit price and output word unit price from the real-time word unit price matrix. Calculate the similarity between the target semantic feature vector and the model capability vector; Obtain the first-byte latency and output throughput of the candidate model, and predict the expected number of output tokens based on the target semantic feature vector; Calculate the quotient of the expected number of output tokens multiplied by a factor of 1000 and the output throughput, and sum the quotient with the first byte delay to obtain the total delay; Calculate the weighted sum of the first byte delay and the total delay to obtain the normalized delay penalty term; The negative values ​​of the similarity, the price vector, and the normalized delay penalty term are weighted according to the weight coefficient matrix to obtain the joint score of the candidate model.

8. A semantically aware multi-model dynamic routing device, characterized in that, Used to perform the semantically aware multi-model dynamic routing method as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the semantically aware multi-model dynamic routing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the semantically aware multi-model dynamic routing method as described in any one of claims 1 to 7.

Images