LLM routing decision method and device based on data flywheel, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides an LLM routing decision method and device based on a data flywheel, equipment and a medium. On the one hand, the method and device are applied to a routing decision system independently deployed based on containerized services in a private cloud environment, can effectively protect data privacy, and can also reduce additional delay overhead through asynchronous connection of various components through a private data pipeline. On the other hand, training samples are constructed based on aligned calling index values and decision point data, can systematically convert real result data accumulated each time into effective training signals for routing optimization, and can make data more comprehensive. On the one hand, the routing decision model is incrementally trained based on incremental samples in a flywheel training data set, can improve training efficiency and save computing power. On the other hand, the routing decision model is doubly verified by using a model hot updating component, can ensure the decision performance of the model, and thus realizes accurate LLM routing decision based on a data flywheel mechanism.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an LLM routing decision method, apparatus, device and medium based on a data flywheel. Background Technology

[0002] Currently, routing strategies in Large Language Model (LLM) inference gateways, such as large-model enterprise gateways, are typically statically configured. This means that routing rules are set by experts based on prior knowledge during system deployment, and the strategy remains unchanged until manual intervention. This static routing strategy has the following core drawbacks: (1) Unable to adapt to model evolution: The capabilities and costs of the LLM model change continuously, and the static policy cannot respond in time, resulting in the routing decision continuously lagging behind the actual optimal; (2) Relying on expert prior knowledge, difficult to scale: Manual configuration strategies rely on domain expert knowledge, and there are obvious scalability bottlenecks in multi-scenario, multi-model, and large-scale deployment. (3) The value of the call data is not utilized: The real result data accumulated from each routing call (such as success rate, quality score, cost) is the most valuable signal for optimizing the routing strategy. Existing solutions do not systematically feed it back to the strategy optimization. Summary of the Invention

[0003] In view of the above, it is necessary to provide an LLM routing decision method, device, equipment and medium based on data flywheel, which aims to solve the problems of insufficient accuracy and difficulty in scaling routing decisions of large language models.

[0004] A data flywheel-based LLM routing decision method is applied to a routing decision system independently deployed based on containerized services in a private cloud environment. The routing decision system includes a data acquisition component, an alignment component, a training sample construction component, an incremental training component, and a model hot update component, asynchronously connected via a private data pipeline. The data flywheel-based LLM routing decision method includes: In response to the routing decision instruction for the Large Language Model (LLM), the data acquisition component is invoked to synchronously record decision point data each time a routing decision is completed; After each model call is completed, the alignment component is used to calculate the call index value, and the call index value is aligned with the decision point data to obtain a sample pair; The training sample construction component is used to transform the sample pairs to obtain training samples, and the training samples are written into the flywheel training dataset according to time partitions. At preset time intervals, the incremental training component is used to obtain an initial model based on the current demand data, and incremental samples are read from the flywheel training dataset. The incremental samples are then used to incrementally train the initial model to obtain a routing decision model. The routing decision model is double-validated using the model hot update component, and when the routing decision model passes the validation, it atomically replaces the old model currently running online.

[0005] An LLM routing decision-making device based on a data flywheel runs on a routing decision-making system independently deployed based on containerized services in a private cloud environment. The routing decision-making system includes a data acquisition component, an alignment component, a training sample construction component, an incremental training component, and a model hot update component, asynchronously connected via a private data pipeline. The LLM routing decision-making device based on the data flywheel includes: The recording unit is used to respond to the routing decision command of the Large Language Model (LLM) by calling the data acquisition component to synchronously record the decision point data each time the routing decision is completed. The alignment unit is used to calculate the call index value using the alignment component after each model call, and to align the call index value with the decision point data to obtain sample pairs. The conversion unit is used to convert the sample pairs using the training sample construction component to obtain training samples, and write the training samples into the flywheel training dataset according to time partitions. The training unit is used to obtain an initial model based on the current demand data using the incremental training component at preset time intervals, and to read incremental samples from the flywheel training dataset and use the incremental samples to incrementally train the initial model to obtain a routing decision model. The verification unit is used to perform dual verification of the routing decision model using the model hot update component, and when the routing decision model passes the verification, it atomically replaces the old model currently running online with the routing decision model.

[0006] A computer device, the computer device comprising: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the data flywheel-based LLM routing decision method.

[0007] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the data flywheel-based LLM routing decision method.

[0008] As can be seen from the above technical solutions, on the one hand, the routing decision system deployed independently based on containerized services in a private cloud environment can effectively protect data privacy, and the asynchronous connection of each component through a private data pipeline also reduces additional latency overhead; on the other hand, the construction of training samples based on aligned call metric values ​​and decision point data can systematically transform the real result data accumulated from each call into effective training signals for route optimization, making the data more comprehensive; on the other hand, the incremental training of the routing decision model based on incremental samples in the flywheel training dataset can improve training efficiency and save computing power; and on the other hand, the use of the model hot update component to perform dual verification of the routing decision model can ensure the decision performance of the model, thereby realizing accurate LLM routing decision that is easy to scale based on the data flywheel mechanism. Attached Figure Description

[0009] Figure 1 This is a flowchart of a preferred embodiment of the LLM routing decision method based on a data flywheel according to the present invention.

[0010] Figure 2 This is the curve showing the evolution of the routing accuracy of this invention over time.

[0011] Figure 3 This is a functional block diagram of a preferred embodiment of the LLM routing decision device based on a data flywheel according to the present invention.

[0012] Figure 4 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the LLM routing decision method based on a data flywheel according to the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the LLM routing decision method based on a data flywheel according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0015] The LLM routing decision method based on data flywheel is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0016] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0017] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0018] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0019] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0021] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0022] The LLM (Large Language Model) routing decision method based on data flywheel described in this embodiment is applied to a routing decision system independently deployed based on containerized services in a private cloud environment. The routing decision system includes a data acquisition component, an alignment component, a training sample construction component, an incremental training component, and a model hot update component that are asynchronously connected through a private data pipeline.

[0023] The entire process of executing all components is completed within the enterprise's private cloud environment, with raw data remaining within the domain. System components are designed as independently deployable containerized services, supporting deployment models such as Kubernetes (K8s, an open-source container orchestration system), bare metal, and virtual machines, and integrating with the enterprise's existing data stack. Furthermore, running entirely within the enterprise's private environment, without relying on the cloud and with raw data remaining within the domain, it meets the data privacy requirements of scenarios with strong compliance requirements.

[0024] By asynchronously connecting various components through private data pipelines, the components are decoupled and do not need to wait for each other to execute, which can effectively reduce the additional latency overhead of online routing.

[0025] Specifically, the data flywheel-based LLM routing decision method includes: S10, in response to the routing decision instruction for the Large Language Model (LLM), the data acquisition component is invoked to synchronously record decision point data each time a routing decision is completed.

[0026] In this embodiment, the routing decision instruction can be automatically triggered when making routing decisions to achieve comprehensive data collection.

[0027] In this embodiment, the decision point data may include, but is not limited to, one or more of the following combinations of data: (1) Request feature vector: Request object vector (128-4096 dimensions) from the request semantic encoding module; (2) Request metadata: scene tag, tenant ID, timestamp; (3) Candidate model set: IDs of all candidate models considered in this decision-making process; (4) Selecting a model and the rationale for the decision; (5) The three objective predictions at the decision-making moment: prediction cost, prediction quality, and prediction delay; (6) Routing model version number.

[0028] Through the above embodiments, call data can be continuously accumulated and used for subsequent components.

[0029] S11. After each model call is completed, the call index value is calculated using the alignment component, and the call index value is aligned with the decision point data to obtain a sample pair.

[0030] In this embodiment, the LLM gateway generates millions to hundreds of millions of real call data points daily. Each data point contains important optimization signals, such as the predicted three target values ​​during routing decisions; the actual token consumption, actual latency, and actual quality after execution; and the success rate, failure rate, and user feedback of downstream services. These signals represent the highest quality supervisory data for optimizing the routing model, but existing solutions do not systematically feed them back to the strategy optimization stage. As a result, this data is only used for post-event reporting, and its value is far from being realized.

[0031] Therefore, in this embodiment, calculating the call index value using the alignment component includes: Obtain the score of the current model based on the pre-built quality assessment model, and get the first score; Obtain the user's explicit feedback score for the current call to the model as the second score; The success rate of downstream tasks is used as the third score. The first score, the second score, and the third score are normalized and weighted to obtain the actual quality quantification value of the model in the current call. Obtain the actual cost of the current call from the model vendor's billing receipt; The actual latency of the current call is measured based on the timestamp on the gateway side. Check whether the current call was successful and obtain the call result; The actual quality quantification value, the actual cost, the actual delay, and the call result are combined to obtain the call indicator value.

[0032] For example, the first score can be generated based on the correctness of the answer using RAGAS (Retrieval-Augmented Generation Assessment Suite), the summary quality using Rouge-L (Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence), and the code execution pass rate using HumanEval (Human Evaluation Dataset). Further, the second score is quantified using explicit user feedback such as likes, dislikes, and star ratings. Even further, the success rate of downstream tasks where the code is compilable or runnable and the question-answer is accepted is used as the third value. Finally, the first, second, and third scores can be normalized and weighted to obtain the actual quality quantification value of the model for that call. Simultaneously, the actual token consumption and cost of that call are obtained from the model provider's billing receipt and used as the actual cost of that call. Furthermore, the actual latency of that call (including first-byte latency and total latency) is measured based on the gateway-side timestamp. Additionally, it checks whether the call was successful, such as the success or failure status and error code, in order to remove abnormal samples caused by non-quality reasons.

[0033] For example, the call indicator value and the decision point data can be precisely aligned based on the request ID to form a pair of samples.

[0034] Through the above embodiments, it is possible to make the quality signal more reliable and comprehensive than that from a single source based on multi-source fusion.

[0035] S12, the training sample construction component is used to transform the sample pair to obtain training samples, and the training samples are written into the flywheel training dataset according to time partition.

[0036] In this embodiment, the step of transforming the sample pairs using the training sample construction component to obtain training samples includes: The request feature vector, candidate model index snapshot, scene label and decision time are obtained from the decision point data, and the request feature vector, the candidate model index snapshot, the scene label and the decision time are concatenated in sequence to obtain the input sample; By sequentially concatenating the actual cost, the actual quality quantification value, the actual delay, and the call result from the call indicator values, sample labels are obtained; The input sample and the sample label are combined sequentially to obtain the training sample.

[0037] For example: Input sample x = (request feature vector, candidate model index snapshot, scene label, decision time), sample label y = (actual cost, actual quality quantification value, actual latency, call result).

[0038] The candidate model index snapshot may include, but is not limited to, a combination of one or more of the following indicators: (1) Performance indicators: latency, throughput; (2) Availability: Error rate, health status; (3) Remaining quota for AI models (balance or token, capacity and rate limit); (4) Price (input unit price, output unit price); (5) Ability dimension.

[0039] Through the above embodiments, the real result data accumulated from each call (such as actual cost, actual quality quantification value, and actual latency) can be systematically transformed into effective training signals for route optimization, realizing the value upgrade of call data from post-event report material to continuous optimization fuel, thereby eliminating the waste of call data in the logs.

[0040] In this embodiment, the training samples are written into the flywheel training dataset and stored in time-partitioned format, which facilitates the rapid reading of incremental samples later.

[0041] In this embodiment, the newly deployed gateway lacks historical call data in the initial stage. If the flywheel is started from scratch, the routing decision quality will be low; if expert configuration is relied upon as the initial strategy, it will face the shortcomings of static strategies. Therefore, it is necessary to balance the availability during the cold start period with the long-term self-optimization capability.

[0042] Specifically, after writing the training samples into the flywheel training dataset according to time partitioning, the method further includes: When the sample size of the flywheel training dataset is less than the sample size threshold, a model pre-trained on a publicly available domain dataset is obtained as the routing decision model until the sample size of the flywheel training dataset is greater than or equal to the sample size threshold.

[0043] Through the above embodiments, a pre-trained routing model (trained based on publicly available industry data) can be used as the initial decision model when sample data accumulation is insufficient, and gradually switched to a flywheel optimization model as enterprise data accumulates. In this way, the system supports using a pre-trained routing model as the initial decision model, providing good basic routing capabilities before data accumulation. As enterprise data accumulates, the data flywheel gradually releases its full value. This progressive characteristic enables the system to work from day one and its quality continuously improves with long-term use.

[0044] S13, at preset time intervals, the incremental training component is used to obtain an initial model based on the current demand data, and incremental samples are read from the flywheel training dataset. The incremental samples are then used to incrementally train the initial model to obtain a routing decision model.

[0045] There is a fundamental contradiction between the static nature of routing strategies and the dynamic nature of the ecosystem. LLM technology and the business ecosystem are undergoing rapid evolution: model versions are updated monthly (performance leaps), pricing is adjusted quarterly (price wars and refinement), new models are constantly emerging (expanding the candidate set), and capability distribution is continuously reshaped (rapid changes in cost-effectiveness models for certain tasks). Once deployed, manually configured static routing strategies become decoupled from these ecosystem changes, and their optimality will predictably and continuously decline. Typically, routing accuracy decreases by 10%-15% after 3 months of deployment, and can drop by more than 30% after 6 months. Therefore, this embodiment continuously performs incremental automated updates to the model.

[0046] In this embodiment, the preset time interval can be configured according to actual scenario requirements, such as real-time, hourly, daily, or weekly.

[0047] In this embodiment, obtaining the initial model using the incremental training component based on the current demand data includes: When the current required data is such that the training speed is greater than the speed threshold and is interpretable, a lightweight gradient boosting model is obtained as the initial model; or When the current required data represents a strongly fitted nonlinear relationship, a small neural network regression model is obtained as the initial model; or When the current demand data contains a large language model identifier, the fine-tuned pre-trained large language model is obtained as the initial model.

[0048] For example, the lightweight gradient boosting model can be XGBoost (Extreme Gradient Boosting), LightGBM (Light Gradient Boosting Machine), etc.; the pre-trained large language model can be a pre-trained large language model fine-tuned using LoRA (Low-Rank Adaptation) and PEFT (Parameter-Efficient Fine-Tuning), which can be used when the routing decision-maker is a large model.

[0049] The above embodiments can support multiple model implementations, allowing enterprises to flexibly choose according to business complexity.

[0050] In this embodiment, when incrementally training the initial model using the incremental samples, the training objective can be to minimize the MSE (Mean Squared Error) loss between the predicted three target values ​​(i.e., cost, quality, and latency) and the actual three target values. It can also support sliding window data (avoiding unbounded growth), negative example augmentation (emphasizing the weight of failed samples), and training by scene label (avoiding dilution between different scenes), thereby comprehensively improving the model training effect.

[0051] Specifically, by accurately quantifying the deviation between prediction and reality, the flywheel can clearly quantify the discrepancy between the predicted value and the actual execution result during decision-making. Furthermore, it enables the routing decision model to learn which dimensions of prediction will deviate in which scenarios, thereby specifically improving prediction accuracy.

[0052] Incremental training allows for updating model parameters only for new data, avoiding the waste of computing power that would otherwise require full retraining each time, and making the training cost far lower than training from scratch.

[0053] Through the above embodiments, the routing strategy can continuously improve with the accumulation of call data, achieving a compound effect of continuous self-optimization. This evolutionary characteristic of becoming more accurate with use can continuously amplify value in long-term operation, far superior to traditional static solutions with one-time configuration. Furthermore, when the LLM model ecosystem changes (such as the launch of new models, pricing adjustments, capability upgrades, API (Application Programming Interface) changes, etc.), the flywheel can naturally adapt by absorbing new data without any manual configuration changes. This adaptive capability enables the solution to adapt to the rapidly evolving LLM ecosystem in the long term.

[0054] For example: Please see Figure 2 This is the curve showing the evolution of the routing accuracy of this invention over time. Figure 2 In the diagram, the upper curve represents the change in routing decision accuracy over runtime under the data flywheel scheme of this embodiment, while the lower curve represents the change in routing decision accuracy over runtime under the traditional static rule baseline scheme. It can be seen that as runtime increases, the routing decision accuracy of this embodiment gradually and steadily improves, exhibiting a clear monotonic upward trend as the called data continues to accumulate, increasing from an initial 55% to 92% after 6 months. In contrast, the accuracy of the static rule baseline scheme remains within a narrow range of 55%-57%, failing to continuously benefit from the accumulated data. Figure 2 This intuitively demonstrates the long-term routing quality advantages brought by the data flywheel.

[0055] S14, the routing decision model is double-validated using the model hot update component, and when the routing decision model passes the validation, the old model currently running online is atomically replaced by the routing decision model.

[0056] In this embodiment, the new routing model after incremental training may actually deteriorate in certain scenarios (such as data distribution drift, overfitting to the nearest sample, etc.).

[0057] Furthermore, there are generally discrepancies between the predicted values ​​at the decision-making moment (based on historical statistics or benchmark tests) and the actual execution results, including: cost prediction deviations, stemming from errors in context length and output length estimation; quality prediction deviations, stemming from inconsistencies between the evaluator and the real-world scenario; and latency prediction deviations, stemming from network jitter and model load fluctuations. If these deviations are not continuously corrected, they will accumulate with compound interest, causing the routing decision to deviate further and further from the optimal path.

[0058] Therefore, it is necessary to ensure that flywheel iteration does not introduce service quality degradation. Specifically, the dual validation of the routing decision model using the model hot update component includes: Obtain a retention set, and perform offline comparison and verification between the routing decision model and the old model based on the retention set; When the routing decision model passes the offline comparison verification, a preset proportion of real traffic is collected, and the real traffic is assigned to the routing decision model to execute the decision. Obtain multiple candidate metrics and the capability upgrade / downgrade verification conditions for each candidate metric; When the rise and fall of each candidate indicator meets the corresponding capability rise and fall verification conditions during the decision-making process, the routing decision model is determined to have passed verification; or When a candidate indicator is detected not to meet the corresponding capability upgrade / downgrade verification conditions during the decision-making process, the indicator type of the candidate indicator is identified, and the routing decision model is determined to pass the verification based on the indicator type. Specifically, when the indicator type is a pending confirmation type, the candidate indicator is pushed to a designated platform for confirmation to obtain a confirmation result, and the routing decision model is determined to have passed the verification based on the confirmation result; when the indicator type is a key indicator, the routing decision model is determined to have failed the verification.

[0059] For example, during offline comparative validation, the prediction accuracy, PR-AUC (Precision-Recall Area Under Curve), and three-objective MAE (Mean Absolute Error) of the new and old models can be compared on a retained set. Furthermore, a certain proportion (e.g., 5%) of real traffic can be copied to the new model for parallel decision-making, and secondary validation can be performed by verifying the rise and fall of multiple candidate metrics.

[0060] During secondary verification, a set of candidate capability indicators are evaluated. AI capabilities typically exhibit an inverse relationship; for example, candidate indicator A must improve by 10%, while candidate indicator B cannot decrease by 10%. If all indicators pass, the routing decision model is deemed validated. If not all indicators pass, further verification can be performed based on the indicator type. For instance, if a failed indicator is of the "pending confirmation" type, it can be sent to a designated platform for manual review (or indicators that fail secondary verification can be configured to be allowed; that is, indicators that fail secondary verification will not respond to the final result of the secondary verification). Alternatively, if a failed indicator is a core indicator, failure to pass secondary verification is not allowed; that is, once such a core indicator is detected as failing secondary verification, the routing decision model is directly deemed to have failed verification.

[0061] If the routing decision model fails the verification, the old model can be retained, and an alarm can be triggered.

[0062] Furthermore, the validated new model can atomically replace the old model running online without restarting the gateway service.

[0063] The newly trained routing decision model must undergo double verification before going online, and only after passing the verification can atomic hot replacement be performed. This conservative iterative mechanism makes the flywheel safety much higher than the traditional direct-to-online solution.

[0064] The above embodiments enable full automation based on flywheel closed loop, eliminating the need for continuous maintenance of routing configurations by experts. Furthermore, it frees up the operations and maintenance team from frequent firefighting and periodic tuning, allowing them to focus more on higher-value optimization work.

[0065] In this embodiment, the routing decision system may further include a flywheel operation data display panel, on which the following data can be displayed: (1) Data flow: Daily number of new samples, sample distribution in various scenarios; (2) Training health: training loss curve, retention set and accuracy trend of real traffic; (3) Evolution of routing quality: Curves showing the changes in routing accuracy and three-objective MAE over time; (4) Abnormal alarms: training failure, accuracy regression, abnormal data distribution drift.

[0066] The above embodiments can provide complete and real-time flywheel closed-loop data, which the operation and maintenance team can use to monitor the flywheel's operating status, identify stagnation or degradation, and determine whether an architecture upgrade is needed.

[0067] In this embodiment, after atomically replacing the currently running old model with the routing decision model, the method further includes: The routing decision model is continuously evaluated according to a preset time window; When the routing decision model is evaluated as having degraded capability within a consecutive preset number of preset time windows, the latest version model is obtained from the model pool that stores multiple historical stable version models. Roll back to the most recent version of the model.

[0068] For example, if the core indicators such as cost, quality, latency, reliability (e.g., error rate, SLA (Service Level Agreement)), and decision consistency decrease for N consecutive evaluation windows after the routing decision model is launched, it will automatically roll back to the previous stable version.

[0069] This embodiment employs the compounding mechanism of a data flywheel: more data makes the model more accurate; a more accurate model makes better decisions; better decisions attract more calls; and more calls generate more data. This compounding effect makes the solution's value over the long term far exceed that of a one-time configuration solution.

[0070] Traditional solutions heavily rely on domain experts' prior knowledge of the mapping relationships between various business scenarios and models. For large-scale deployments involving multiple tenants, multiple business scenarios (e.g., dozens to hundreds), and multiple models (e.g., a dozen to dozens), maintaining the workload of expert configuration, knowledge integrity, and cross-scenario consistency becomes difficult. When an expert leaves or the maintenance team changes, the routing strategy may drift into an unmaintained state due to knowledge gaps, leading to the non-scalability of expert configuration. Therefore, this embodiment addresses these issues by constructing a complete self-reinforcing closed-loop system that systematically transforms the real result data accumulated from each call into continuous optimization signals for the routing strategy, while ensuring enterprise data privacy, cold start availability, and stable iteration.

[0071] As can be seen from the above technical solutions, on the one hand, the routing decision system deployed independently based on containerized services in a private cloud environment can effectively protect data privacy, and the asynchronous connection of each component through a private data pipeline also reduces additional latency overhead; on the other hand, the construction of training samples based on aligned call metric values ​​and decision point data can systematically transform the real result data accumulated from each call into effective training signals for route optimization, making the data more comprehensive; on the other hand, the incremental training of the routing decision model based on incremental samples in the flywheel training dataset can improve training efficiency and save computing power; and on the other hand, the use of the model hot update component to perform dual verification of the routing decision model can ensure the decision performance of the model, thereby realizing accurate LLM routing decision that is easy to scale based on the data flywheel mechanism.

[0072] like Figure 3 The diagram shown is a functional block diagram of a preferred embodiment of the LLM routing decision device based on a data flywheel according to the present invention. The LLM routing decision device 11 based on a data flywheel runs on a routing decision system independently deployed based on containerized services in a private cloud environment. The routing decision system includes a data acquisition component, an alignment component, a training sample construction component, an incremental training component, and a model hot update component, asynchronously connected through a private data pipeline. Specifically, it includes a recording unit 110, an alignment unit 111, a conversion unit 112, a training unit 113, and a verification unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0073] The recording unit 110 is used to respond to the routing decision instruction of the large language model LLM and call the data acquisition component to synchronously record the decision point data when each routing decision is completed. The alignment unit 111 is used to calculate the call index value using the alignment component after each model call, and to align the call index value with the decision point data to obtain a sample pair. The conversion unit 112 is used to convert the sample pairs using the training sample construction component to obtain training samples, and write the training samples into the flywheel training dataset according to time partitions. The training unit 113 is used to obtain an initial model based on the current demand data using the incremental training component at preset time intervals, and to read incremental samples from the flywheel training dataset and use the incremental samples to incrementally train the initial model to obtain a routing decision model. The verification unit 114 is used to perform dual verification of the routing decision model using the model hot update component, and when the routing decision model passes the verification, it atomically replaces the old model currently running online with the routing decision model.

[0074] As can be seen from the above technical solutions, on the one hand, the routing decision system deployed independently based on containerized services in a private cloud environment can effectively protect data privacy, and the asynchronous connection of each component through a private data pipeline also reduces additional latency overhead; on the other hand, the construction of training samples based on aligned call metric values ​​and decision point data can systematically transform the real result data accumulated from each call into effective training signals for route optimization, making the data more comprehensive; on the other hand, the incremental training of the routing decision model based on incremental samples in the flywheel training dataset can improve training efficiency and save computing power; and on the other hand, the use of the model hot update component to perform dual verification of the routing decision model can ensure the decision performance of the model, thereby realizing accurate LLM routing decision that is easy to scale based on the data flywheel mechanism.

[0075] like Figure 4 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the LLM routing decision method based on a data flywheel according to the present invention.

[0076] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as an LLM routing decision program based on a data flywheel.

[0077] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0078] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0079] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of an LLM routing decision program based on a data flywheel, but also to temporarily store data that has been output or will be output.

[0080] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing an LLM routing decision program based on a data flywheel), and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0081] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the LLM routing decision method based on data flywheel described above, for example... Figure 1 The steps are shown.

[0082] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a recording unit 110, an alignment unit 111, a conversion unit 112, a training unit 113, and a verification unit 114.

[0083] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the LLM routing decision method based on a data flywheel as described in the various embodiments of this invention.

[0084] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0085] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0086] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0087] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0088] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 4 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0089] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0090] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0091] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0092] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0093] It will be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0094] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement an LLM routing decision method based on a data flywheel, and the processor 13 can execute the multiple instructions to achieve: In response to the routing decision instruction for the Large Language Model (LLM), the data acquisition component is invoked to synchronously record decision point data each time a routing decision is completed; After each model call is completed, the alignment component is used to calculate the call index value, and the call index value is aligned with the decision point data to obtain a sample pair; The training sample construction component is used to transform the sample pairs to obtain training samples, and the training samples are written into the flywheel training dataset according to time partitions. At preset time intervals, the incremental training component is used to obtain an initial model based on the current demand data, and incremental samples are read from the flywheel training dataset. The incremental samples are then used to incrementally train the initial model to obtain a routing decision model. The routing decision model is double-validated using the model hot update component, and when the routing decision model passes the validation, it atomically replaces the old model currently running online.

[0095] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0096] It should be noted that all the data involved in this case was legally obtained.

[0097] If any AI models, software tools, or components not belonging to this company appear in the embodiments of this invention, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this invention has been obtained by an entity authorized (with the knowledge and consent) or fully authorized by all parties through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0098] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0099] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0100] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0103] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0104] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An LLM routing decision method based on a data flywheel, characterized in that, A routing decision system is applied to independently deployed containerized services in a private cloud environment. The routing decision system includes a data acquisition component, an alignment component, a training sample construction component, an incremental training component, and a model hot update component, asynchronously connected via a private data pipeline. The LLM routing decision method based on a data flywheel includes: In response to the routing decision instruction for the Large Language Model (LLM), the data acquisition component is invoked to synchronously record decision point data each time a routing decision is completed; After each model call is completed, the alignment component is used to calculate the call index value, and the call index value is aligned with the decision point data to obtain a sample pair; The training sample construction component is used to transform the sample pairs to obtain training samples, and the training samples are written into the flywheel training dataset according to time partitions. At preset time intervals, the incremental training component is used to obtain an initial model based on the current demand data, and incremental samples are read from the flywheel training dataset. The incremental samples are then used to incrementally train the initial model to obtain a routing decision model. The routing decision model is double-validated using the model hot update component, and when the routing decision model passes the validation, the old model currently running online is atomically replaced by the routing decision model. The dual verification of the routing decision model using the model hot update component includes: obtaining a retention set and performing offline comparison verification between the routing decision model and the old model based on the retention set; when the routing decision model passes the offline comparison verification, collecting a preset proportion of real traffic and assigning the real traffic to the routing decision model to execute the decision; obtaining multiple candidate indicators and capability upgrade / downgrade verification conditions for each candidate indicator; when the upgrade / downgrade of each candidate indicator meets the corresponding capability upgrade / downgrade verification conditions during the decision execution process, determining that the routing decision model has passed verification; or when a candidate indicator is detected not meeting the corresponding capability upgrade / downgrade verification conditions during the decision execution process, identifying the indicator type of the candidate indicator and determining whether the routing decision model has passed verification based on the indicator type; wherein, when the indicator type is a pending confirmation type, pushing the candidate indicator to a designated platform for confirmation to obtain a confirmation result, and determining whether the routing decision model has passed verification based on the confirmation result; when the indicator type is a key indicator, determining that the routing decision model has failed verification.

2. The LLM routing decision method based on a data flywheel as described in claim 1, characterized in that, The calculation of the call indicator value using the alignment component includes: Obtain the score of the current model based on the pre-built quality assessment model, and get the first score; Obtain the user's explicit feedback score for the current call to the model as the second score; The success rate of downstream tasks is used as the third score. The first score, the second score, and the third score are normalized and weighted to obtain the actual quality quantification value of the model in the current call. Obtain the actual cost of the current call from the model vendor's billing receipt; The actual latency of the current call is measured based on the timestamp on the gateway side. Check whether the current call was successful and obtain the call result; The actual quality quantification value, the actual cost, the actual delay, and the call result are combined to obtain the call indicator value.

3. The LLM routing decision method based on a data flywheel as described in claim 2, characterized in that, The step of using the training sample construction component to transform the sample pairs to obtain training samples includes: The request feature vector, candidate model index snapshot, scene label and decision time are obtained from the decision point data, and the request feature vector, the candidate model index snapshot, the scene label and the decision time are concatenated in sequence to obtain the input sample; By sequentially concatenating the actual cost, the actual quality quantification value, the actual delay, and the call result from the call indicator values, sample labels are obtained; The input sample and the sample label are combined sequentially to obtain the training sample.

4. The LLM routing decision method based on a data flywheel as described in claim 1, characterized in that, After writing the training samples into the flywheel training dataset according to time partitioning, the method further includes: When the sample size of the flywheel training dataset is less than the sample size threshold, a model pre-trained on a publicly available domain dataset is obtained as the routing decision model until the sample size of the flywheel training dataset is greater than or equal to the sample size threshold.

5. The LLM routing decision method based on a data flywheel as described in claim 1, characterized in that, The step of using the incremental training component to obtain the initial model based on the current demand data includes: When the current required data is such that the training speed is greater than the speed threshold and is interpretable, a lightweight gradient boosting model is obtained as the initial model; or When the current required data represents a strongly fitted nonlinear relationship, a small neural network regression model is obtained as the initial model; or When the current demand data contains a large language model identifier, the fine-tuned pre-trained large language model is obtained as the initial model.

6. The LLM routing decision method based on a data flywheel as described in claim 1, characterized in that, After replacing the currently running old model with the routing decision model atomically, the method further includes: The routing decision model is continuously evaluated according to a preset time window; When the routing decision model is evaluated as having degraded capability within a consecutive preset number of preset time windows, the latest version model is obtained from the model pool that stores multiple historical stable version models. Roll back to the most recent version of the model.

7. An LLM routing decision-making device based on a data flywheel, characterized in that, Used to perform the data flywheel-based LLM routing decision method as described in any one of claims 1 to 6.

8. A computer device, characterized in that, The computer device includes: A memory for storing at least one instruction; and a processor for executing the instructions stored in the memory to implement the LLM routing decision method based on a data flywheel as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the LLM routing decision method based on a data flywheel as described in any one of claims 1 to 6.

Images