Large language model switching method and device and related equipment
By using dynamic priority transfer technology, the processing requests of large language models are obtained and updated in real time, which solves the problem of low efficiency in switching large models and achieves seamless switching and efficient service.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-03-24
AI Technical Summary
With limited computing resources, existing technologies require waiting for the current engine to process all requests before starting a new engine when switching between large models, which can lead to service interruptions or performance degradation, affecting user experience, especially in high-traffic scenarios where efficiency is low.
By acquiring real-time information on the processing requests in the large language model, dynamically updating the transfer priority, and transferring the processing requests to the new large language model for continued processing, the seamlessness and efficiency of the model switching process are ensured.
It enables rapid switching of large language models without service interruption, improving switching efficiency and ensuring continuity of user experience and service quality.
Smart Images
Figure CN121724147A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, and related equipment for switching large language models. Background Technology
[0002] The main challenge in deploying large-scale model services is effectively managing user traffic fluctuations and providing high-concurrency, high-quality services with limited computing resources. In related technologies, model switching requires waiting for the current engine to process all requests before starting the new engine, leading to service interruptions or performance degradation, impacting user experience. This is especially true in high-traffic scenarios, where prolonged waiting times significantly degrade the user experience, resulting in low efficiency for large-scale model switching. Summary of the Invention
[0003] This application provides a method, apparatus, and related equipment for switching large language models, which solves the problem of low efficiency in switching large models in the prior art.
[0004] To solve the above problems, this application is implemented as follows:
[0005] In a first aspect, embodiments of this application provide a method for switching large language models, applied to a computing platform, the computing platform including at least a first large language model and a second large language model, the method including:
[0006] When a model switching instruction is received to switch from the first major language model to the second major language model, the N processing requests that are being processed in the first major language model and are waiting to be transferred are obtained, where N is a positive integer;
[0007] Obtain N runtime information corresponding to N processing requests in real time;
[0008] The transfer priority of N processing requests is dynamically updated based on N pieces of runtime information;
[0009] Based on the transfer priority, the runtime information corresponding to the processing request is transferred to the second language model, which then continues to process the corresponding processing request based on the transferred runtime information.
[0010] When all N processing requests meet the condition that the information transfer is complete or has been executed by the first language model, confirm that the model switching instruction has been executed.
[0011] Optionally, the transfer priority of N processing requests is dynamically updated based on N pieces of runtime information, including:
[0012] Based on the runtime information of the processing request, calculate the amount of data to be transferred, where the runtime information includes the intermediate processing data of the processing request;
[0013] Based on the amount of data to be transferred, obtain the transfer cost for the corresponding processing request;
[0014] The transfer priority is determined based on the transfer cost of each processing request, where the transfer cost and transfer priority are negatively correlated.
[0015] Optionally, the intermediate processing data includes at least one of the following: processed token sequences, caches in the attention mechanism, dialogue history context, partial output text, the state required by the first language model, and the hidden states of each layer.
[0016] Optionally, the second language model continues to process the corresponding processing requests based on the transition runtime information, including:
[0017] The differences between the first and second language models are analyzed, and these differences represent the structural differences between the model structures of the first and second language models.
[0018] The target operation information is processed based on the difference information to obtain the target operation information. The data structure of the target operation information is adapted to the second language model. The target processing includes at least one of the following: interpolation processing and truncation processing.
[0019] The second largest language model then continues to process the corresponding processing requests based on the target runtime information.
[0020] Optionally, before receiving the model switching instruction to switch from the first to the second largest language model, the method further includes:
[0021] Obtain historical and current traffic data from the computing platform;
[0022] Based on historical and current traffic data, the traffic data of the computing platform in the target time period is predicted to obtain the target traffic data.
[0023] If the currently running model combination cannot meet the target traffic data, a model switching instruction is generated. The model switching instruction includes a complete switching instruction and a partial switching instruction. The currently running model combination includes the first major language model.
[0024] Optionally, if the currently running model combination cannot meet the target traffic data, a model switching instruction is generated, including:
[0025] Based on the target traffic data, generate multiple candidate model combinations, each of which includes at least one large language model.
[0026] Calculate the evaluation metrics for each candidate model combination. The evaluation metrics indicate the service changes of the computing platform after switching from the currently running model combination to the candidate model combination.
[0027] The weights of the evaluation indicators are dynamically updated based on the current status information of the computing platform.
[0028] Based on the evaluation indicators and their corresponding weights, a target model combination is determined from multiple candidate model combinations, including the second largest language model.
[0029] Generate a model switching instruction to switch the first largest language model to the second largest language model.
[0030] Optionally, calculate the evaluation metrics for each candidate model combination, including:
[0031] Calculate the resource utilization rate for each candidate model combination. The resource utilization rate indicates the service efficiency of the corresponding candidate model combination.
[0032] Calculate the switching cost from the currently running model combination to each candidate model combination. The switching cost indicates the resource consumption of switching from the currently running model combination to each candidate model combination.
[0033] The evaluation indicators include resource utilization and switching costs.
[0034] Secondly, embodiments of this application provide a large language model switching device, the device comprising:
[0035] The request acquisition module is used to acquire N processing requests that are being processed and are waiting to be transferred in the first language model when a model switching instruction is received to switch the first language model to the second language model. N is a positive integer.
[0036] The runtime information acquisition module is used to acquire N runtime information corresponding to N processing requests in real time;
[0037] The priority determination module is used to obtain the dynamically updated transfer priority of N processing requests based on N pieces of runtime information;
[0038] The transfer module is used to transfer the runtime information corresponding to the processing request to the second language model according to the transfer priority. The second language model then continues to process the corresponding processing request based on the transferred runtime information.
[0039] The confirmation module is used to confirm that the model switching instruction has been executed when all N processing requests meet the conditions of completed information transfer or completed execution by the first language model.
[0040] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps of the method described in the first aspect above.
[0041] Fourthly, embodiments of this application also provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps of the method described in the first aspect above.
[0042] Fifthly, embodiments of this application also provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps in the method described in the first aspect above. Attached Figure Description
[0043] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is one of the flowcharts illustrating a large language model switching method provided in an embodiment of this application;
[0045] Figure 2 A second flowchart illustrating a large language model switching method provided in an embodiment of this application;
[0046] Figure 3 The third flowchart illustrating a large language model switching method provided in this application embodiment;
[0047] Figure 4 The fourth flowchart illustrating a large language model switching method provided in this application embodiment;
[0048] Figure 5 The fifth flowchart illustrating a large language model switching method provided in this application embodiment;
[0049] Figure 6 A flowchart illustrating a large language model switching method provided in this application embodiment is shown in Figure 6.
[0050] Figure 7 A schematic diagram of a large language model switching device provided in an embodiment of this application;
[0051] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0053] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, computing platform, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.
[0054] See Figure 1 , Figure 1 This is one of the flowcharts illustrating the large language model switching method provided in the embodiments of this application. Figure 1 The large language model switching method shown can be executed by a computer. For example... Figure 1 As shown, the large language model switching method may include the following steps:
[0055] Step 200: When a model switching instruction is received to switch the first language model to the second language model, obtain the N processing requests that are being processed in the first language model and are waiting to be transferred, where N is a positive integer.
[0056] In this embodiment, the large language model switching method provided in this application is applied to a computing platform, which includes at least a first large language model and a second large language model. After the computing platform receives a model switching instruction that requires switching the current first large language model to the second large language model, it obtains multiple processing requests currently being executed in the first large language model and awaiting transfer. The model switching instruction is either a complete switching instruction or a partial switching instruction, and correspondingly, the multiple processing requests awaiting transfer are all currently being processed requests or some currently being processed requests.
[0057] In this embodiment, the computing platform can consist of the following three core components: StrategyGenerator: analyzes real-time traffic data and generates the optimal large language model configuration strategy; LoadBalancer: intelligently allocates user requests to available large language models; and Engine Manager: manages the lifecycle of large language models and executes the instructions of the strategy generator.
[0058] Specifically, when traffic prediction indicates that the size of the large language model needs to be adjusted, the strategy generator generates a new large language model configuration. In this embodiment, the large language model configuration is illustrated by switching the first large language model to the second large language model. In other embodiments, it may also include multiple other model switching instructions, such as a model switching instruction to switch the second large language model to the first large language model, a model switching instruction to switch the third large language model to both the second and first large language models, and so on.
[0059] Step 400: Obtain N running information corresponding to N processing requests in real time.
[0060] In this embodiment, after obtaining N processing requests that are being processed or awaiting transfer in the first large language model, N pieces of runtime information corresponding to the N processing requests are obtained in real time. This runtime information may include request status, processing time, resource utilization, and processing logs, etc. Request status indicates whether the processing request is being processed or has failed. Processing time indicates the current processing time of the request. Resource utilization indicates the CPU, memory, and network resources used by the current processing request, etc. Processing logs indicate the task progress and context information of the current processing request, etc.
[0061] It should be noted that since the process of switching from the first language model to the second language model is a dynamic switching process, while the data of the first language model is gradually transferred to the second language model, the first language model continues to process the data that has not been transferred, rather than stopping. That is, the running information in this embodiment is dynamic information, so it is necessary to acquire the running information in real time to ensure the accuracy of N running information.
[0062] Step 600: Obtain the dynamically updated transfer priority of N processing requests based on N running information.
[0063] In this embodiment, the obtained runtime information is sorted to determine the transfer priority of the corresponding processing requests. The transfer priority indicates whether the corresponding processing request needs to be transferred first. The priority of the processing requests is dynamically adjusted based on the currently obtained runtime information, thereby ensuring that computing platform resources are utilized more effectively and that critical requests can be processed in a timely manner when needed.
[0064] Transfer priority can be determined by several factors, such as the importance of the request, the urgency of the request, and the resource consumption rate of the request. For example, more important requests should be processed first, more urgent requests should be processed first, and requests with low resource consumption rates should be processed first.
[0065] Furthermore, for large language models that simultaneously process multiple requests within a computing platform, the platform can dynamically adjust the priority ranking rules and execution flow for request transfers. For example, when the large language model is under high load, it can process some simpler requests first to reduce its overall load, and then transfer requests based on their resource consumption from smallest to largest. If business requirements change and the priority criterion for transfer needs to be switched from resource consumption to request importance, the priority of all requests to be transferred will be reordered in real time based on the new criteria and the changes will take effect. Moreover, when multiple transfer priority criteria are considered, the weights of each criterion can be adjusted according to the core criteria at the current stage, and the final transfer priority of the requests to be transferred can be determined based on the weighted result.
[0066] Among them, when there is only one processing request being processed and waiting to be transferred in the first major language model, that is, when N equals 1, the processing request is given the highest priority.
[0067] When there are multiple processing requests that are being processed and awaiting transfer in the first major language model, that is, when N is a positive integer greater than 1, the transfer priority is dynamically updated based on the running information corresponding to the multiple processing requests.
[0068] Step 800: Transfer the runtime information corresponding to the processing request to the second language model according to the transfer priority. The second language model then continues to process the corresponding processing request based on the transferred runtime information.
[0069] In this embodiment, according to the order of transfer priority, the running information corresponding to one or more of the N processing requests is transferred from the first large language model to the second large language model, and the second large language model processes the transferred processing requests according to the transferred running information.
[0070] It should be noted that after the transfer of runtime information corresponding to a processing request is successful, the first language model will no longer process that request. If the transfer of runtime information fails, the processing request corresponding to the failed transfer will continue to be processed by the first language model. New processing requests generated during model switching, if according to the allocation rules they should be processed by the second language model, will be directly sent to the second language model for processing, and will not be sent to the first language model.
[0071] Step 1000: When all N processing requests meet the condition that the running information transfer is complete or has been executed by the first major language model, confirm that the model switching instruction has been executed.
[0072] In this embodiment, when all N processing requests meet the condition of either being transferred or completed by the first language model, it indicates that there are no longer any requests that the first language model needs to process. These N processing requests have either been taken over by the second language model or have been completed by the first language model. At this point, if the model switching instruction is a complete switching instruction (i.e., transferring all requests being processed by the first language model to the second language model), then the first language model can be shut down, and the complete switching instruction has been executed. If the model switching instruction is a partial switching instruction (i.e., transferring some requests being processed by the first language model to the second language model), then the first language model will no longer process these N processing requests, and the partial switching instruction has been executed.
[0073] To illustrate with a specific implementation, for example, when the computing platform detects that the user concurrency has increased from 1000 to 3000, the platform's policy generator determines that a larger language model is needed to support more concurrency and generates an optimal large language model configuration strategy. The platform's load balancer determines which large language models need to be transferred from the currently running large language models. The platform's large language model manager transfers the runtime information of the currently processed, pending transfer requests to the corresponding newly deployed large language models. The new large language model begins receiving new requests, while the old large language model continues to process the remaining requests until all pending transfers are completed. The computing platform then fully switches to the new large language model combination. When the traffic drops to 1000, the computing platform can reverse the switch to provide optimal service with the best resource configuration.
[0074] Upon receiving a model switching instruction, this embodiment analyzes the N processing requests currently being processed and awaiting transfer in the first large language model, obtains the running information corresponding to each request in real time, determines the transfer priority of each of the N requests based on the running information, and then transfers the running information of the N requests to the second large language model according to the transfer priority. This allows processing requests to be transferred from the first large language model to the second large language model without interrupting processing. This achieves seamless switching of large language models and improves the switching efficiency.
[0075] In some feasible implementations, optionally, step 600, obtaining the dynamically updated transfer priority of N processing requests based on N operational information, includes:
[0076] Step 610: Calculate the amount of data to be transferred based on the running information of the processing request, wherein the running information includes the intermediate processing data of the processing request;
[0077] Step 620: Based on the amount of data to be transferred, obtain the transfer cost of the corresponding processing request;
[0078] Step 630: Determine the corresponding transfer priority based on the transfer cost of each processing request, wherein the transfer cost is negatively correlated with the transfer priority.
[0079] In this embodiment, as Figure 2 As shown, Figure 2 This is the second flowchart illustrating the large language model switching method provided in this application embodiment. After acquiring the runtime information corresponding to each processing request in real time, the amount of data to be transferred for each processing request is calculated based on the runtime information. The runtime information includes the intermediate processing data of the processing request. Therefore, the transfer cost corresponding to each processing request is calculated based on the amount of data to be transferred. For example, the higher the transfer cost of a processing request, the lower its transfer priority; conversely, the lower the transfer cost of a processing request, the higher its transfer priority. If most requests are in a high-cost state, the transfer is temporarily suspended, and some low-priority requests are processed first to reduce the overall state complexity. Based on the number and type of remaining pending requests, the priority transfer target is dynamically selected: requests with stable states and low costs are prioritized for transfer.
[0080] It should be noted that when the processing request is in any of the following stages: the initial context ready stage after pre-filling, the autoregressive generation and M-token check stage, the attention cache stabilization stage, or the complex operator execution gap stage, the processing request is in a low-switch-cost stage. By analyzing the internal processing flow of the large language model, the optimal switching time is determined to minimize the amount of data that needs to be transferred, thereby reducing the switching time.
[0081] Specifically, runtime information includes intermediate processing data for processing requests. Optionally, intermediate processing data includes at least one of the following: processed token sequences, cache in the attention mechanism, dialogue history context, partial output text, the required state of the first large language model, and the hidden states of each layer. Intermediate processing data represents intermediate computation results and contextual information generated during the internal processing of the large language model. This information collectively constitutes the "runtime state" of the request during model inference, which is crucial for the model to continuously generate coherent responses. The main purpose of transferring runtime information is to achieve seamless service during model switching. In traditional scaling solutions, when a large language model needs to be replaced (e.g., switching from a large model to a small model to improve concurrency), the computing platform must wait for all ongoing requests to be fully processed on the old large language model before terminating the old large language model and starting the new one. This waiting process can lead to service delays of several seconds or even longer, severely impacting user experience. The technical solution of this application, by migrating runtime information from the old large language model to the new large language model, allows the computing platform to directly "take over" and continue processing these requests without interrupting service, thereby significantly shortening the switching time and avoiding service interruption. During the information transfer process, the request being processed is handled by the old large language model. After a successful transfer, the new large language model takes over and continues the subsequent reasoning process. The lifecycle of the request is not interrupted; only the execution environment is smoothly migrated from the old large language model to the new large language model.
[0082] During the request transfer process, the new large language model begins receiving new requests, while the old large language model continues processing the remaining requests until all requests have been transferred. "Remaining requests" refer to those requests that have entered the processing flow but have not yet completed response generation when the computing platform decides to switch large language models. These requests are in a "running" state, and their running information has been captured and transferred. During the running information transfer, the old large language model continues processing requests that have not yet been transferred or have failed to be transferred, while the new large language model begins processing newly arriving requests and the subsequent computation of the "remaining requests" transferred from the old large language model. Once all remaining requests have been transferred to the new large language model for processing, or have already been processed by the old large language model (e.g., failed transfer requests, or requests that were processed in the old large language model before the transfer), the old large language model will no longer have any active requests awaiting transfer, thus completing the entire switchover process.
[0083] Optionally, step 800 involves the second language model continuing to process the corresponding processing request based on the transition runtime information, including:
[0084] Step 810: Analyze the difference information between the first and second language models. The difference information represents the structural differences between the model structure of the first and second language models.
[0085] Step 820: Perform target processing on the transferred operation information based on the difference information to obtain target operation information. The data structure of the target operation information is adapted to the second language model. Target processing includes at least one of the following: interpolation processing and truncation processing.
[0086] Step 830: The second language model continues to process the corresponding processing requests based on the target runtime information.
[0087] In this embodiment, as Figure 3 As shown, Figure 3 This is the third flowchart illustrating the large language model switching method provided in this application embodiment. The structural differences between the first and second large language models may include differences in parameter scale, architecture, resource quantity, etc.
[0088] The transferred runtime information undergoes target processing to adapt it to the second major language model. This target processing includes at least one of the following: interpolation and truncation. Interpolation refers to situations where data may need to be supplemented or transformed to fit the input format or context required by the second major language model. Specifically, interpolation can take the form of numbers, text, or expansions of contextual sentences. Truncation refers to truncating specific parts of the output if the output of the first major model exceeds the processing capacity of the second major language model (e.g., input length), retaining important parts and removing redundant information. The second major language model understands the changes in the target runtime information and continues to process the corresponding processing requests based on the target runtime information.
[0089] The following describes a specific implementation method. Specifically, the computing platform detects a large language model switching command and triggers a request transfer process. It collects currently processed and pending transfer requests, obtaining their current runtime information. The complexity of the runtime information for each request is evaluated, and the transfer cost is calculated based on this complexity. It determines whether the current time is optimal for transfer: if most requests are in a high-cost state, the transfer is postponed, and some low-priority requests are processed first to reduce overall costs. Based on the number and type of remaining pending requests, a priority transfer target is dynamically selected: requests with stable states and low costs are prioritized for transfer. The compatibility of the converted runtime information on the new large language model is verified, ensuring that the input dimension matches the new model structure, and interpolation or truncation is performed if necessary. The converted runtime information is injected into the corresponding request instance in the new large language model, marking the request as "migrated." The new large language model continues inference from the migrated state, generating subsequent tokens to ensure output consistency with the original large language model. After confirming successful request migration, the old large language model stops processing the request and releases related resources. This ensures that users do not perceive service interruption during the request transfer process, and that request processing results remain consistent. Through precise control and verification mechanisms, the correctness and consistency of request transfer are ensured, providing a seamless user experience.
[0090] Optionally, prior to step 200, the large language model switching method provided in this application embodiment further includes:
[0091] Step 110: Obtain historical and current traffic data from the computing platform;
[0092] Step 120: Based on historical traffic data and current traffic data, predict the traffic data of the computing platform in the target time period to obtain the target traffic data;
[0093] Step 130: If the currently running model combination cannot meet the target traffic data, generate a model switching instruction. The model switching instruction includes a complete switching instruction and a partial switching instruction. The currently running model combination includes the first major language model.
[0094] In this embodiment, as Figure 4 As shown, Figure 4 This is the fourth flowchart illustrating the large language model switching method provided in this application. Historical traffic data refers to traffic data collected over a certain period of time in the past. This traffic data can help identify traffic patterns, trends, and seasonal changes. Current traffic data refers to the traffic status observed at the current moment. By comparing historical traffic data and current traffic data, it is possible to better understand the current traffic position within the normal range.
[0095] By combining historical and current traffic data, the traffic data of the computing platform during a target time period is predicted to obtain the target traffic data. The prediction method can be various, including time series analysis, regression analysis, and machine learning, and is not specifically limited in this embodiment. The target traffic data refers to the traffic demand predicted by the model for a future time period. If the currently running model combination (including the first large language model) cannot meet the target traffic data, it indicates a need for more processing power. This means a second large language model, larger than the first, is needed, or a second large language model needs to be added to share some of the requests from the first large language model, thereby ensuring that the current computing platform can meet the traffic demand. When the first language model needs to be completely replaced by the second language model, a complete switch instruction is generated, which transfers all requests being processed in the first language model to the second language model. When the second language model needs to be added to share the load of the first language model, a partial switch instruction is generated, which transfers some of the requests being processed in the first language model to the second language model. The part to be transferred can be determined randomly or based on the complexity, importance, and urgency of the requests. This embodiment does not limit this.
[0096] In this embodiment, the traffic can be analyzed and predicted by the strategy generator in the computing platform to generate the optimal large language model combination configuration strategy. The load balancer then allocates different requests to be transferred to the large language models. These different requests include processing requests that are currently being processed in the large language model and those that are to be transferred, as well as new processing requests.
[0097] Optionally, step 130, if the currently running model combination cannot meet the target traffic data, generates a model switching instruction, including:
[0098] Step 131: Based on the target traffic data, generate multiple candidate model combinations, each candidate model combination including at least one large language model;
[0099] Step 132: Calculate the evaluation index for each candidate model combination. The evaluation index indicates the service change data of the computing platform after switching from the currently running model combination to the candidate model combination.
[0100] Step 133: Dynamically update the weights corresponding to the evaluation indicators based on the current status information of the computing platform;
[0101] Step 134: Based on the evaluation indicators and their corresponding weights, determine the target model combination from multiple candidate model combinations, wherein the target model combination includes the second largest language model;
[0102] Step 135: Generate a model switching instruction to switch the first major language model to the second major language model.
[0103] In this embodiment, as Figure 5 As shown, Figure 5 This is the fifth flowchart illustrating the large language model switching method provided in this application. The candidate model combination includes different combinations of multiple large language models, such as a combination of the first and second large language models, a combination of the second and third large language models, or a single second large language model. By selecting candidate model combinations, it is ensured that the computing platform can maintain high performance and high availability during peak target traffic periods.
[0104] The evaluation metrics for each candidate model combination are used to indicate the changes in computing platform services before and after the switch. For example, evaluation metrics may include response time, accuracy, resource utilization, switching costs, and user satisfaction, etc. By calculating the evaluation metrics for each candidate model combination, the degree of matching of each model combination in the current computing platform is determined, thereby determining the optimal model combination.
[0105] In addition, the weights of the evaluation metrics can be dynamically updated based on the current status information of the computing platform. This status information can include load and current traffic, thereby dynamically updating the weight of each evaluation metric. For example, taking traffic as the status information, during periods of rapid traffic increase, the computing platform prioritizes service capacity, assigning a higher weight to resource utilization; during periods of stable traffic, the stability of the computing platform is emphasized, assigning a higher weight to switching costs (the lower the total switching cost, the better), to ensure service quality.
[0106] Based on the evaluation metrics and weights corresponding to each candidate model combination, a score is calculated for each combination, resulting in multiple score values. The target model combination (including the second largest language model) is then determined based on these scores. The target model combination outperforms other combinations in terms of overall evaluation metrics and is adapted to current traffic demands. This generates a model switching instruction to switch the first largest language model to the second largest. It should be noted that there may be overlapping large language models between the candidate model schemes and the currently running model scheme; the higher the overlap, the lower the switching cost.
[0107] Optionally, step 132 involves calculating the evaluation metrics for each candidate model combination, including:
[0108] Step 1321: Calculate the resource utilization rate corresponding to each candidate model combination. The resource utilization rate indicates the service efficiency of the corresponding candidate model combination.
[0109] Step 1322: Calculate the switching cost from the currently running model combination to each candidate model combination. The switching cost indicates the resource consumption of switching from the currently running model combination to each candidate model combination.
[0110] The evaluation indicators include resource utilization and switching costs.
[0111] In this embodiment, as Figure 6 As shown, Figure 6 This is the sixth flowchart illustrating the large language model switching method provided in this application. In this embodiment, the resource utilization and switching cost of different large language model combinations are calculated based on predicted traffic, and the optimal large language model combination is selected. Resource utilization reflects the service efficiency of the current large language model combination under predicted traffic, that is, the combined performance of the number of concurrent requests and response quality supported under limited computing resources (such as GPU memory and computing power). Switching cost measures the time, computational overhead, and impact on service continuity required to change from the current large language model combination to the new combination. It mainly includes the startup time of the new large language model combination, the termination overhead of the old large language model combination, the resource consumption of state transition, and potential service jitter risks.
[0112] The computing platform employs a dynamic prioritization evaluation framework. Specifically, the platform first generates multiple feasible candidate combinations of large language models based on predicted traffic (e.g., 3 large models + 1 medium model, or 2 medium models + 4 small models, etc.). For each candidate, the platform evaluates its resource utilization score and switching cost score.
[0113] Resource utilization score is primarily based on a comprehensive evaluation of the combination's theoretical throughput, average response latency, and model service quality level under predicted load. Higher utilization and lower latency result in a higher score. Switching cost score is evaluated based on the difference between the current running combination and the target combination, including the number of large language models that need to be started / terminated and the number of requests involving state transitions. The greater the difference and the more complex the operation, the higher the cost score (i.e., the less optimal).
[0114] The final "optimal combination" dynamically adjusts its preferences based on the computing platform's operational status. For example, during periods of rapid traffic increase, the computing platform prioritizes service capacity, assigning higher priority to resource utilization; while during periods of stable traffic, it focuses more on platform stability, prioritizing solutions with lower switching costs to reduce disruption. Through this adaptive scoring and ranking mechanism, the computing platform ensures service quality while minimizing unnecessary frequent switching, achieving optimal long-term operational efficiency.
[0115] It should be further explained that in the selection of engine combinations, priority should be given to the option with the smallest difference between two consecutive combinations, so as to reduce the number of engines that need to be started / stopped.
[0116] For example, in practice, neural networks are used to model historical traffic data to predict traffic for the next 10 minutes and 1 hour. This module can identify periodic patterns and trends in traffic, providing accurate traffic predictions. Based on the predicted traffic, resource utilization and switching costs for different combinations of large language models are calculated. Using a multi-objective optimization algorithm, the optimal balance between resource utilization and switching costs is found, and the optimal combination of large language models is selected. This embodiment considers historical switching data and the current computing platform status to ensure that the selected combination has optimal performance in long-term operation.
[0117] Upon receiving a model switching instruction, this embodiment analyzes the N processing requests currently being processed and awaiting transfer in the first large language model, obtains the running information corresponding to the processing requests in real time, determines the transfer priority of the N processing requests based on the running information, and then transfers the running information of the N processing requests to the second large language model according to the transfer priority. Thus, without interrupting the processing requests, the processing requests are transferred from the first large language model to the second large language model for continued processing, achieving seamless switching of large language models and improving the switching efficiency of large language models.
[0118] See Figure 7 , Figure 7 This is a structural diagram of the large language model switching device provided in an embodiment of this application. For example... Figure 7 As shown, the large language model switching device 1200 includes:
[0119] The request acquisition module 1210 is used to acquire N processing requests that are being processed and are to be transferred in the first language model when a model switching instruction is received to switch the first language model to the second language model, where N is a positive integer;
[0120] The operation information acquisition module 1220 is used to acquire N operation information corresponding to N processing requests in real time;
[0121] The priority determination module 1230 is used to obtain the transfer priority of N processing requests that are dynamically updated based on N running information.
[0122] The transfer module 1240 is used to transfer the running information corresponding to the processing request to the second language model according to the transfer priority, and the second language model continues to process the corresponding processing request based on the transferred running information.
[0123] The confirmation module 1250 is used to confirm that the model switching instruction has been executed when all N processing requests meet the conditions of completed running information transfer or completed execution by the first major language model.
[0124] The large language model switching device provided in this application analyzes the N processing requests that are being processed and are awaiting transfer in the first large language model when a model switching instruction is received. It obtains the running information corresponding to the processing requests in real time, determines the transfer priority of the N processing requests based on the running information, and then transfers the running information of the N processing requests to the second large language model according to the transfer priority. Thus, the processing requests are transferred from the first large language model to the second large language model for continued processing without interrupting the processing requests, achieving seamless switching of large language models and improving the switching efficiency of large language models.
[0125] Optionally, the priority determination module 1230 includes:
[0126] The calculation submodule is used to calculate the amount of data to be transferred based on the runtime information of the processing request, wherein the runtime information includes the intermediate processing data of the processing request;
[0127] The acquisition submodule is used to obtain the transfer cost of the corresponding processing request based on the amount of data to be transferred;
[0128] The determination submodule is used to determine the corresponding transfer priority for each processing request based on the transfer cost, wherein the transfer cost is negatively correlated with the transfer priority.
[0129] Optionally, the intermediate processing data includes at least one of the following: processed token sequences, caches in the attention mechanism, dialogue history context, partial output text, the state required by the first language model, and the hidden states of each layer.
[0130] Optionally, the transfer module 1240 includes:
[0131] The analysis submodule is used to analyze the difference information between the first and second language models. The difference information represents the structural differences between the model structure of the first language model and the model structure of the second language model.
[0132] The information processing submodule is used to perform target processing on the transferred operation information based on the difference information to obtain target operation information. The data structure of the target operation information is adapted to the second language model. The target processing includes at least one of the following: interpolation processing and truncation processing.
[0133] The request processing submodule is used by the second language model to continue processing the corresponding processing requests based on the target runtime information.
[0134] Optional, also includes:
[0135] The traffic acquisition module is used to acquire historical and current traffic data of the computing platform;
[0136] The traffic prediction module is used to predict the traffic data of the computing platform within a target time period based on historical traffic data and current traffic data, so as to obtain the target traffic data.
[0137] The generation module is used to generate model switching instructions when the currently running model combination cannot meet the target traffic data. The model switching instructions include complete switching instructions and partial switching instructions. The currently running model combination includes the first major language model.
[0138] Optionally, the generation module includes:
[0139] The model generation submodule is used to generate multiple candidate model combinations based on the target traffic data. Each candidate model combination includes at least one large language model.
[0140] The indicator calculation submodule is used to calculate the evaluation indicator for each candidate model combination. The evaluation indicator indicates the service change data of the computing platform after switching from the currently running model combination to the candidate model combination.
[0141] The weight calculation submodule is used to dynamically update the weights corresponding to the evaluation indicators based on the current status information of the computing platform.
[0142] The combination determination submodule is used to determine the target model combination from multiple candidate model combinations based on the evaluation indicators and their corresponding weights. The target model combination includes the second largest language model.
[0143] The generation submodule is used to generate model switching instructions that switch the first major language model to the second major language model.
[0144] Optionally, the indicator calculation submodule includes:
[0145] The resource calculation unit is used to calculate the resource utilization rate corresponding to each candidate model combination. The resource utilization rate indicates the service efficiency of the corresponding candidate model combination.
[0146] The cost calculation unit is used to calculate the switching cost from the currently running model combination to each candidate model combination. The switching cost indicates the resource consumption of switching from the currently running model combination to each candidate model combination.
[0147] The evaluation indicators include resource utilization and switching costs.
[0148] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 8The electronic device may include a processor 1301, a memory 1302, and a program 13021 stored in the memory 1302 and executable on the processor 1301.
[0149] When program 13021 is executed by processor 1301, it can achieve the following: Figure 1 Any step in the corresponding method embodiment:
[0150] When a model switching instruction is received to switch from the first major language model to the second major language model, the N processing requests that are being processed in the first major language model and are waiting to be transferred are obtained, where N is a positive integer;
[0151] Obtain N runtime information corresponding to N processing requests in real time;
[0152] The transfer priority of N processing requests is dynamically updated based on N pieces of runtime information;
[0153] Based on the transfer priority, the runtime information corresponding to the processing request is transferred to the second language model, which then continues to process the corresponding processing request based on the transferred runtime information.
[0154] When all N processing requests meet the condition that the information transfer is complete or has been executed by the first language model, confirm that the model switching instruction has been executed.
[0155] Optionally, the transfer priority of N processing requests is dynamically updated based on N pieces of runtime information, including:
[0156] Based on the runtime information of the processing request, calculate the amount of data to be transferred, where the runtime information includes the intermediate processing data of the processing request;
[0157] Based on the amount of data to be transferred, obtain the transfer cost for the corresponding processing request;
[0158] The transfer priority is determined based on the transfer cost of each processing request, where the transfer cost and transfer priority are negatively correlated.
[0159] Optionally, the intermediate processing data includes at least one of the following: processed token sequences, caches in the attention mechanism, dialogue history context, partial output text, the state required by the first language model, and the hidden states of each layer.
[0160] Optionally, the second language model continues to process the corresponding processing requests based on the transition runtime information, including:
[0161] The differences between the first and second language models are analyzed, and these differences represent the structural differences between the model structures of the first and second language models.
[0162] The target operation information is processed based on the difference information to obtain the target operation information. The data structure of the target operation information is adapted to the second language model. The target processing includes at least one of the following: interpolation processing and truncation processing.
[0163] The second largest language model then continues to process the corresponding processing requests based on the target runtime information.
[0164] Optionally, before receiving the model switching instruction to switch from the first to the second largest language model, the method further includes:
[0165] Obtain historical and current traffic data from the computing platform;
[0166] Based on historical and current traffic data, the traffic data of the computing platform in the target time period is predicted to obtain the target traffic data.
[0167] If the currently running model combination cannot meet the target traffic data, a model switching instruction is generated. The model switching instruction includes a complete switching instruction and a partial switching instruction. The currently running model combination includes the first major language model.
[0168] Optionally, if the currently running model combination cannot meet the target traffic data, a model switching instruction is generated, including:
[0169] Based on the target traffic data, generate multiple candidate model combinations, each of which includes at least one large language model.
[0170] Calculate the evaluation metrics for each candidate model combination. The evaluation metrics indicate the service changes of the computing platform after switching from the currently running model combination to the candidate model combination.
[0171] The weights of the evaluation indicators are dynamically updated based on the current status information of the computing platform.
[0172] Based on the evaluation indicators and their corresponding weights, a target model combination is determined from multiple candidate model combinations, including the second largest language model.
[0173] Generate a model switching instruction to switch the first largest language model to the second largest language model.
[0174] Optionally, calculate the evaluation metrics for each candidate model combination, including:
[0175] Calculate the resource utilization rate for each candidate model combination. The resource utilization rate indicates the service efficiency of the corresponding candidate model combination.
[0176] Calculate the switching cost from the currently running model combination to each candidate model combination. The switching cost indicates the resource consumption of switching from the currently running model combination to each candidate model combination.
[0177] The evaluation indicators include resource utilization and switching costs.
[0178] Upon receiving a model switching instruction, this embodiment analyzes the N processing requests currently being processed and awaiting transfer in the first large language model, obtains the running information corresponding to the processing requests in real time, determines the transfer priority of the N processing requests based on the running information, and then transfers the running information of the N processing requests to the second large language model according to the transfer priority. Thus, without interrupting the processing requests, the processing requests are transferred from the first large language model to the second large language model for continued processing, achieving seamless switching of large language models and improving the switching efficiency of large language models.
[0179] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described large language model switching method embodiments and achieves the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium provided in this application can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0180] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described large language model switching method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.
[0183] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for switching large language models, characterized in that, Applied to a computing platform, wherein the computing platform includes at least a first large language model and a second large language model, the method includes: When a model switching instruction is received to switch the first large language model to the second large language model, N processing requests that are being processed and are to be transferred in the first large language model are obtained, where N is a positive integer; Real-time acquisition of N operational information corresponding to the N processing requests; The transfer priority of the N processing requests is dynamically updated based on the N operational information. According to the transfer priority, the running information corresponding to the processing request is transferred to the second major language model, and the second major language model continues to process the corresponding processing request based on the transferred running information; When all N processing requests meet the condition that the running information transfer is complete or has been executed by the first large language model, the model switching instruction is confirmed to have been executed.
2. The method according to claim 1, characterized in that, The transfer priority of the N processing requests, dynamically updated based on the N operational information, includes: Based on the running information of the processing request, the amount of data to be transferred is calculated, wherein the running information includes the intermediate processing data of the processing request; Based on the amount of data to be transferred, obtain the transfer cost corresponding to the processing request; Based on the transfer cost of each processing request, its corresponding transfer priority is determined, wherein the transfer cost is negatively correlated with the transfer priority.
3. The method according to claim 2, characterized in that, The intermediate processing data includes at least one of the following: processed token sequence, cache in the attention mechanism, dialogue history context, partial output text, the required state of the first large language model, and the hidden state of each layer.
4. The method according to claim 1, characterized in that, The process of the second language model continuing to process the corresponding processing request based on the transferred runtime information includes: Analyze the difference information between the first large language model and the second large language model, where the difference information represents the structural differences between the model structures of the first large language model and the second large language model; The target operation information of the transfer is processed according to the difference information to obtain target operation information. The data structure of the target operation information is adapted to the second language model. The target processing includes at least one of the following: interpolation processing and truncation processing. The second major language model then continues to process the corresponding processing requests based on the target runtime information.
5. The method according to claim 1, characterized in that, Before receiving the model switching instruction to switch the first large language model to the second large language model, the method further includes: Obtain historical and current traffic data of the computing platform; Based on the historical traffic data and the current traffic data, the traffic data of the computing platform in the target time period is predicted to obtain the target traffic data; If the currently running model combination cannot meet the target traffic data, a model switching instruction is generated, wherein the model switching instruction includes a complete switching instruction and a partial switching instruction; the currently running model combination includes the first major language model.
6. The method according to claim 5, characterized in that, When the currently running model combination cannot meet the target traffic data, the generation of the model switching instruction includes: Based on the target traffic data, multiple candidate model combinations are generated, and each candidate model combination includes at least one large language model. Calculate an evaluation metric for each candidate model combination, the evaluation metric indicating the service change data of the computing platform after switching from the currently running model combination to the candidate model combination; The weights corresponding to the evaluation indicators are dynamically updated based on the current status information of the computing platform. Based on the evaluation indicators and their corresponding weights, a target model combination is determined from the plurality of candidate model combinations, wherein the target model combination includes the second major language model; Generate a model switching instruction to switch the first major language model to the second major language model.
7. The method according to claim 6, characterized in that, The calculation of the evaluation metric for each candidate model combination includes: Calculate the resource utilization rate for each candidate model combination, where the resource utilization rate indicates the service efficiency of the corresponding candidate model combination. Calculate the switching cost from the currently running model combination to each of the candidate model combinations, wherein the switching cost indicates the resource consumption of switching from the currently running model combination to each of the candidate model combinations; The evaluation metrics include the resource utilization rate and the switching cost.
8. A large language model switching device, characterized in that, Applied to a computing platform, the computing platform including at least a first large language model and a second large language model, the device includes: The request acquisition module is used to acquire N processing requests that are being processed and are to be transferred in the first large language model when a model switching instruction is received to switch the first large language model to the second large language model, where N is a positive integer; The operation information acquisition module is used to acquire N operation information corresponding to the N processing requests in real time; The priority determination module is used to obtain the dynamically updated transfer priority of the N processing requests based on the N running information. The transfer module is used to transfer the running information corresponding to the processing request to the second large language model according to the transfer priority, and the second large language model continues to process the corresponding processing request based on the transferred running information. The confirmation module is used to confirm that the model switching instruction has been executed when all N processing requests meet the state that the running information transfer is complete or has been executed by the first large language model.
9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps of the large language model switching method as described in any one of claims 1 to 7.
10. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the large language model switching method as described in any one of claims 1 to 7.