Online Caching and Service System for Large-Scale Edge Device Models Based on Multi-Armed Bandit

Abstract

This invention relates to the field of edge computing and large model inference optimization technology, and provides an online caching and service system for large models on edge devices based on Multi-Armed Bandit. The system includes: a feature information receiving module for receiving a set of feature information of candidate LLMs in the cloud; an LLM quality prediction module for predicting the utility vector of each LLM and calculating the UCB value and utility offset based on the feature information set using a neural network; an edge sorting and selection module for sorting the sum of UCB values ​​and utility offsets in descending order, selecting a preset number of LLMs to generate an LLM subset, caching the LLM subset, and deploying it to the edge; and an asynchronous processing output training module for collecting service quality data and neural network gradient data, and asynchronously training and optimizing the parameters of the neural network model in the LLM quality prediction module; the above steps are repeated continuously to achieve online caching and service of large models on edge devices based on Multi-Armed Bandit.

Description

Technical Field

[0001] This invention relates to the field of edge computing and large model inference optimization technology, specifically to an online caching and service system for large models of edge devices based on Multi-ArmedBandit. Background Technology

[0002] With the rapid development of edge computing technology, the demand for intelligent devices is increasing, especially in large-scale language model (LLM) inference applications such as intelligent voice assistants, translation services, and text summarization generation. To meet the low latency and high computational efficiency requirements of these services, edge devices are typically responsible for tasks close to the user. Because traditional full-model caching methods cannot meet resource constraints, researchers have proposed various model compression techniques, including quantization, knowledge distillation, and pruning, to adapt to the storage and computational limitations of edge devices. These techniques aim to reduce the model's memory footprint and computational requirements while maintaining performance. However, edge devices have limited computational resources and storage space, and different models correspond to different service qualities, so it is necessary to select appropriate model caching to optimize service quality.

[0003] (1) In the prior art, many joint optimization methods combine multiple factors such as caching, computation offloading, and bandwidth allocation for optimization. These methods usually rely on static or semi-static models and aim to improve the overall performance of the system, such as reducing latency, reducing energy consumption, or increasing cache hit rate. However, joint optimization methods usually involve multiple variables and constraints, resulting in high problem complexity and difficulty in real-time solution. Moreover, since these methods are usually based on predefined models and parameters, they are difficult to adapt to dynamically changing user requests and environments.

[0004] (2) In recent years, the introduction of deep learning and reinforcement learning technologies, especially in the fields of task scheduling and resource optimization, has significantly improved the intelligence level of systems. For example, scheduling strategies based on deep Q-networks (DQN) can adaptively learn the processing methods of different tasks and gradually optimize task offloading decisions. However, these methods often assume that the service quality requirements provided by the model are fixed or predictable. For the processing requirements of large-scale language model (LLM) requests, the service quality is often unpredictable, mainly due to the following factors: First, the performance of LLMs in processing tasks is affected by the complexity of the task. Different models have different effects in tasks such as generating long texts, answering domain-specific questions, or open-ended dialogues. Second, the structure and number of parameters of the models are different. Larger models usually provide better performance, but require more computing resources and time. In addition, the uncontrollability of the request context causes the performance of the same model to fluctuate in different environments. Especially when multiple tasks are concurrent, the resource status and network conditions of edge devices also affect inference efficiency. Furthermore, some LLMs may perform well in specific domain knowledge but poorly in other domains. These factors together make it difficult to predict the service quality of different LLMs.

[0005] (3) Some other methods treat different models as homogeneous to simplify the complexity of the optimization problem. These methods usually rely on a uniform model caching strategy, aiming to improve the overall performance of the system through fixed parameters and rules, such as reducing latency and increasing cache hit rate. However, this approach fails to fully consider the unique characteristics of the Large Language Model (LLM) architecture, resulting in the inability to effectively utilize the advantages of each model during the optimization process. For example, different LLMs have significant differences in task processing capabilities, response speed, and resource requirements, and the uniform approach ignores these differences, making it difficult to achieve the expected performance in practical applications. Summary of the Invention

[0006] This invention proposes an online caching method for large language models (LLMs) on edge devices based on Multi-Armed Bandit (MAB). By learning in real time and selecting a preset number of suitable LLMs for caching, it optimizes resource utilization and inference performance of edge devices.

[0007] The feature information receiving module is used to receive a set of feature information of candidate LLMs in the cloud of the edge server device. The set of feature information of candidate LLMs in the cloud includes the number of model parameters, computational requirements, memory requirements, type of task used, inference latency baseline and multi-task support.

[0008] The LLM quality prediction module is used to predict the utility vector of each LLM using a neural network based on the set of feature information, and to calculate the UCB value and utility offset corresponding to each LLM utility vector.

[0009] The edge sorting selection module is used to sort the sum of the UCB values ​​and utility offsets in descending order and select a preset number of top values. Each LLM generates an LLM subset, defines the LLM subset as a superarm, and caches the superarm before deploying it to the edge to provide services.

[0010] The asynchronous processing output training module is used to collect service quality data and neural network gradient data, and asynchronously train and optimize the neural network model in the LLM quality prediction module.

[0011] The above modules are run sequentially in a loop to achieve online caching and service of large edge device models in Multi-ArmedBandit.

[0012] Optionally, in the LLM quality prediction module, the operation process of the neural network prediction specifically includes:

[0013] Approximate the unknown function in the utility round model using a neural network function with a fully connected structure:

[0014]

[0015] in, It is the ReLU activation function. m The width of the hidden layer in the neural network. Feature vectors for candidate LLMs in the cloud. This is the parameter matrix of the neural network model. This is the weight matrix of the first layer of the neural network. This is the weight matrix for the last layer. This is the weight matrix for the penultimate layer.

[0016] Optionally, the utility round model is:

[0017]

[0018] in, For the first j The first arm t The utility of each round, For unknown functions, For a v-th subgaussian noise, For the first j The first arm t Feature vectors from each round.

[0019] Optionally, the LLM quality prediction module includes a utility vector prediction submodule and a utility offset calculation submodule;

[0020] The utility vector prediction submodule is used to calculate the UCB value of the utility vector based on the parameters and covariance matrix of the neural network model.

[0021]

[0022] in, For the first j The first arm t The utility of each round The predicted value, Scaling factor For unknown functions The gradient;

[0023] The utility offset calculation submodule is used to calculate the utility offset based on the parameters and covariance matrix of the neural network model:

[0024]

[0025] in, These are the weighting coefficients. K For the capacity of edge servers, L To handle the delay, This is the regularization parameter.

[0026] Optionally, in the asynchronous processing output training module, the service quality data is the service quality obtained by the superarm based on the accuracy A of completing the task, the processing latency L, and the computational cost C. :

[0027]

[0028] in, , These are the weight parameters.

[0029] Optionally, in the asynchronous processing output training module, service quality data and neural network gradient data are collected, and the neural network model in the LLM quality prediction module is trained and its parameters optimized asynchronously, specifically including:

[0030] Examine the training signal of the neural network model trained in this round, copy the neural network model in the LLM quality prediction module, and train and optimize the copied neural network based on the collected service quality data;

[0031] Based on the trained and optimized neural network model, the gradient descent method is used to update the parameters and covariance matrix of the neural network model;

[0032] The optimized neural network model parameters are copied to the neural network model in the LLM quality prediction module.

[0033] Optionally, updating the covariance matrix of the neural network model using the gradient descent method specifically includes:

[0034]

[0035] in, It is a function gradient, It is the accumulated feedback set during the period when the most recent training signal was false. When the training signal is false, the parameters from the previous round remain unchanged. and .

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1) Dynamically Optimized Caching Strategy: Based on the context-based Multi-Armed Bandit model, this invention can dynamically select and cache a preset number of large language models (LLMs) best suited for the current task in each round of decision-making. This real-time learning mechanism allows edge devices to flexibly adjust caching strategies according to actual request demands and system resource conditions, significantly improving system response speed and task processing efficiency, and overcoming the limitations of static caching strategies in existing technologies that are difficult to adapt to changes.

[0038] 2) Nonlinear Performance Optimization: By defining the context feature vector of LLM and establishing a nonlinear mapping between performance and context, this invention can more accurately predict the model's performance in different environments, thereby achieving comprehensive performance optimization. This nonlinear relationship allows the system to better balance latency, accuracy, and computational cost when handling tasks of varying complexity, improving the overall throughput of the system.

[0039] 3) Efficient Feedback Mechanism: By providing real-time feedback on task execution performance (including latency, accuracy, resource consumption, etc.), the system can continuously learn and improve its decision-making process. The Contextual Neural Network (UCB) algorithm used in this invention can progressively optimize caching strategies based on this service quality data, ensuring that the system can continuously improve performance in a dynamic environment and achieve near-optimal performance in long-term operation. It has been theoretically proven that this system exhibits stable performance in long-term operation, and its error growth rate gradually slows down, eventually approaching zero.

[0040] 4) Improved resource utilization efficiency: By effectively selecting and caching LLMs, this invention significantly reduces the storage and computing burden on edge devices. Compared with existing full-model caching or simple task offloading solutions, this invention can better utilize the limited resources of edge devices, especially in resource-constrained environments, resulting in a significant improvement in system performance and resource utilization. Attached Figure Description

[0041] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the structure of the online caching and service system for large edge device models based on Multi-ArmedBandit, according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of a fully connected neural network according to an embodiment of the present invention;

[0044] Figure 3 This is a flowchart illustrating the working method of the online caching and service system for large edge device models based on Multi-ArmedBandit, as described in an embodiment of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] First, the technical content used in this invention will be explained and elaborated:

[0047] In this invention, an LLM represents an arm and corresponds to a feature vector. This feature vector is encoded by the feature information of the corresponding LLM. That is, through encoding, an LLM corresponds to a set of feature information. All candidate LLMs each have a set of feature information. Multiple LLMs result in multiple sets of feature information, and a feature information set is formed based on these multiple sets of feature information.

[0048] A set of feature information corresponding to an LLM is defined as the "context" information of this LLM, and a Multi-Armed Bandit containing this setting is called a "context Multi-Armed Bandit".

[0049] This can be understood as follows: There are 6 features used to characterize an LLM, namely, the number of model parameters, computational requirements, memory requirements, task type, inference latency baseline, and multi-task support. Different LLMs have different values ​​for these 6 features. According to a predetermined encoding scheme, the feature information of each LLM is encoded into a vector. Our algorithm selects multiple arms in each round. These selected arms form a set, which is called a "superarm". This kind of Multi-ArmedBandit is called "Combined Multi-ArmedBandit".

[0050] Number of parameters: The total number of parameters in a model, usually used to measure the size of the model. The larger the number of parameters, the more complex the model can usually handle, but it will also consume more resources.

[0051] Example: An LLM with 100 million parameters is generally faster than an LLM with 1 billion parameters, but is limited in complexity.

[0052] Computational requirements: The computational resources required by the model during inference, usually expressed in FLOPs (floating-point operations) or other units that measure computational load.

[0053] Example: A model with a computational requirement of 200 GFLOPs will save more resources at runtime than a model with a computational requirement of 500 GFLOPs.

[0054] Memory requirements: The memory space occupied by the model during inference, including the memory required to load the model and process data.

[0055] Example: A model with a memory requirement of 4GB is suitable for devices with limited resources, while a model with a memory requirement of 16GB requires stronger hardware support.

[0056] Applicable task types: The types of tasks the model can handle, such as text generation, classification, and translation. Different models may be optimized for different tasks.

[0057] Example: Some LLMs are particularly good at text classification, while others perform better in text generation.

[0058] Inference latency baseline: The average latency of a model processing a single inference task on standard hardware. Lower latency results in faster response times.

[0059] Example: One model has an inference latency of 100 milliseconds, while another model has an inference latency of 500 milliseconds. The former is more suitable for application scenarios that require fast response.

[0060] Multi-task support: Does the model have the ability to process multiple tasks in parallel, thereby improving the system's throughput and processing efficiency?

[0061] Example: An LLM that supports multitasking can handle multiple user requests simultaneously, while an LLM that does not support multitasking can only handle requests one by one.

[0062] Example Explanation: Assuming a real-world application scenario, this embodiment has the following two alternative LLMs:

[0063] Model A:

[0064] Number of parameters: 100 million;

[0065] Computational requirements: 200 GFLOPs;

[0066] Memory requirement: 4GB;

[0067] Applicable task types: text classification, sentiment analysis;

[0068] Inference latency baseline: 100 milliseconds;

[0069] Multitasking support: Supported;

[0070] Model B:

[0071] Number of parameters: 1 billion;

[0072] Computational requirements: 500 GFLOPs;

[0073] Memory requirement: 16GB;

[0074] Applicable task types: text generation, question-answering systems;

[0075] Inference latency baseline: 500 milliseconds;

[0076] Multitasking support: Not supported.

[0077] In order to encode these feature information into numerical feature vectors and input them into the neural network, this embodiment can convert each feature into an appropriate numerical or vector format and combine them into a complete feature vector according to the following scheme.

[0078] Number of parameters: The number of parameters is a numerical value that measures the size of the model. It is usually taken as a logarithm to reduce the impact of numerical differences.

[0079] Encoding scheme: Take the logarithm of the number of parameters. For example, assuming the model has 1 billion parameters, the value of this feature could be log(10^9) = 9.

[0080] Computational requirements: Computational requirements are usually expressed in FLOPs, or the logarithm can be used to narrow down the numerical range.

[0081] Encoding scheme: Take the logarithm of the computational requirement (e.g., FLOPs). For example, if the computational requirement is 500 GFLOPs, the value of this feature could be log(500 GFLOPs). 10^9).

[0082] Memory requirements: Memory requirements are expressed in GB and are directly numerically represented.

[0083] Encoding scheme: Directly use the numerical value of memory requirement. For example, a model with a memory requirement of 16GB can be directly encoded as 16.

[0084] Applicable task types: Applicable task types are usually discrete categories, such as text classification, generation, and translation. This can be represented using one-hot encoding.

[0085] Encoding scheme: Define a set of applicable task types, such as: [text classification, sentiment analysis, text generation, translation, question answering]. Each task type corresponds to a position, with suitable task types set to 1 and unsuitable task types set to 0.

[0086] Example: If a model is suitable for text generation and question answering systems, its encoding is [0,0,1,0,1].

[0087] Inference latency baseline: Inference latency represents response speed, usually expressed in milliseconds (ms). To reduce the impact of numerical range, inference latency can be normalized or its reciprocal (i.e., 1 / latency) can be taken to make it proportional to the response speed.

[0088] Encoding scheme: 1 / inference delay or the logarithm of the delay can be used. Assuming the inference delay is 500 milliseconds, the encoding can be 1 / 500 = 0.002.

[0089] Multitasking support: Multitasking support is a boolean attribute that can be processed using binary encoding.

[0090] Encoding scheme: If the model supports multitasking, encode it as 1; otherwise, encode it as 0.

[0091] Example of a comprehensive feature vector: Assume the model has the following features:

[0092] Number of parameters: 1 billion;

[0093] Computational requirements: 500 GFLOPs;

[0094] Memory requirement: 16GB;

[0095] Applicable task types: text generation, question-answering systems;

[0096] Inference latency baseline: 500 milliseconds;

[0097] Multitasking support: Not supported.

[0098] Based on the above encoding scheme, the feature vector of this model can be represented as:

[0099] Number of parameters: log(10^9) = 9;

[0100] Calculation requirement: log(500) 10^9)≈11.7;

[0101] Memory requirement: 16;

[0102] Applicable task type (one-hot encoding): [0,0,1,0,1];

[0103] Inference delay baseline: 1 / 500 = 0.002;

[0104] Multitasking support: 0.

[0105] The final combined feature vector is:

[0106] [9,11.7,16,0,0,1,0,1,0.002,0].

[0107] Example

[0108] An online caching and service system for large-scale edge devices based on Multi-ArmedBandit, such as Figure 1 As shown, the system includes:

[0109] The feature information receiving module is used to receive the feature information set of candidate LLMs in the cloud of the edge server device. The feature information set of candidate LLMs in the cloud includes the number of model parameters, computational requirements, memory requirements, type of task used, inference latency baseline, and multi-task support.

[0110] Initialize neural network model parameters These parameters will be used in subsequent utility predictions and model training. Simultaneously, the covariance matrix is ​​set. ,in It is a regularization parameter. It is an identity matrix.

[0111] In the feature information receiving module, the cloud-based alternative LLM specifically refers to defining a round as the time window during which the edge device server processes a fixed number of requests. t In the middle, select according to the hardware configuration of the edge device server. Each LLM is cached; at this time, the number of candidate LLMs in the cloud is recorded as follows: .

[0112] Specifically, consider a system that treats each LLM as an arm in a context-combined Multi-Armed Bandit framework. All candidate LLMs are stored in the cloud. Each LLM is embedded into a vector via a feature encoder. As a neural network function The input parameters are To predict the utility of LLM. Feature vectors It can consist of a series of features, such as a vector of feature variables including model parameter count, computational requirements, memory requirements, applicable task types, inference latency baseline, and multi-task support. In this embodiment, the time window for the edge server to process a fixed number of requests is defined as one round. t ,use Indexing. In rounds In the middle, depending on the hardware configuration, it can be selected. Each LLM is cached, and the number of alternative LLMs in the cloud is denoted as . Assuming For any .

[0113] The LLM quality prediction module is used to predict the utility vector of each LLM using a neural network based on the set of feature information, and to calculate the UCB value and utility offset corresponding to each LLM utility vector.

[0114] In each round (from 1 to In the cloud, the feature vector is composed of a set of feature information (type parameters, computational requirements, memory requirements, applicable task types, inference latency baseline, and multi-task support) of the candidate LLMs. LLM Corresponding feature vector This information represents the characteristics of a large language model and affects the quality of the services it subsequently provides.

[0115] The service quality of each LLM is predicted using a neural network by calculating the UCB value of the utility vector. and utility offset This step involves utilizing the parameters of the neural network model from the previous round. Covariance Matrix This allows for predictions, providing a basis for optimizing choices in cached LLMs.

[0116] Calculate the cloud-based [function] in round t. UCB value of an LLM and utility offset The pseudocode is shown in Table 1:

[0117] Table 1

[0118]

[0119] In the LLM quality prediction module, the operation of the neural network specifically includes:

[0120] Due to the lack of historical data, this embodiment uses the service quality predicted by a neural network as a substitute for the actual service quality. Once service is provided and feedback on the actual service quality is obtained, it is used to train the neural network model and update its parameters. In the method proposed in this embodiment, candidate LLM feature information vectors are used as their corresponding context vectors, which are then input into the neural network to predict utility. Figure 2 As shown, the unknown function in the utility vector model is approximated using a neural network function with a fully connected structure:

[0121]

[0122] in, It is the ReLU activation function. It is a weight matrix. m Let be the width of the hidden layer. The parameter matrix can be represented as... ,in ,and Vectorize the matrix into a flat vector. The gradient of this function is denoted as... Given confidence parameters Define the scaling factor as:

[0123]

[0124]

[0125] These are some positive constants. The network parameters are initialized from a Gaussian distribution. Specifically, for ,set up Each of them Independently generated from .

[0126] The utility vector model is:

[0127]

[0128] in, For unknown functions, For a v-th subgaussian noise, For the first j The first arm t The utility of each round, For the first j The first arm t Feature vectors from each round.

[0129] The cloud-based alternative LLM programs are first selected based on scores. Sort in descending order, then select the top Each LLM forms a subset. As a set of LLMs to be cached at the edge.

[0130] The LLM quality prediction module includes a utility vector prediction submodule and a utility offset calculation submodule;

[0131] The utility vector prediction submodule is used to calculate the utility vector based on the parameters and covariance matrix of the neural network model.

[0132]

[0133] in, For the first j The first arm t The utility of each round The predicted value, Scaling factor For unknown functions The gradient.

[0134] The utility offset calculation submodule is used to calculate the utility offset based on the parameters and covariance matrix of the neural network model:

[0135] .

[0136] The edge sorting and selection module is used to sort the utility vector and the sum of the UCB values ​​in descending order, select the first number of LLMs to generate an LLM subset, define the LLM subset as a superarm, cache the superarm and deploy it to the edge to provide services.

[0137] Based on the calculated score Sort the candidate LLMs in descending order to ensure that the one cached at the edge is the most advantageous. Select the top-ranked LLMs after sorting. Each LLM forms a subset of LLMs. This subset will be used for caching on edge servers.

[0138] The asynchronous processing output training module is used for caching and deploying the superarm and for asynchronously training and optimizing the neural network model in the LLM quality prediction module. In the asynchronous processing output training module, the service quality data is the service quality obtained by the superarm based on the accuracy A of task completion, processing latency L, and computational cost C. :

[0139]

[0140] in, , These are the weight parameters.

[0141] In the asynchronous processing output training module, service quality data is collected to train and optimize the neural network model. The trained and optimized parameters are then used to update the neural network model in the LLM quality prediction module. Specifically, this includes:

[0142] Examine the training signal of the neural network model in this training round, and train and optimize the neural network model based on the feedback data during the training process;

[0143] When a new training cycle of the neural network model begins, the training signal changes from "true" to "false," indicating that training is in progress. When a training cycle ends, the training signal changes from "false" to "true," indicating that a new training cycle of the neural network model can begin. To maintain real-time processing efficiency, this embodiment uses an asynchronous method to train the neural network model in the LLM quality prediction module. The neural network model parameters in the LLM quality prediction module are not updated before the current training process of the neural network model ends; instead, they are updated after training is completed. For example, if the number of rounds... During the training phase of a neural network model, the number of rounds... Feedback generated during this period may be used immediately. Instead, it is accumulated and incorporated into the dataset for the next round of model training at the end of the neural network model training. This results in delayed feedback, where the reward of one round may affect the model training of subsequent rounds. This strategy ensures that edge inference remains timely and uninterrupted while efficiently handling model updates. Specifically, for any Real utility After The model observed the results after each round. In round... The agent received the feedback set ,in .make Indicates the round The set of all feedback indices received so far is represented as .

[0144] The optimization objective is to maximize the overall quality of the edge servers, while the utility value is initially unknown, achieved by minimizing the cumulative expected regret. In order to achieve this goal. The potential reward loss caused by suboptimal decisions is quantified and formally defined as follows:

[0145] ,

[0146] in It is a utility vector. It is a round The best offline superarm.

[0147] Based on the trained and optimized neural network model, the parameters and covariance matrix of the neural network model are updated using the gradient descent method; the utility vector and utility offset are calculated based on the updated parameters and covariance matrix.

[0148] Updating the covariance matrix of a neural network model using gradient descent specifically includes:

[0149]

[0150] in, It is a function gradient, It is the accumulated set of feedback during the period when the most recent training signal was false.

[0151] Selected LLM subset at the edge end Implement cache deployment and record the service quality of each LLM during processing as feedback on the selection of this model. This data is used for subsequent neural network model training to help improve the accuracy of model predictions.

[0152] The neural network model is optimized using the quality of service (QS) data collected in each round. The QS data consists of the accuracy (A), processing latency (L), and computational cost (C) of the selected LLM in completing the task, from which the quality of service (Q) is calculated.

[0153] ,

[0154] By using service quality data as the prediction target of the neural network model, the system can progressively train the model parameters, improve its predictive ability, thereby increasing the accuracy of subsequent LLM utility calculations and ultimately optimizing the performance and service quality of edge devices.

[0155] Since the goal of this embodiment is to improve the overall LLM service quality of the edge server, LLMs with higher actual service quality are more suitable for caching on the edge server. Therefore, rounds Midarm The utility Defined as the quality of service provided by the LLM ,Right now The quality of service is defined as:

[0156] .

[0157] in, Accuracy represents the relevance and accuracy of the model's generated results; To handle latency, this represents the time from receiving a request to generating a response; For computational cost, it represents the computational resources (such as CPU time, memory usage, etc.) consumed when processing a request. and The weights represent the importance of latency and computational cost in service quality assessment. These characteristics have a complex, non-linear relationship with service quality. This design allows service quality assessment to reflect the multidimensional characteristics of model performance, providing more accurate and dynamic performance metrics. Service quality cannot be predicted in advance due to the diversity of task complexity, model performance variability, and dynamic contextual effects. Specifically, different user requests involve different task types and complexities, and different LLMs will perform differently under the same task. Furthermore, the context and environmental changes of requests (such as network status and device load) are often unpredictable, making it difficult to accurately assess service quality. Feedback latency further limits the model's predictive ability in the initial stages, and the relationship between service quality and influencing factors is usually non-linear. These factors affect the true utility. The information is unpredictable and can only be observed after the service is provided (accuracy, processing latency, and computational cost), serving as feedback for the arm. Utility is modeled as:

[0158]

[0159] in It is an unknown function. It is - Gaussian noise, conditional on history ,and For the selected superarm ,award Defined as The total utility of the middle arm represents the overall quality of service provided by this LLM caching combination on the edge server.

[0160] ,

[0161] At the end of each round, the training signal is checked. If the training signal is "true," the neural network model training phase begins immediately. The training signal is then changed to "false" and maintained until the end of training before being changed back to "true." Simultaneously, all feedback data collected up to this point is used to train the neural network, employing gradient descent to obtain the updated parameters. And calculate the covariance matrix. :

[0162] ,

[0163] in, It is a function gradient, This represents the accumulated set of feedback during the period when the most recent training signal value was false. If the training signal is false, the parameters from the previous round are kept unchanged. and .

[0164] The above steps are repeated for each round. During this process, the system dynamically adjusts based on the feature information and service feedback of each LLM. (Asynchronous processing) This ensures that the LLM cache scheduling process and the neural network model training process are performed asynchronously to avoid the model training process affecting the real-time cache scheduling performance and to ensure the efficient and timely response of the edge server. The pseudocode for the neural network training method is shown in Table 2:

[0165] Table 2

[0166]

[0167] like Figure 3 As shown, this invention addresses the resource constraints, unpredictable service quality, and inflexible caching strategies faced by edge devices during large model inference in existing technologies. It proposes an online caching and service system for large models on edge devices based on Multi-ArmedBandit (MAB). This method dynamically selects and caches small, large language models suitable for the current task, adapting to changing user requests in real time to optimize overall service quality. Simultaneously, by incorporating contextual features, it intelligently manages resources through the Multi-ArmedBandit model, maximizing edge device performance. Furthermore, this invention utilizes feedback learning mechanisms and nonlinear modeling methods to accurately evaluate the performance of different miniLLMs in specific contexts, ensuring that the selected model achieves optimal results in practical applications. For dynamically changing network conditions and user needs, the dynamic caching method of this invention can continuously update and optimize caching decisions based on real-time service quality data, enhancing the system's adaptability and flexibility, and improving service quality.

[0168] Specifically, the pseudocode of the algorithm for the Multi-ArmedBandit (MAB) edge device large-scale online caching and service system is shown in Table 3:

[0169] Table 3

[0170]

[0171] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An online caching and service system for large-scale edge device models based on Multi-Armed Bandit, characterized in that, The system includes: The feature information receiving module is used to receive a set of feature information of candidate LLMs in the cloud of the edge server device. The set of feature information of candidate LLMs in the cloud includes the number of model parameters, computational requirements, memory requirements, type of task used, inference latency baseline and multi-task support. The LLM quality prediction module is used to predict the utility vector of each LLM using a neural network based on the set of feature information, and to calculate the UCB value and utility offset corresponding to each LLM utility vector. The edge sorting selection module is used to sort the sum of the UCB value and the utility offset in descending order and select the top edge sorting module. Each LLM generates an LLM subset, defines the LLM subset as a superarm, and caches the superarm before deploying it to the edge to provide services. The asynchronous processing output training module is used to collect service quality data and neural network gradient data, and asynchronously train and optimize the neural network model in the LLM quality prediction module. The above modules are run sequentially in a loop to achieve online caching and service of large edge device models using Multi-Armed Bandit. In the LLM quality prediction module, the operation process of the neural network prediction specifically includes: Approximate the unknown function in the utility round model using a neural network function with a fully connected structure: in, It is the ReLU activation function. m The width of the hidden layer in the neural network. Feature vectors for candidate LLMs in the cloud. This is the parameter matrix of the neural network model. This is the weight matrix of the first layer of the neural network. This is the weight matrix for the last layer. This is the weight matrix for the penultimate layer; The utility cycle model is as follows: in, For the first j The first arm t The utility of each round, For unknown functions, For a v-th subgaussian noise, For the first j The first arm t Feature vectors for each round; The LLM quality prediction module includes a utility vector prediction submodule and a utility offset calculation submodule; The utility vector prediction submodule is used to calculate the UCB value of the utility vector based on the parameters and covariance matrix of the neural network model. : in, For the first j The first arm t The utility of each round The predicted value, Scaling factor For unknown functions The gradient; The utility offset calculation submodule is used to calculate the utility offset based on the parameters and covariance matrix of the neural network model: in, These are the weighting coefficients. K For the capacity of edge servers, L To handle delays, This is the regularization parameter.

2. The online caching and service system for large edge device models based on Multi-Armed Bandit according to claim 1, characterized in that, In the asynchronous processing output training module, the service quality data is the service quality obtained by the superarm based on the accuracy A of completing the task, the processing latency L, and the computational cost C. : in, , These are the weight parameters.

3. The online caching and service system for large edge device models based on Multi-Armed Bandit according to claim 2, characterized in that, The asynchronous processing output training module collects service quality data and neural network gradient data, and asynchronously trains and optimizes the neural network model in the LLM quality prediction module, specifically including: Examine the training signal of the neural network model trained in this round, copy the neural network model in the LLM quality prediction module, and train and optimize the copied neural network based on the collected service quality data; Based on the trained and optimized neural network model, the gradient descent method is used to update the parameters and covariance matrix of the neural network model; The optimized neural network model parameters are copied to the neural network model in the LLM quality prediction module.

Images