Light-weight large-model intelligent customer service deployment method for edge calculation

Abstract

The invention discloses an edge computing-oriented lightweight large-model intelligent customer service deployment method, and relates to the technical field of artificial intelligence and edge computing. The method comprises the following steps: acquiring a pre-trained large language model, and pruning based on a lottery hypothesis to obtain a winner ticket sub-network; remapping the sparse structure into a V: N: M structured sparse format, and loading the V: N: M structured sparse format together with the weight and the mask to an on-chip memory; inserting a lateral branch outlet in the converter layer and carrying out token pruning; monitoring the confusion degree and the confidence degree, and dynamically recovering the pruned weight which does not exceed 5% when the load is high; and uploading the 8-bit quantization hidden state of the low-confidence or sensitive token to the large cloud model for collaborative reasoning, and combining the result with local reasoning output. According to the method, the reasoning delay and the communication overhead of the edge equipment are reduced, and the response efficiency and the deployment economy of the intelligent customer service system in a low-computing-power environment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence model compression and edge computing deployment, and particularly relates to a lightweight large model intelligent customer service deployment method for edge computing. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent customer service systems based on large language models (LLMs) have gradually become an important tool for enterprise digital transformation. Typical large language models, such as GPT, LLaMA, ChatGLM, etc., through pre-training with massive corpora, possess excellent semantic understanding and content generation capabilities, showing great potential in application scenarios such as customer consultation, product recommendation, and fault troubleshooting in the field of intelligent customer service. However, the parameter scale of large language models is often extremely large. For example, the parameter scale of GPT-3 exceeds 175 billion, and the LLaMA model also has more than 7 billion to 13 billion parameters, resulting in very high requirements for computing resources and storage resources in actual deployment.

[0003] At present, in order to deploy large language models in actual applications, the industry mainly adopts cloud deployment, that is, deploying the model in a data center and providing intelligent customer service inference services to edge devices through the network. The advantage of this solution is that the centralized computing power is strong, which can effectively solve the problems of inference accuracy and response quality. However, there are obvious defects in actual deployment: First, the cloud deployment solution strongly depends on the network communication environment, and the communication delay between the edge device and the cloud directly affects the customer service response speed, and the user experience is likely to decline significantly in the case of network fluctuations or high concurrency scenarios; Second, cloud deployment requires long-term high data center operation costs, network bandwidth costs, and energy consumption costs, which is particularly prominent when the actual business scale expands; Third, there are security risks in the transmission process of sensitive information and user privacy data, especially in fields such as finance and healthcare where strict data security requirements are imposed.

[0004] To alleviate the above problems, edge computing has emerged rapidly in recent years. The industry attempts to sink some or all of the inference capabilities of large language models to edge devices close to the user side, such as enterprise gateway devices, customer terminals, or Internet of Things terminals, to reduce the excessive dependence on cloud computing power. This solution can effectively alleviate communication delays and data security risks, but the computing power, memory, and energy consumption of edge devices are generally limited and it is difficult to accommodate a complete large-scale language model. To achieve efficient inference on the edge side, lightweight model deployment has become the key, and the main technical routes include model pruning, low-bit quantization, knowledge distillation, etc.

[0005] Specifically, model pruning techniques reduce the computational complexity and storage footprint of a model by identifying and removing redundant or less contributing weights and neurons, thereby reducing deployment costs. Although traditional unstructured pruning can achieve a high sparsity level, it is difficult to directly utilize the hardware acceleration capabilities of existing chips due to the lack of structural constraints, and thus cannot significantly improve inference performance. Structured pruning, while suitable for hardware acceleration, often comes with a significant loss of accuracy and is usually fixed in a specific sparse pattern (such as the 2:4 pattern), lacking flexibility and being difficult to meet higher compression ratio requirements.

[0006] Low-bit quantization techniques, on the other hand, significantly reduce model computation and memory usage by reducing the number of bits used to represent weights and activation data (such as INT8, INT4, or even INT2). This method has been widely applied in industrial practice. However, low-bitwidth quantization may lead to a loss of model accuracy and usually requires complex calibration or quantization-aware training (QAT) processes. Additionally, extreme quantization schemes also have limited applicability and are difficult to achieve universality for all models and scenarios.

[0007] Knowledge distillation techniques transfer knowledge from a teacher model (a large model) to a student model (a small model) to achieve inference performance close to that of the teacher model with a smaller network. However, in the intelligent customer service scenario, knowledge distillation usually requires a large amount of high-quality labeled data or data generated by the teacher model, increasing the complexity and cost of implementation, and there is still a certain gap between the actual inference accuracy and the original model.

[0008] In addition, to address the elastic demand and accuracy loss issues in edge deployment, in recent years, technical routes such as dynamic sparsity techniques, edge-cloud collaborative inference techniques, and early exit network architectures have emerged. Dynamic sparsity techniques attempt to dynamically adjust the model sparsity according to the load demand during runtime to improve the elasticity and robustness of model deployment. However, existing dynamic sparsity techniques usually require recompiling the computation graph, resulting in a large runtime overhead and being difficult to meet the requirements of real-time scenarios. Edge-cloud collaborative inference techniques dynamically allocate complex tasks between the edge and the cloud to reduce latency and ensure inference quality. However, existing collaborative inference schemes usually use sentence-level or token-level uploading, resulting in a large network traffic and still having obvious network overhead issues in actual deployment.

[0009] In addition, although existing early-exit networks (such as inserting multiple intermediate exits in a Transformer network to output results as early as possible) can effectively reduce inference latency, most existing technologies fail to effectively utilize the token-level redundancy of the attention mechanism, resulting in waste of computing resources and cache storage, and there is a large room for optimization. There is no clear solution in the existing public literature on how to simultaneously achieve token-level pruning and hierarchical early exit, and reuse the key-value cache (KV-Cache) under the strict memory constraints of edge devices.

[0010] Although the existing technical routes can partially solve the problems in large model deployment, when comprehensively considering multiple dimensions such as model sparse compression and hardware friendliness, dynamic sparsity and robustness, edge-cloud collaborative communication efficiency, and joint optimization of early-exit networks and token-level pruning, there is still a lack of a mature method and technical solution that can effectively solve the above multi-objective optimization problems at the same time. Especially under the condition of severely limited resources of edge devices, how to maximize the maintenance of inference accuracy, while significantly reducing inference latency, reducing communication load and deployment cost, has become one of the long-term technical challenges in the industry.

[0011] Therefore, how to provide a lightweight large model intelligent customer service deployment method for edge computing is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0012] The objective of the present invention is to address the problems of high inference latency, large resource overhead, and heavy communication burden during the deployment of large language models on edge devices. A method is provided to reduce the edge inference overhead by combining multiple pruning compressions and dynamic inference optimizations, so as to significantly improve the model inference speed and reduce cross-device communication.To achieve the above objectives, the present invention adopts the following technical solutions: First, based on the "Lottery Hypothesis" theory, a sub-network screening is performed on the pre-trained large language model to explore the high-potential sub-model structure with performance close to the original model, and the sub-network is remapped into a structured sparse form of V:N:M mode, thereby greatly reducing the number of parameters and the amount of calculation while maintaining the model accuracy, so that the obtained sparse model can be compatible with hardware acceleration units such as TensorCore or NPU, and realize efficient acceleration of sparse matrix operations; second, a perplexity monitoring mechanism is constructed during the inference process. When it is monitored that the perplexity of the model for the input sequence increases significantly (that is, the uncertainty of the model increases), the sparse matrix is triggered. The sparse regeneration mechanism dynamically restores some of the network channels that were previously pruned (the number of restored channels does not exceed 5% of the total number of channels) to temporarily increase the model capacity when necessary and enhance the model's adaptability to complex or abnormal inputs. The regeneration mechanism is only triggered in a very small proportion of cases, so the impact on the overall computational overhead remains controllable. Third, multiple exit nodes are introduced in each layer of the model encoder, and combined with the attention pruning strategy, a dynamic early exit mechanism combining token level and layer level is implemented: for each input token, if its representation in a certain intermediate layer has reached a predetermined confidence threshold, the token can be exited early through the exit of that layer. Continue deeper calculations, while the remaining tokens that have not exited continue to undergo deeper processing; through the above-mentioned multi-exit design and pruning of low-contribution attention calculations, invalid calculations are effectively reduced, and each token can be represented only after the necessary number of layers, and the key-value (KV) cache of the attention module is shared in all exit processes, thereby avoiding repeated access and calculations, greatly improving the reasoning efficiency of the encoding stage; Fourth, the edge-cloud collaborative reasoning mechanism is introduced in the decoding stage, and the next reasoning strategy is dynamically determined according to the local model's confidence in the current generation result: when the confidence is high enough, the next predicted token is directly generated by the local model; if the confidence is insufficient, the next step of the prediction token is generated. After lightweight quantization, the hidden state is uploaded to the cloud-based MoE (Mixture of Experts) model for collaborative computing. The cloud-based MoE model performs large-scale expert reasoning on the uploaded hidden state and returns the key token predicted for the current step. The edge device then continues to perform subsequent local decoding processes and continuously reuses a unified KV cache throughout the entire cross-end inference process to maintain consistency in the context state across devices. In addition, by designing an expert prefetching strategy, possible cloud-based expert models can be selected and scheduled in advance to mask possible delay jitter in the edge-cloud interaction process, ensuring the stability and real-time performance of cross-device collaboration in the decoding stage.The above-mentioned modules cooperate closely in the inference process. By minimizing model redundant calculations and invalid data transmissions at the edge side to the greatest extent, the present invention only limitedly restores some pruning parameters or invokes the cloud expert model to provide assistance when the local model confidence is insufficient or higher-precision results are required. Such a design significantly reduces the computational load and cross-edge communication overhead of edge devices while ensuring the inference accuracy of the model, effectively alleviating the computational pressure and communication bottleneck problems faced by large models during edge deployment as a whole. The beneficial effects of the present invention include: achieving an average inference speed increase of about 1.8 - 2.0 times on edge devices, obtaining a significant acceleration effect at the cost of only a minimal accuracy loss (the BLEU evaluation index drops by no more than 0.3, and the translation quality remains basically the same) compared with the unoptimized large model, and maintaining the end-to-end inference latency within 80 milliseconds to meet the real-time application requirements; in the tests of common sense question-answering tasks such as CommonsenseQA, only about 7% of the tokens need to be calculated with the help of the cloud model, and the remaining 93% of the inferences are independently completed by local devices, reducing the cross-device data communication volume by two orders of magnitude compared with the traditional solution that completely relies on cloud inference; by reducing the number of cloud participations and data transmissions, the present invention reduces the overall deployment and inference cost by more than 50%. It can be seen that the present invention achieves a good balance between the inference performance and resource consumption of large models on the premise of ensuring model accuracy, providing an efficient and feasible technical solution for the edge deployment of large language models.

[0013] The beneficial effects of the present invention are as follows: (1) The present invention iteratively prunes the pre-trained large model based on the Lottery Ticket Hypothesis (LTH), then remaps the pruning results into a hardware-friendly V:N:M structured sparse format, and combines runtime perplexity monitoring with a sparse dynamic regrowth mechanism to dynamically restore no more than 5% of the pruned channels when the model has a high perplexity, effectively realizing the efficient inference of a highly sparse model on edge devices. On the premise that the BLEU index loss does not exceed 0.3pt, the inference latency of the edge-side model is significantly reduced to within 80ms, effectively improving the inference response speed and model stability.

[0014] (2) By setting multi-exit nodes in the Transformer network encoder, combining token-level pruning of the attention mechanism, and sharing the key-value cache (KV-Cache), the present invention realizes a dynamic early-exit inference mode that combines token granularity and layer granularity, significantly reducing the unnecessary computational burden during model operation. Actual tests show that the solution of the present invention increases the average inference speed of the edge device side by about 1. .8 - 2.0 times, showing better resource adaptability in edge environments with low computing power and low memory.

[0015] (3) During the edge-cloud collaborative decoding and inference process of the present invention, the hidden states corresponding to low-confidence or sensitive tokens are selectively uploaded to the cloud mixture-of-experts (MoE) model in an 8-bit quantization format driven by confidence. At the same time, the expert prefetch strategy is used to hide the cross-edge communication jitter, effectively solving the problems of high communication load and poor real-time performance in existing edge-cloud collaborative inference technologies. The cross-device communication volume is reduced by two orders of magnitude, and in the CommonsenseQA task, it is actually measured that only about 7% of the tokens require cloud assistance to achieve a significant improvement in accuracy, breaking through the communication bottleneck of traditional cloud-edge collaborative technologies, thereby effectively improving the application performance and deployment economy of real-time interaction scenarios such as intelligent customer service. Description of the Drawings

[0016] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a schematic diagram of the overall process of a lightweight large model intelligent customer service deployment method for edge computing proposed by the present invention; Figure 2 It is a schematic diagram of the V:N:M structured sparse mapping and dynamic sparse regeneration mechanism of a lightweight large model intelligent customer service deployment method for edge computing proposed by the present invention; Figure 3 It is a schematic diagram of the edge-cloud collaborative decoding and inference process of a lightweight large model intelligent customer service deployment method for edge computing proposed by the present invention. Detailed Embodiments

[0017] In the embodiments of the present invention, in step S1, the pre-trained LLaMA-2-7B large language model is used as the base model, and after the model pre-training is completed, weight quantization is performed with an INT8 bit width. The iterative pruning in step S2 specifically adopts the Lottery Ticket Hypothesis (LTH), the number of training iterations is set to 5 times, and the pruning rate for each time is set to 20% to gradually obtain the sub-network; subsequently, in step S3, block rearrangement is performed on the unstructured sparse winning ticket sub-network, where the value of V:N:M is specifically 64:2:5 to ensure compatibility with the support for structured sparse computing by the TensorCore hardware in the currently widely used NVIDIA Ampere architecture GPUs. In step S4, the sparse weights, masks, and complete weights of the present invention are uniformly loaded into the on-chip SRAM in a 128-byte aligned manner, and data transmission is achieved through the on-chip DMA controller to reduce the data transfer overhead and latency. In step S5, a side branch exit is inserted every 3 transformer layers in the Transformer network, and the adaptive token pruning algorithm specifically dynamically discards 50% of the tokens with the lowest scores in each layer according to the sorting of the attention scores of each token; the sorting of the attention scores is implemented using the quicksort algorithm, and the shared key-value cache (KV-Cache) is organized in a circular cache form to avoid the overhead of repeated calculations and frequent cache updates. In step S6, the runtime monitoring module continuously calculates the perplexity of the token sequence in a sliding window manner (the window length is fixed at 64 tokens), and when the perplexity continuously exceeds the preset threshold (the perplexity threshold is set to 40), the regrowth mechanism in step S8 is executed. At this time, the gradient sensitivity is calculated in real time in a backpropagation manner, and the pruned channel ratio is restored to 5% of the total number of channels according to the sorting from high to low of the gradient sensitivity. In step S9, when the confidence level is lower than the predetermined threshold (the confidence level threshold is set to 0.85) or a sensitive keyword is recognized, the corresponding hidden state vector is linearly quantized and mapped to the 8-bit integer range, and then transmitted to the mixture of experts (MoE) model in the cloud through the gRPC communication protocol, and the output of the local model is supplemented by the target token or log probability returned by the cloud. During this process, the edge side concurrently predicts the expert model parameters required for the next step and prefetches them in advance to mask the communication latency and jitter. Finally, in step S10, the local and cloud inference results are merged and output and recorded in the log.

[0018] The lightweight large model intelligent customer service deployment method for edge computing provided by the present invention, by adopting innovative mechanisms such as the lottery hypothesis combined with V:N:M structured sparse mapping, dynamic sparse recovery mechanism, adaptive token-level pruning, and edge-cloud collaborative decoding, reduces the edge inference latency of the model to within 80 ms, and the communication volume is reduced by more than two orders of magnitude compared with the prior art. In practical tasks such as CommonsenseQA, only about 7% of the tokens need to rely on cloud computing to achieve a significant improvement in accuracy; the overall deployment cost is reduced by more than 50%, while ensuring that the loss of dialogue accuracy does not exceed the 0.3ptBLEU metric, thus significantly improving the comprehensive performance and cost-effectiveness of deploying intelligent customer service applications on edge devices.

[0019] In the embodiment of the present invention, the specific step S1 further clarifies the collection and preprocessing method of the intelligent customer service historical corpus The data volume of the corpus is clearly defined as 1 million historical dialogue data. In step S1-2, the subword encoding specifically adopts the BPE (BytePairEncoding) algorithm, and the size of the subword vocabulary is clearly set to 32,000 subwords. In step S1-3, the specific number of architecture layers of the Transformer pre-training network is clearly set to 24 layers, the dimension size of the hidden layer vector is 4096 dimensions, and the number of self-attention heads is 32; the initial training parameters are initialized using the Xavier uniform initialization method; the learning rate during the pre-training process is set to 0.001, the optimizer adopts Adam, the training batch size is 2048, and each training totals 500,000 steps. Step S1-4 clarifies that the scale of the incremental instruction fine-tuning dataset is 100,000 instruction data, the learning rate is adjusted to one-tenth of the pre-training stage, that is, 0.0001, the training batch size during the fine-tuning process is 1024, and the total number of fine-tuning steps is clearly 100,000 steps. In step S1-5, the position encoding matrix specifically adopts the sine-cosine function position encoding method, and the final parameters are encapsulated as a weight file with FP32 floating-point precision. Step S1-6 clearly uses the SHA-256 algorithm to calculate the check value of the weight file and store it to ensure the consistency and integrity of the weight file during the model deployment process. The edge deployment pipeline in step S1-7 clearly uses the Docker containerized deployment method to load the model into the edge device.

[0020] Through the above-defined pre-training and fine-tuning processes, the present invention achieves the efficient adaptation of large language models to specific scenarios of intelligent customer service, ensuring the consistency, effectiveness, and stability of model parameters; by adopting a standardized sub-word encoding, model initialization, and training hyperparameter setting method, the accuracy and robustness of the model in the edge computing environment are improved, laying the foundation for subsequent pruning, structured sparsity, and edge inference optimization, and overall enhancing the implementability and actual performance of intelligent customer service applications during the deployment process on edge devices.

[0021] In the embodiment of the present invention, the parameter mask in step S2-1 is clearly initialized as a mask of all 1s, and the initial parameters are the initial parameters obtained through pre-training . In step S2-2, the number of iterative pruning times is clearly set to 5 times, and the proportion of parameters removed in each pruning is set to 20%. Specifically, the pruning is based on sorting the absolute values of the parameters, and the weight parameters with the smallest absolute values are removed; when continuing to train the network after each pruning, the learning rate is clearly set to 0.0005, the number of training steps is 100,000 steps, and the network is retrained after each round of iterative pruning to ensure the stability of the model accuracy. The parameters of the winner ticket subnet in step S2-3 are obtained by element-wise multiplication of the mask and the parameters . In step S2-4, the block size for rearranging the mask into a V:N:M structured sparsity is clearly set to 64, and the number of non-zero weights retained within each block is clearly set to 2, and the remaining 62 weights are all set to zero, thus satisfying the block constraint conditions of structured sparsity. Step S2-5 multiplies the structured sparsity mask and the parameters element-wise to output the structured sparsity weights , and these sparse weights are subsequently used for the deployment and inference on edge devices.

[0022] Through the above-defined LTH iterative pruning and structured rearrangement methods, the present invention obtains a winner ticket subnet in the V:N:M structured sparsity format, effectively reducing the computational load and memory consumption of the model during edge deployment, while ensuring high compatibility and hardware acceleration effects for mainstream hardware such as the NVIDIA Ampere architecture; by clearly determining the pruning ratio and training steps, it ensures that while significantly reducing the parameter scale of the sparse network, the model performance and inference accuracy basically do not suffer obvious losses, effectively improving the deployment efficiency and feasibility of large model intelligent customer service on edge devices.

[0023] In the embodiment of the present invention, in step S4-1, the bit width of the dense weights is further clarified to be 32 bits (FP32), the bit width of the sparse weights is 8 bits (INT8), and the bit width of the mask is 1 bit; the total number of parameters of the dense weights is the total number of final model parameters, specifically clarified to be 7 billion; the number of sparse weight parameters is clarified to be the number of non-zero parameters after structured sparse mapping, totaling 220 million; the number of masks is the total number of bits of the sparse mask, specifically clarified to be 7 billion bits. In step S4-2, the on-chip SRAM is clearly stored with an address division in a 128-byte alignment manner. The three address segments of the dense weights, sparse weights, and masks are stored in the form of consecutive address segments. Among them, the starting address of the sparse weight address segment follows the ending address of the dense weight address segment, and the starting address of the mask address segment follows the ending address of the sparse weight address segment. During the DMA transfer process from step S4-3 to S4-5, data blocks with a single transfer length of 4096 bytes are clearly used to copy the dense weight parameters, sparse weight parameters, and structured sparse masks to the corresponding on-chip address segments in sequence to ensure the efficiency of data transfer. In step S4-6, the SHA-256 algorithm is clearly used to calculate the check values of the dense weights, sparse weights, and mask data one by one and compare them with the pre-stored check values to ensure the consistency and integrity of the data; in step S4-7, the on-chip storage protection unit clearly enables the hardware locking method to configure the above address segments as read-only states to prevent the parameters and masks from being accidentally modified during the inference process.

[0024] Through the above-mentioned clear bit width setting, parameter scale, storage division, and data verification mechanism, the present invention effectively reduces the usage requirements and power consumption of the on-chip SRAM, ensures the efficient transfer and loading of sparse weights and masks, and at the same time ensures the integrity and security of the data during the inference process, further improving the deployment performance and reliability of the intelligent customer service large model on edge computing devices.

[0025] In the embodiment of the present invention, in step S5-1, the total number of layers of the Transformer network is clearly set to 24 layers, and the interval parameter for inserting the side branch exits is clearly set to 3. Therefore, a total of 8 side branch exits are inserted, located at the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th layers respectively; in step S5-2, in each layer that meets the insertion condition a side branch exit module is clearly newly built , and the exit module is composed of a fully connected layer and a softmax function, which is used to calculate the confidence of this layer. In step S5-3, it is clearly stated that in each transformer layer, when calculating the self-attention matrix the query vector With the key vector The dimension of is set to 128, and then the attention scores are calculated for each token respectively , and the importance ranking of each token is determined; the pruning ratio of step S5-4 is clearly set to 0.5, that is, for each transformer layer, 50% of the tokens with the lowest attention scores are dynamically discarded, and the key-value caches (KV-Caches) corresponding to the remaining 50% of the tokens with higher scores are updated and shared to the next transformer layer to avoid repeated calculations. The side-branch confidence threshold in step S5-5 is clearly set to 0.85. When the confidence of the current layer calculated at the side-branch exit is greater than or equal to 0.85, the side-branch exit directly outputs the inference result, otherwise the token continues to enter the next transformer layer for further inference calculation.

[0026] Through the above specific side-branch exit insertion, multi-level token pruning and dynamic KV cache sharing mechanism, the present invention effectively reduces the computational redundancy in the inference process, significantly reduces the computational overhead and memory occupation during the inference of edge devices, and further reduces the latency of the intelligent customer service inference process on the basis of ensuring the inference accuracy of the model, improving the real-time response ability and resource utilization efficiency of the intelligent customer service system in the edge computing environment.

[0027] In the embodiment of the present invention, step S6-1 clearly states that at runtime, every fixed-length token window is 64 tokens, and the prediction probability of each token in the window is calculated and the logarithmic average is taken to obtain the current window perplexity value . Step S6-2 further clarifies that the sliding average method is used to update the perplexity value, where the sliding average weight factor is clearly set to 0.8, that is, the weight of the current window perplexity value in the sliding average is 0.2. For each token in the window in step S6-3, the token confidence value is clearly recorded, which is the maximum value of the prediction output probability. The perplexity threshold in step S6-4 is clearly set to 40. When the sliding average perplexity exceeds the threshold 40 and continuously exceeds windows, the high perplexity flag is set to 1. Step S6-5 synchronously and clearly writes the updated sliding average perplexity , the token confidence set and the high perplexity flag value into the runtime monitoring buffer (the buffer size is clearly set to 1024 bytes) for use in subsequent steps.

[0028] Through the above-mentioned clear perplexity real-time monitoring mechanism and clearly set parameters, the present invention realizes the precise monitoring of the model running state, timely detects and identifies high-perplexity situations, ensures the accurate triggering of the subsequent sparse regrowth mechanism, thereby effectively guaranteeing the robustness and precision stability of model inference, and further improving the inference performance and user experience quality of the intelligent customer service system on edge devices.

[0029] In the embodiment of the present invention, step S7-1 clarifies the token set input at the initial inference The number of tokens in is clarified to be 64, and the key-value cache is initially an empty set. In step S7-2, when processing the transformer layer index layer by layer, the update method of the set is clearly defined as the retained set obtained according to attention pruning in the previous layer , and the retained set is specifically the set corresponding to the 32 tokens with the highest scores, and the update method of the key-value cache is specifically the union update of the previous layer cache and the key vectors calculated from each token in the current layer and the value vectors. In step S7-3, when calculating the side branch exit confidence, the dimension of the weight matrix is clearly defined as , and the dimension of the hidden state vector is clearly defined as 4096, and the confidence vector is a 256-dimensional vector, and the maximum element therein is the side branch exit confidence value . In step S7-4, the side branch exit confidence threshold is clearly defined as 0.85. When the hierarchical confidence reaches or exceeds this threshold, the currently generated token sequence is immediately output, and there is no need to continue the subsequent transformer layer inference. If the confidence does not reach the threshold, it is clearly defined in step S7-5 to continue the inference calculation process of the next transformer layer .

[0030] Through the above-mentioned clear token pruning and side branch confidence threshold settings, the present invention significantly reduces the redundancy of the calculations of each layer of the transformer during the inference process, ensures the early output of results when the confidence is high, effectively reduces the computational resource requirements and inference latency on edge devices, ensures the inference performance and response speed of the intelligent customer service model, and further improves the actual application effect of the intelligent customer service system in resource-constrained edge environments.

[0031] In the embodiment of the present invention, step S8-1 clarifies that during operation, the moving average perplexity is continuously monitored and exceeds the set threshold When it continuously exceeds the set number of times it enters the regeneration stage. In step S8-2, during the regeneration stage, the gradient sensitivity of the parameters corresponding to each pruned channel in the current model is calculated explicitly through real-time backpropagation . The gradient sensitivity is specifically defined as the magnitude of the two-norm of the corresponding parameter gradient, thereby clearly indicating the influence degree of each pruned channel on the model loss function. In step S8-3, the regeneration ratio is specifically 0.05, that is, the number of restored channels is clearly 5% of the total number of pruned channels. The selection of the regenerated channels is clearly based on the gradient sensitivity ranking, specifically restoring the top 5% of the channels with the highest sensitivity to generate the corresponding new mask . In step S8-4, the model sparse weights are updated explicitly in the way of element-wise multiplication of the mask and the complete parameters , and it is immediately applied to the subsequent inference process. In step S8-5, it is clearly stipulated that when the sliding average perplexity is continuously lower than or equal to the set threshold 40 for 3 consecutive times, the regeneration stage is stopped, and the original V:N:M structured initial mask is restored and the initial sparse weights are correspondingly restored . .

[0032] Through the above-mentioned clear sparse regeneration mechanism and parameter settings, the intelligent customer service model on the edge computing device dynamically improves the model capacity in complex input scenarios, quickly alleviates the problem of accuracy decline caused by high-perplexity inputs, and at the same time, by strictly limiting the regeneration ratio, effectively avoids a significant increase in computational and storage burdens, improves the stability and dynamic adaptability of the model inference accuracy in practical applications, and ensures the efficient operation of the edge device and the quality of the user interaction experience

[0033] In the embodiment of the present invention, in step S9-1, the confidence of each generated token is calculated explicitly , which is the maximum value of the probability of the corresponding token in the probability distribution. In step S9-2, the set confidence threshold is 0.85, and the sensitive keyword set clearly includes 500 pre-defined sensitive words. When the confidence of the generated token is lower than 0.85 or belongs to the sensitive word set, it enters the subsequent edge-cloud collaboration stage. In step S9-3, it is further clarified that the dimension of the hidden state vector is 4096, and the value of the quantization factor is clearly set to 0.02, and it is mapped to an 8-bit integer vector by linear quantization Step S9-4 specifies that the quantized hidden state is sent to the cloud in the form of a data packet through the gRPC communication protocol on the edge device. The size of the data packet is clearly limited to 4096 bytes. After receiving the data, the cloud large model outputs the target token in 32-bit floating-point format and the corresponding log probability vector Step S9-5 appends the target token returned by the cloud to the edge local token sequence and writes it into the edge device KV cache for subsequent inference. Step S9-6 specifies that according to the expert prediction score vector returned by the cloud MoE model gating unit, specifically using the Top-k algorithm to clearly select the top 3 experts with the highest scores for prefetching, and loading the corresponding expert model parameters into the edge cache in advance to reduce the latency and jitter caused by cross-edge communication

[0034] Through the above-mentioned clearly defined confidence-driven and edge-cloud collaborative mechanism, the present invention significantly reduces the communication load between the edge and the cloud while ensuring the inference accuracy and content security, effectively reducing the impact of latency and network jitter on real-time intelligent customer service applications; through the expert model prefetching strategy, the response speed and stability of the edge device are further optimized, significantly improving the comprehensive performance and user experience quality of intelligent customer service in the edge computing environment

[0035] In the embodiment of the present invention, step S10-1 specifies the token sequence locally generated by the edge device length is at most 64 tokens, and the token sequence returned by the cloud length is clearly limited to at most 8 tokens. The two sequences are merged through a simple concatenation operation to obtain a complete inference sequence In step S10-2, the reverse mapping function of the subword vocabulary constructed during pre-training is clearly used to map the merged token sequence to a customer service reply text sequence that can be directly displayed In step S10-3, the human-computer interaction interface of the edge device is clearly used to send the text reply to the user terminal through the HTTP / HTTPS protocol. The size of the text reply is limited to at most 128 characters; at the same time, in step S10-4, the merged complete token sequence and the confidence vector corresponding to each token are clearly written into the local log buffer. The size of the log buffer is clearly set to 2048 bytes, and the log buffer adopts a circular queue storage structure to complete the real-time recording and subsequent analysis of this conversation

[0036] Through the above-mentioned explicit sequence merging, vocabulary reverse mapping, and logging mechanisms, the present invention effectively ensures the timeliness and accuracy of the intelligent customer service response content, ensures the consistency of the content displayed on the user terminal, and at the same time provides detailed conversation records through the logging mechanism, which is beneficial to subsequent service quality analysis and model optimization, significantly improving the overall operation quality and maintainability of the intelligent customer service system of edge computing devices.

[0037] Example 1

[0038] To verify the feasibility of the present invention, the present invention is applied to a series of intelligent voice assistant devices under a large smart home brand B company.

[0039] In recent years, with the rapid development of the smart home industry, the sales volume of B company's intelligent voice assistant devices has been increasing continuously, and the devices have been widely deployed in homes, office places, hotels, and public service facilities. However, with the rapid increase in the number of devices, the traditional cloud-based centralized processing voice assistant architecture has gradually exposed serious problems, such as high response latency, huge bandwidth requirements, and increasing risks of privacy data leakage, which have severely restricted the user experience and product competitiveness. To address these issues, B company decided to deploy the lightweight large model intelligent customer service deployment method for edge computing proposed by the present invention to optimize the inference speed, communication volume, memory usage, and privacy security protection of the voice assistant.

[0040] In specific applications, B company first loaded the lightweight large language model that has been pre-trained and pruned proposed by the present invention on the intelligent voice assistant terminal device. During deployment, the model parameters built into the device further compress the model scale using the V:N:M structured sparse mapping technology to ensure that the model can run efficiently on edge devices. During the actual application process, when the user asks about the weather, news, or daily life information through the intelligent voice assistant, the large model on the device side will first directly perform local inference and determine whether to request cloud assistance through the confidence threshold and security keyword filtering mechanism. Specifically, requests with high confidence and no sensitive keywords are completely processed locally, and only a small number of tokens with low confidence or involving sensitive information need to be processed in cooperation with the cloud, thus greatly reducing the network bandwidth consumption and the risk of privacy leakage. At the same time, the adaptive token pruning and dynamic sparse regeneration mechanisms are also adopted during the local inference process to ensure that the device side can run efficiently with extremely low memory occupancy and faster response speed.

[0041] To objectively evaluate the effect of the present invention after actual deployment, B company conducted a deployment effect comparison test for three consecutive months from February 2024 to April 2024 at a smart home user test site in Shenzhen. The specific test device is a smart home voice assistant device equipped with 8GB of memory, and the total number of installed and deployed devices exceeds 1500. The comparison data is detailedly recorded in Tables 1, 2, and 3.

[0042] Table 1 Comparison Table of Device Inference Performance Before and After Deployment (Average Response Delay) Scene type Average response latency before deployment (milliseconds) Average response latency after deployment (milliseconds) Weather query 620 78 News information query 810 95 Smart home appliance control command 530 65 Chat interaction 780 90 As can be seen from the data in Table 1, after the deployment of the present invention, the average response delay of the intelligent voice assistant device in various common usage scenarios has been significantly reduced. For example, the response speed of weather query has increased by nearly 7.9 times, and the query speed of news and information has increased by about 8.5 times, greatly enhancing the real-time experience of users.

[0043] Table 2 Comparison Table of Communication Volume Optimization Effect Before and After Deployment (Daily Average Communication Volume per Device) Data category Daily average traffic before deployment Daily average traffic after deployment Reduction ratio of traffic Weather and information query 25.4MB 1.05MB 95.9% Smart device control 18.7MB 0.87MB 95.3% Voice interaction and chatting 35.2MB 1.43MB 95.9% As can be seen from Table 2, through the token-level communication control strategy of the present invention, the communication data volume between the intelligent voice assistant device and the cloud has been significantly reduced, and the average reduction ratio of the communication volume exceeds 95%, effectively alleviating the burden on the home broadband and the pressure on the cloud server.

[0044] Table 3 Comparison Table of Device Memory Occupancy and Privacy Protection Before and After Deployment Data category Memory occupancy before deployment Memory occupancy after deployment Reduction ratio of memory occupancy Privacy protection effect Model memory occupancy (average) 6.8GB 1.2GB 82.3% Greatly reduce the risk of sensitive data leakage Temporary data memory occupancy (peak) 4.2GB 0.75GB 82.1% Real-time filter sensitive keywords by more than 99% From the data in Table 3, through quantization, structural sparsity, and adaptive token pruning technologies on the device side, the present invention reduces the memory occupancy of the device-side model and temporary data by about 82%. At the same time, through token-level sensitive information filtering and low-confidence upload strategies, the security protection of privacy data is realized, effectively preventing the unnecessary transmission of sensitive data, and the accuracy of real-time filtering of sensitive keywords exceeds 99%, greatly enhancing the level of user privacy protection.

[0045] In addition, to further confirm the actual user experience and the improvement effect of security performance after deployment, Company B randomly selected 500 users at the test site for a questionnaire survey. The survey content was the changes in user satisfaction with the voice assistant experience and privacy protection trust. The specific survey data is shown in Table 4 below: Table 4 User Questionnaire Satisfaction Survey Form Investigation content Very satisfied (%) Satisfied (%) Average (%) Dissatisfied (%) Response speed satisfaction (after deployment) 75.8 19.2 3.4 1.6 Response speed satisfaction (before deployment) 10.4 21.3 45.2 23.1 Privacy protection trust (after deployment) 82.7 14.5 2.1 0.7 Privacy protection trust (before deployment) 9.2 18.7 41.8 30.3 As can be seen from Table 4, after the deployment of the present invention, the user satisfaction with the response speed of the voice assistant has rapidly increased from less than 32% to more than 95%, and the privacy protection trust has also significantly increased from less than 28% to more than 97%. Users generally report that the device is significantly faster and the sense of privacy security is significantly improved.

[0046] By implementing the lightweight large model intelligent customer service deployment method for edge computing described in the present invention on the smart home voice assistant device of Company B, the core problems such as high response latency, huge bandwidth requirements, and insufficient privacy protection in the prior art are effectively solved. The performance of the smart home voice assistant product and user satisfaction are significantly improved, and at the same time, the data security guarantee ability is greatly enhanced. The data in the implementation cases fully prove that the present invention has outstanding deployment effects and significant technical advantages in real scenarios, and has the technical conditions and market prospects for large-scale popularization and application.

Claims

1. A lightweight large model intelligent customer service deployment method for edge computing, including the steps: S1. Obtain and load the pre-trained large language model and its parameters; S2. Iteratively prune based on the lottery hypothesis to obtain the winning ticket subnet; S3. Convert the winning ticket subnet into a structured sparse format to generate a sparse mask; S4. Load the complete weights, sparse weights, and sparse masks into the edge device for storage; S5. Insert side branch exits in the model transformer layer and set a token pruning operator based on attention scores in the self-attention module; S6. Establish a monitoring module to calculate the perplexity and token confidence of the conversation sequence; S7. Dynamically discard low-score tokens during inference, reuse the key-value cache, and directly output if the side branch exit is higher than the first threshold, otherwise execute the next layer; S8. When the perplexity continuously exceeds the second threshold, restore no more than 5% of the pruned weights and reapply the sparse mask; S9. Quantize the hidden states of low-confidence or sensitive tokens and upload them to the cloud, receive the returned results, and prefetch the expert parameters; S10. Merge the local and cloud tokens to generate the customer service reply for output.

2. The lightweight large model intelligent customer service deployment method for edge computing according to claim 1, wherein The specific steps of S1 include: S1-1, collect the historical dialogue corpus of intelligent customer service to form a corpus set ; S1-2, for the corpus Perform subword encoding to obtain a token sequence and build a vocabulary ; S1-3. On the distributed training cluster, with an autoregressive objective function ; perform pre-training on the Transformer network to obtain initial parameters ; S1-4, continue to train the dataset with incremental instructions to obtain the final parameters ; S1-5, encapsulate the vocabulary , the position encoding matrix and the final parameters into a binary weight file to form a pre-trained large language model; S1-6. Calculate the SHA-256 checksum of the weight file and store it; S1-7. Load the pre-trained large language model into the edge deployment pipeline.

3. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that, The specific steps of S2 include: S2-1, Initialize the parameter mask ; S2-2, for the iteration index perform the following steps in sequence: , , ; S2-3, the winner ticket network is obtained after iteration ; S2-4, rearrange the mask to satisfy the structural constraints of the mask , whose block-level constraints are ; S2-5, output sparse weights .

4. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that The specific steps of S3 include: S3-1, Set the block size And retain the number of non-zero weights , in the winner ticket mask Partition the block set , satisfy ; S3-2, for each piece calculate ; S3-3, when , only retain the index with the largest absolute weight and assign 0 to the remaining masks; S3-4, when From the indexes with a mask of 0, select The one with the largest weight, and assign 1 to the mask; S3-5, after completing all block processing, obtain a structured mask that satisfies ; ; S3-6, calculate sparse weights , and store them in the on-chip memory of the edge device.

5. The lightweight large model intelligent customer service deployment method for edge computing according to claim 1, wherein The specific steps of S4 include: S4-1, set the dense weight bitwidth , the sparse weight bitwidth , the mask bitwidth , and calculate ； ； ； S4-2, divide the address segment in the on-chip static random access memory in 128B alignment , , , Among them ; S4-3, the dense weights are copied to the corresponding address segment via the direct memory access controller ; S4-4, the sparse weights are copied to the corresponding address segment via a direct memory access controller ; S4-5, the sparse mask is copied to the corresponding address segment via the direct memory access controller ; S4-6. Calculate the SHA-256 checksums of the dense weight data, sparse weight data, and sparse mask data respectively and compare them; S4-7. When the comparison is consistent, enable the memory protection unit to set the 128B aligned address segments for storing dense weight data, the 128B aligned address segments for storing sparse weight data, and the 128B aligned address segments for storing sparse mask data as read-only.

6. The lightweight large model intelligent customer service deployment method for edge computing according to claim 1, wherein, The specific steps of S5 include: S5-1, set the total number of converter layers and the side branch insertion interval ( ) to judge whether the layer index meets the condition ; S5-2, when the formula (S5-1) is satisfied, after the output of the layer, a new lateral branch outlet is added ; S5-3, in each transformer layer, the self-attention matrix ; is calculated ; S5-4, set pruning percentage , discard tokens to obtain the retained set and synchronously update the key-value cache; S5-5, Set the confidence of the lateral branch , when ; directly output the inference result, otherwise continue to execute the layer.

7. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that The specific steps of S6 include: S6-1, Calculate the current perplexity ; S6-2, Update the sliding average perplexity ; S6-3, Record Token Confidence ; S6-4, when and continuously times, set the high perplexity flag bit; S6-5, write , and the flag bit into the runtime monitoring buffer.

8. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that The specific steps of S7 include: S7-1, set the token set at the start of inference and the key-value cache ; S7-2, layer index Execute sequentially: (a) ； (b) ； S7-3, calculation ; S7-4, when , immediately output the current generated sequence; S7-5, when is true, continue to execute layer .

9. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that The specific steps of S8 include: S8-1, when and continuously times, enter the regeneration stage; S8-2, Calculate gradient sensitivity ; S8-3, Set the regeneration ratio , Select channels and set them to 1 to obtain a mask ; S8-4, Update Sparse Weights ; S8-5, when and continuously times, restore the initial mask and restore the weights .

10. A lightweight large model intelligent customer service deployment method for edge computing according to claim 1, characterized in that, The specific steps of S9 include: S9-1, Generate a token Calculate ; S9-2, when or execute steps S9-3 to S9-6; S9-3, Quantifying Hidden States ; S9-4, send to the cloud large model via gRPC and receive the target token and the log probability ; S9-5, Append the token to the local sequence and write it to the local key-value cache; S9-6, according to the cloud gating score Predict and prefetch the corresponding expert weights in parallel; The specific steps of S10 include: S10-1, Set the locally generated token sequence , Set the token sequence returned by the cloud , and merge to obtain ; S10-2, merge the sequences into the inverse mapping function of the vocabulary to obtain the text reply ; S10-3, output a text response through the edge device human-machine interface and send it to the user terminal; S10-4, write the merged token sequence and the confidence vector to the log buffer to complete the conversation record.

Images