Efficiency and privacy-oriented model-end cloud collaborative inference method and system

Abstract

The application relates to an efficient and privacy-considered large model end-cloud collaborative inference method and system, which comprises the following steps: obtaining input and configuring KV cache control parameters through an end side; performing embedding calculation to generate a feature tensor and uploading the feature tensor to a cloud side; performing decoding layer inference on the cloud side according to authorization to call the KV cache and returning a hidden state; in a current generation step, performing speculative decoding prediction on subsequent tokens based on the hidden state on the end side, and calculating an end-side local inference result corresponding to a low-dimensional feature according to a dimension splitting rule; calculating a cloud-side local inference result corresponding to a high-dimensional feature on the cloud side, and returning the cloud-side local inference result after language self-adaptive cutting and encryption; fusing the two-end local inference results to obtain a complete distribution, sampling to generate an incremental token, uploading the incremental token to the cloud side for verification, and updating the KV cache.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a large-scale model edge-cloud collaborative reasoning method and system that balances efficiency and privacy. Background Technology

[0002] In recent years, large-scale pre-trained models have been widely used in smart terminals and online services. However, their inference deployment faces an engineering dilemma: it is difficult to simultaneously satisfy the requirements of "computing power cost, interaction latency, and privacy compliance," especially in collaborative scenarios across different types of devices. On the one hand, the computing power and storage of different types of edge devices (pure CPU, GPU, and edge devices) vary greatly, and there are adaptation challenges in deploying complete models. GPU edge devices lack efficient model splitting and multi-session scheduling solutions, edge devices have poor cross-platform compatibility, and cross-device data transmission can easily exacerbate the risk of privacy leakage. On the other hand, although cloud inference has sufficient computing power, it needs to transmit user input or history_ids-like information, which poses a serious risk of privacy leakage. Furthermore, when KVCache is synchronized across different types of devices, the differences in device communication capabilities and computing power lead to a surge in bandwidth overhead and latency.

[0003] In existing solutions, the hidden state transmitted between the edge and cloud is vulnerable to being reconstructed by attackers through enumeration of different model parameters. Furthermore, they generally lack a dual encryption design of "traditional encryption algorithm + hidden state protection." Relying solely on a single encryption method or local layer fine-tuning is insufficient to resist complex attacks, resulting in significant shortcomings in privacy protection. At the same time, due to the lack of GPU acceleration, network latency accounts for a very high proportion of the overall inference latency in pure CPU edge devices. Existing technologies lack latency hiding mechanisms for pure CPU scenarios, leading to low efficiency in collaborative inference. The computation of the full classification head on the edge consumes a lot of computing resources, further exacerbating the latency problem. The large amount of data transmitted in the distributed transmission of complete logits results in significant transmission latency in low-bandwidth scenarios.

[0004] For the aforementioned technologies, existing solutions generally lack KVCache permission control mechanisms for different types of device scenarios, cross-platform compatibility solutions for edge devices, and pure CPU-side latency optimization solutions, making it difficult to achieve both efficient inference and deep privacy protection under various types of consumer terminals.

[0005] Based on this, this application provides a large-scale model edge-cloud collaborative reasoning method and system that balances efficiency and privacy. Summary of the Invention

[0006] To address the shortcomings of existing solutions, such as the lack of KVCache permission control mechanisms for different device types, cross-platform compatibility solutions for edge devices, and pure CPU-based latency optimization solutions, which make it difficult to achieve both efficient inference and deep privacy protection under various types of consumer terminals, this application provides a large-model edge-cloud collaborative inference method and system that balances efficiency and privacy.

[0007] Firstly, this application provides a large-scale model-cloud collaborative inference method that balances efficiency and privacy, employing the following technical solution: including:

[0008] The system acquires user input data from the edge device, performs preprocessing to generate an input representation, and configures inference control parameters. These inference control parameters include KVCache control parameters, which instruct the cloud server whether to enable and for how long to use the historical KVCache for this inference. Simultaneously, the control edge device configures device adaptation parameters according to its own hardware type. These device adaptation parameters include at least one of the following: block transfer parameters for pure CPU edge devices, model splitting parameters for GPU edge devices, and ONNX runtime parameters for edge devices.

[0009] Embedding computation is performed based on the input representation to generate an initial feature tensor; the initial feature tensor is adapted according to the device adaptation parameters, and the processed feature tensor, KVCache control parameters, and language type identifier are uploaded to the cloud.

[0010] Based on the data received from the cloud server, and according to the KVCache control parameters, authorize the invocation or reconstruction of KVCache, combine the received feature tensor to perform decoding layer inference, generate the first hidden state, and return the first hidden state to the end device;

[0011] In the current token generation step, the edge device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates an edge-side partial inference result and a speculative decoding prediction result; the edge uploads the incremental token and the speculative decoding prediction result to the cloud; the speculative decoding prediction result is verified and the KVCache is updated, and at the same time, cloud-based partial inference is performed based on the high-dimensional part of the current hidden state to generate a cloud-based partial inference result; the cloud sends back the cloud-based partial inference result after pruning and encryption; the edge merges the edge-side partial inference result and the cloud-based partial inference result to generate the complete inference result of the current step and outputs the incremental token;

[0012] Based on the complete inference result and the incremental token, the control-side device decodes and post-processes the final generated token sequence and outputs the inference result.

[0013] Preferably, after the user input data acquired from the edge device is preprocessed to generate an input representation and inference control parameters are configured, the method further includes:

[0014] The language type identifier is automatically determined based on the ratio of Chinese to English in the input prompts, or determined based on the user's language preferences, including Chinese monolingual, English monolingual, or mixed Chinese and English;

[0015] KVCache control parameters include cache enable flag and cache usage length. Cache is disabled by default for new sessions and enabled by default for non-new sessions. The cache usage length is dynamically adjusted by the end device based on the session context, and its value ranges from 1 to the current total length of KVCache.

[0016] The device adaptation parameters include: For pure CPU-side devices, the block transmission parameters include the number of blocks, the block order identifier and the merging rules. The number of blocks is adaptively configured according to the feature tensor dimension and the CPU computing power, and the dimension of each sub-feature tensor is consistent.

[0017] For GPU-based edge devices, the model splitting parameters include the splitting granularity and the number of sessions in parallel processing per device. The splitting granularity supports splitting by Transformer level or attention head level, and the number of sessions in parallel processing is configured according to the GPU memory capacity.

[0018] For edge devices, ONNX runtime parameters include quantization bit width and hardware acceleration instruction set call configuration.

[0019] Preferably, the step of performing embedding computation based on the input representation to generate an initial feature tensor; performing adaptation processing on the initial feature tensor according to device adaptation parameters; and uploading the processed feature tensor, KVCache control parameters, and language type identifier to the cloud includes:

[0020] Embedding computation is performed based on the input representation to generate the initial feature tensor. The embedding computation includes a weighted summation of word embeddings, position embeddings, and type embeddings.

[0021] For pure CPU-side devices, the initial feature tensor is divided into multiple sub-feature tensors by columns, and verification information is added to each sub-feature tensor to ensure data integrity.

[0022] For GPU-side devices, the model slices are loaded at a preset layer granularity using the nano-vllm engine, and a tensor parallel strategy is used to accelerate the embedding computation.

[0023] For edge devices, embedded computation is performed by calling the hardware acceleration instruction set through ONNXRuntime, and the quantization results in the input preprocessing are reused;

[0024] The processed feature tensor, KVCache control parameters, and language type identifier are uploaded to the cloud.

[0025] Preferably, the step of receiving data from a cloud server, authorizing the invocation or reconstruction of the KVCache according to KVCache control parameters, performing decoding layer inference in conjunction with the received feature tensor to generate a first hidden state, and returning the first hidden state to the end device includes:

[0026] Based on receiving data from the cloud server and authorizing the invocation or reconstruction of KVCache according to KVCache control parameters, the feature tensor is converted into the first hidden state through self-attention mechanism and feedforward network operation.

[0027] When the first hidden state is returned to the edge device via the cloud server, for a pure CPU edge device, the first hidden state is split into multiple sub-hidden states according to the block parameters uploaded by the edge device, and then transmitted sequentially to the edge device in a streaming manner, while supporting a retransmission mechanism; for GPU edge devices and edge devices, the complete first hidden state is returned directly.

[0028] Preferably, in the current token generation step, the edge device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates edge-side partial inference results and speculative decoding prediction results; the edge device uploads the incremental token and speculative decoding prediction results to the cloud; the speculative decoding prediction results are verified and the KVCache is updated, including:

[0029] Based on the edge device, the low-dimensional part of the current first hidden state is extracted according to the preset dimensional splitting rules, and input into the locally deployed lightweight MLP computation to obtain the edge local inference result.

[0030] The cloud server extracts the high-dimensional part of the current first hidden state according to the preset dimension splitting rules, inputs it into the complete MLP computation deployed in the cloud, and obtains the cloud local inference result;

[0031] The complete inference result is obtained by adding the local inference result on the edge and the local inference result on the cloud element by element. The vocabulary dimension corresponding to the non-target language in the complete inference result is masked according to the language type identifier. The sampling probability is then obtained by the Softmax function, and an incremental token is generated based on the sampling probability.

[0032] Based on the current first hidden state, the speculative decoding prediction result is generated by the speculative decoding module deployed on the edge. The speculative decoding module is an MLP structure or a Transformer layer plus an MLP structure, and its prediction step size is dynamically configured according to the type of edge device.

[0033] The client uploads the incremental token and the speculative decoding prediction result to the cloud.

[0034] The cloud server determines the consistency between the predicted token and the actual decoded token. If the verification is successful, the prediction result is reused directly, the decoding calculation is skipped, and the KVCache is updated.

[0035] Preferably, the method further includes:

[0036] The lightweight computing module deployed locally on the edge device is implemented based on a model that has undergone full-layer LoRA fine-tuning. Full-layer LoRA fine-tuning includes:

[0037] A low-rank adapter is added to each layer of the base model. During fine-tuning, only the weights of the low-rank adapter are updated, while the backbone parameters of the base model are frozen. The weights of the low-rank adapter after fine-tuning are only stored on the edge device and are not transmitted to the cloud.

[0038] The rank is finely tuned based on the type of edge device, with higher rank used for GPU edge devices and lower rank used for edge devices and pure CPU edge devices.

[0039] For pure CPU-based devices, the fine-tuned weights are split into multiple sub-weight matrices to adapt to block-based computation.

[0040] For GPU-based devices, the fine-tuned weights are fused with the base model weights and then split into layers with granularity.

[0041] For edge devices, the fine-tuned weights are embedded into the ONNX model computation graph and quantized and compressed during export.

[0042] Preferably, the cloud transmits the partially inference result back to the endpoint after pruning and encrypting it, including:

[0043] The end-side device performs quantization processing on the data to be uploaded and transmitted, converting floating-point numbers into integers of a specified bit width. The data to be transmitted includes the processed feature tensor, the incremental token, and the feature data corresponding to the speculative decoding prediction result.

[0044] The quantized data is encrypted using AES-GCM with a pre-shared key and a random initialization vector, and the ciphertext is then uploaded to the cloud.

[0045] The cloud-based partial inference results are subjected to the same quantization and encryption processing on the cloud server; at the receiving end, the same key and initialization vector are used for decryption and dequantization to restore the original data.

[0046] For pure CPU-based edge devices, each sub-feature tensor and its corresponding return data are assigned an independent initialization vector and bound to a block identifier; for GPU-based edge devices and edge devices, an independent initialization vector is assigned per session; the decryption and computation processes are performed in tandem, so that the decryption time overlaps with the computation time, thereby reducing latency.

[0047] Secondly, this application provides a large-scale model edge-cloud collaborative inference device that balances efficiency and privacy, employing the following technical solution:

[0048] The data input module is used to preprocess user input data acquired from the edge device, generate an input representation, and configure inference control parameters. The inference control parameters include KVCache control parameters, which are used to indicate whether the cloud server enables and how long of the historical KVCache is used for this inference. At the same time, the control edge device configures device adaptation parameters according to its own hardware type. The device adaptation parameters include at least one of the following: block transfer parameters for pure CPU edge devices, model splitting parameters for GPU edge devices, and ONNX runtime parameters for edge devices.

[0049] The feature calculation module is used to perform embedding calculations based on the input representation to generate an initial feature tensor; to perform adaptation processing on the initial feature tensor according to the device adaptation parameters; and to upload the processed feature tensor, KVCache control parameters and language type identifier to the cloud.

[0050] The cloud receiving module is used to receive data based on the cloud server, authorize the invocation or reconstruction of KVCache according to the KVCache control parameters, perform decoding layer inference in combination with the received feature tensor, generate the first hidden state, and return the first hidden state to the end device.

[0051] The data inference module is used in the current token generation step. The edge device receives the first hidden state returned from the cloud, performs partial inference based on a locally deployed lightweight computing module, and generates an edge-side partial inference result and a speculative decoding prediction result. The edge device uploads the incremental token and the speculative decoding prediction result to the cloud. The speculative decoding prediction result is verified and the KVCache is updated. Simultaneously, based on the high-dimensional part of the current hidden state, cloud-based partial inference is performed to generate a cloud-based partial inference result. The cloud-based partial inference result is pruned, encrypted, and then sent back to the edge device. The edge device merges the edge-side partial inference result and the cloud-based partial inference result to generate the complete inference result for the current step and outputs the incremental token.

[0052] The data decoding module is used to control the terminal device to decode and post-process the final generated token sequence based on the complete inference result and the incremental token, and output the inference result.

[0053] Thirdly, this application also provides a control device, the device comprising:

[0054] It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the large-scale model edge-cloud collaborative inference method described above, which balances efficiency and privacy.

[0055] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the large-scale model edge-cloud collaborative reasoning method that balances efficiency and privacy.

[0056] In summary, this application addresses the issues of high privacy leakage risk, low collaboration efficiency, and poor multi-device compatibility in traditional solutions through edge-cloud functional layering and access control. The technical solution adopts an architecture combining lightweight edge processing and heavy cloud computing: the edge handles input preprocessing, embedding computation, speculative decoding, and low-dimensional feature computation of the classification head, while the cloud performs complete decoding layer inference, KV cache management, and high-dimensional feature computation of the classification head. For security, a dual encryption system of "traditional encryption algorithm + hidden state protection" is constructed. Transmitted data is encrypted using AES-GCM, and a full-layer LoRA fine-tuning model is deployed on the edge. Fine-tuning weights are only stored on the edge to prevent the hidden state from being enumerated and restored. The edge controls cache access permissions through KV cache control parameters to avoid the plaintext transmission of historical identifiers. In terms of efficiency, edge-side speculative decoding predicts subsequent tokens in advance, reducing interaction rounds; pure CPU edge-side uses block transmission and parallel computing to hide network latency; GPU edge-side improves throughput through nano-VLLM model segmentation; edge devices export ONNX format to achieve cross-platform compatibility; classification headers are split and calculated in 96-dimensional and 4000-dimensional formats, reducing edge-side computing power consumption; language adaptive pruning reduces the amount of data transmitted. Thus, under the same bandwidth conditions, throughput exceeds that of the native VLLM engine, edge-cloud collaboration latency is significantly reduced, and it supports full-scenario adaptation for pure CPU, GPU, and edge devices, achieving dual protection of efficiency and privacy. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a large-scale edge-cloud collaborative inference method that balances efficiency and privacy.

[0058] Figure 2 This is a structural block diagram of a large-scale edge-cloud collaborative inference device that balances efficiency and privacy. Detailed Implementation

[0059] The following combination Figures 1-2 This application will be described in further detail.

[0060] The subject of this application is a control system, which includes two parts: an edge subsystem and a cloud subsystem.

[0061] The edge-side subsystem adapts to the hardware characteristics of different types of edge-side devices, and mainly includes the following modules:

[0062] Input preprocessing module: Converts raw user input into a model-processable representation using an input representation transformation formula; Language type recognition module: Detects language features of the input prompt or reads user language preferences, outputting a language type identifier; Embedding calculation module: Generates initial feature tensors using an embedding calculation formula; KVCache parameter configuration module: Supports dynamic configuration of cache enable flags, cache usage length, and device adaptation parameters; Full-layer LoRA fine-tuning model deployment module: Deploys models fine-tuned using full-layer LoRA, with weight updates following the full-layer LoRA fine-tuning formula; Speculative decoding module: An MLP or Transformerlayer+MLP structure used to predict subsequent tokens in advance using speculative decoding formulas; Distributed classification head. The computation module is responsible for feature tensor dimension splitting, edge-side local logits (edge-side local inference results) calculation, and fusion with cloud-based logits, implemented through a distributed calculation formula for the classification head; the language adaptive distribution processing module prunes and pads the complete logits (complete inference results) with negative infinity based on language type identifiers, implemented through a language adaptive distribution pruning formula; the sampling control module performs sampling based on distribution probability; the result output module post-processes and displays the token sequence; the pure CPU block collaboration module is responsible for feature tensor block partitioning, sub-block reception, parallel computation scheduling, and result merging; the nano-vllm adaptation module is responsible for model splitting, slice loading, tensor parallel scheduling, and multi-session management; and the ONNX model deployment module is responsible for ONNX model loading, hardware EP matching, quantization adaptation, and computation graph optimization.

[0063] The cloud subsystem is deployed on a cloud server and mainly includes the following modules: KVCache authorization call module: extracts authorization cache entries according to end-side parameters; Decoding layer inference module: performs decoding layer operations, implemented through decoding layer inference formulas; Cache maintenance module: responsible for the creation and updating of KVCache; Speculative decoding verification module: responsible for validating the speculative decoding prediction results transmitted from the end-side; Multi-GPU parallel inference engine: supports multi-GPU collaborative computing; Pure CPU block transmission module: responsible for hidden state block segmentation, streaming push, retransmission control, and block identifier management; nano-vllm collaboration. The module is responsible for aligning the communication protocol with the edge-side nano-vllm adaptation module; the ONNX adaptation communication module is responsible for optimizing the transmission protocol between the edge device and the cloud; the classification head cloud computing module extracts high-dimensional features based on the edge-side splitting rules, performs complete cloud MLP computation to obtain cloud local logits (cloud local inference results), and implements it through the classification head distributed computing formula; the logits pruning and transmission module prunes the cloud local logits according to the language type identifier, transmits them to the edge after encryption and quantization, and implements it through the language adaptive distributed pruning formula and encryption-related formula.

[0064] The edge-cloud communication interface adopts the HTTP / 3 protocol and supports the quantization, encryption, and transmission of feature tensors, speculative decoding prediction results, and pruned logits, following the encryption and quantization process formula. For pure CPU edge scenarios, block encoding and streaming transmission mechanisms are enabled; for GPU edge scenarios, the efficiency of parallel tensor transmission is optimized; and for edge device scenarios, efficient transmission of ONNX quantized data is supported.

[0065] In this application, GPU-side devices refer to PCs, flagship mobile phones, high-end vehicle terminals, etc.; edge devices refer to IoT gateways, entry-level vehicle terminals, etc.; and pure CPU devices refer to old PCs, lightweight industrial control computers, etc.

[0066] Reference Figure 1 The embodiments of this application include at least steps S10 to S50.

[0067] S10: Based on the user input data obtained from the terminal device, preprocess it to generate an input representation and configure inference control parameters; at the same time, control the terminal device to configure device adaptation parameters according to its own hardware type.

[0068] The inference control parameters include KVCache control parameters, which indicate whether the cloud server enables and how long of the historical KVCache is used for this inference; the device adaptation parameters include at least one of the following: block transfer parameters for pure CPU-side devices, model splitting parameters for GPU-side devices, and ONNX runtime parameters for edge devices.

[0069] S20 performs embedding computation based on the input representation to generate an initial feature tensor; adapts the initial feature tensor according to the device adaptation parameters, and uploads the processed feature tensor, KVCache control parameters, and language type identifier to the cloud.

[0070] S30 receives data from the cloud server, authorizes the invocation or reconstruction of KVCache according to the KVCache control parameters, performs decoding layer inference in combination with the received feature tensor, generates the first hidden state, and returns the first hidden state to the end device.

[0071] S40, in the current token generation step, the terminal device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates the terminal-side partial inference result and the speculative decoding prediction result; the terminal uploads the incremental token and the speculative decoding prediction result to the cloud; the speculative decoding prediction result is verified and the KVCache is updated, and at the same time, the cloud partial inference is performed based on the high-dimensional part of the current hidden state to generate the cloud partial inference result; the cloud sends back the cloud partial inference result after pruning and encryption; the terminal merges the terminal-side partial inference result and the cloud partial inference result to generate the complete inference result of the current step and outputs the incremental token.

[0072] S50, based on the complete inference result and the incremental token, the control-side device decodes and post-processes the finally generated token sequence and outputs the inference result.

[0073] It should be understood that an incremental token refers to a single output token newly generated in the current inference step during the autoregressive generation process of a large model.

[0074] Specifically, the processing on the cloud and edge devices includes the following steps:

[0075] a) Edge-side input acquisition, preprocessing, and control parameter configuration: The edge device receives user input data, preprocesses it, and converts the raw input into an input representation for large-scale model inference using an input representation conversion formula; it detects the language type of the input prompt or obtains the user's set response language preference to determine the language type identifier (Chinese monolingual / English monolingual / Chinese-English mixed); simultaneously, it configures KVCache control parameters, including a cache enable flag (flag_cache) and cache usage length (L_cache). Except for new sessions, caching is enabled by default and a reasonable usage length is configured; if the edge is purely CPU-based... For the device, additional block parameters are configured, including the number of blocks (N_block), block order identifier, and merging rules; if the device is a GPU-enabled device, configure the nano-vllm model splitting parameters, including the splitting granularity and the number of parallel sessions per device; if the device is an edge device, confirm the model ONNX export status and quantization configuration parameters (4bit / 8bit / 16bit); configure the classification head distributed computing parameters, and clarify the feature tensor dimension splitting rules (the number of dimensions processed on the device side D_end, the number of dimensions processed on the cloud D_cloud, satisfying D_end+D_cloud=total total feature dimension D_total).

[0076] The input represents the conversion formula as follows:

[0077] ;

[0078] in, Input data for the original user, This is a word segmentation function. For the input projection weight matrix, For bias terms, For layer normalization operations, satisfy , For the mean of the input data, For the variance of the input data, To prevent tiny values ​​with a denominator of zero (range of values) );

[0079] Language type identification rules: If the proportion of Chinese characters in the input prompt is ≥95%, it is determined to be Chinese monolingual; ≤5%, it is determined to be English monolingual; 5%-95%, it is determined to be Chinese-English mixed; dynamic switching of language type during conversation is supported, and the distribution and pruning rules are re-identified and adjusted in real time on the device side;

[0080] The configuration rules for KVCache control parameters are as follows: flag_cache=false for a new session, flag_cache=true by default for a non-new session, L_cache can be set by the client side according to the session context, and the value range is from 1 to the current total length of KVCache. The client side can dynamically adjust the parameters according to privacy requirements.

[0081] The configuration rules for block parameters on the pure CPU side are as follows: the number of blocks (N_block) is adaptively configured according to the feature tensor dimension and CPU computing power, and the value is an integer from 2 to 8. When the feature tensor dimension is [1,D], the dimension of each sub-feature tensor is [1,D / N_block]. The block order identifier is marked as 1 to N_block according to the transmission order. The merging rule is to concatenate the blocks column by column according to the block order identifier.

[0082] The configuration rules for the splitting parameters of the GPU-enabled nano-vllm model are as follows: the splitting granularity supports Transformer-level / attention-head-level splitting, and the number of parallel sessions per device is configured to 2-8 based on the GPU memory capacity (≥4GB), ensuring that the memory usage of a single session is ≤40% of the total GPU memory.

[0083] The ONNX configuration parameters for edge devices are as follows: the default quantization bit width is 8 bits, low memory devices (memory ≥ 2GB and < 4GB) are adapted to 4-bit quantization, and the Embedding layer, speculative decoding module, full-layer LoRA fine-tuning related adaptation layer and classification head lightweight MLP computation graph optimization are retained when the model is exported. It supports hardware acceleration instruction set calls such as ARMNEON and IntelAVX.

[0084] Classification head distributed computing parameter configuration rules: The total feature dimension D_total is fixed at 4096, the edge processing dimension D_end=96, the cloud processing dimension D_cloud=4000, the edge lightweight MLP weight dimension is [96,150000], and the cloud full MLP weight dimension is [4000,150000]. All weights are fine-tuned and optimized through full-layer LoRA.

[0085] b. Edge-side Embedding Calculation: Based on the input representation, the edge device generates an initial feature tensor of dimension [1, D_total] (D_total=4096) using the relevant formulas for embedding calculation. The pure CPU edge device divides the initial feature tensor into N columns according to the configured number of blocks (N_block), generating N sub-feature tensors with the same shape, and annotates the block order. The GPU-enabled edge device loads model slices according to the preset layer granularity using nano-vllm to accelerate the embedding calculation. The edge device calls the ONNX format model through ONNXRuntime to perform embedding calculation. The edge device sends the initial feature tensor (non-pure CPU edge) or sub-feature tensor (pure CPU edge) along with KVCache control parameters, block parameters (pure CPU edge), nano-vllm splitting parameters (including GPU edge), ONNX configuration parameters (edge ​​device), language type identifier, and classification head distributed calculation parameters to the cloud server.

[0086] The relevant formulas for Embedding calculation are as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] in, Let i be the index of the i-th token. Embed a weight matrix for the token. Embed a vector for the token; Embed a vector to the position of the token in the sequence. Embed a weight matrix for the location; For token type identification, Embedded vectors of type, The initial feature tensor (dimension [1,4096]);

[0091] The pure CPU-side block partitioning process satisfies the following: the consistency error of the sub-feature tensor dimensions after partitioning is ≤1%; the partitioning only involves dimension splitting and does not change the tensor element values; and data integrity is ensured by CRC32 verification after partitioning, with the verification formula being: ;in, For the k-th sub-feature tensor, The number of elements in the subtensor;

[0092] The GPU-side nano-vllm model slice loading rule is as follows: the Embedding layer, speculative decoding module, full-layer LoRA adaptation layer and lightweight MLP of classification head are split to GPU memory according to the slice layer granularity, and the computing tasks are allocated by tensor parallel strategy, with a single slice computing latency of ≤5ms.

[0093] The ONNX model calling rule for edge devices is as follows: the hardware platform is automatically matched through the EP (ExecutionProvider) of ONNXRuntime, and the quantization results of the input preprocessing are reused when performing Embedding calculations, which reduces the calculation time by more than 30% compared with the native model.

[0094] c. Cloud Decoding and KV Cache Establishment / On-Demand Calling: After the cloud server receives the feature tensor (including sub-feature tensors) and control parameters, if caching is enabled (flag_cache=true), it calls the corresponding KVCache entry based on the specified cache length L_cache, and performs large model decoding layer inference by combining the feature tensor with the cloud decoding layer inference formula; if caching is not enabled, it directly decodes based on the feature tensor; it generates the hidden state before the final MLP as an intermediate output (the dimension is consistent with the initial feature tensor); in pure CPU edge scenarios, the cloud divides the hidden state into blocks according to the same block division rules and transmits them block by block in a streaming manner; GPU edge and edge devices directly transmit the complete hidden state and establish / update the corresponding session's key-value cache KVCache in the cloud.

[0095] The inference formula for the cloud decoding layer is as follows:

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] in, This refers to either the feature tensor from the previous stage (not purely CPU-based) or the concatenated complete feature tensor (when processing on the cloud from a purely CPU-based perspective). , , These are the query, key, and value projection weight matrices, respectively. The dimension of the key vector; This is a mask matrix used to mask future location information; This is the result of self-attention calculation. For feedforward network operations, , Here is the weight matrix of the feedforward network. , For bias terms, The output hiddenstate (dimension [1,4096]) after decoding.

[0102] Pure CPU-based edge-cloud block transmission satisfies the following requirements: push the sub-hiddenstate block by block according to the edge-side block sequence, return an acknowledgment flag after each block is transmitted, and support a retransmission mechanism. The retransmission is triggered when the receiving end does not return an acknowledgment flag within 100ms.

[0103] The cloud-based call process for KVCache is as follows: After receiving the control parameters from the client side, the cloud matches the corresponding KVCache using the session identifier sid. If flag_cache=true and L_cache≤the current total cache length, the first L_cache entries of the KVCache are extracted as historical context to aid decoding. If L_cache exceeds the current total cache length, the call is performed according to the actual cache length. If flag_cache=false, the existing KVCache is ignored, and a new KVCache is re-decoded based on the initial features and established. In a pure CPU client-side scenario, after receiving all sub-feature tensors, the cloud verifies them according to the block identifier to prevent order discrepancies. The concatenation formula is as follows: Then, KVCache matching and decoding operations are performed. In scenarios involving GPU-based edge devices, the cloud directly performs KVCache matching and decoding based on the complete feature tensor, without the need for additional splicing processing.

[0104] d-cloud incremental interactive reasoning

[0105] (1) In the incremental generation stage, the cloud server, based on the KVCache authorized by the end side, performs incremental decoding on the new token position of the current generation step through the cloud decoding layer inference formula and the cloud incremental decoding formula to obtain the hidden state (dimension [1, D_total]) before the input of the position to the final MLP. In the pure CPU end-side scenario, the cloud divides the hidden state into blocks according to the block rules and then streams the sub-hidden state. The GPU end-side and edge devices directly transmit the complete hidden state to the end-side device.

[0106] The incremental decoding formula in the cloud is as follows:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, This refers to either an incremental token feature tensor (not purely CPU-based) or a concatenated complete incremental feature tensor (when processing on the cloud from a purely CPU-based perspective). , For key-value tensors cached in the cloud, , This is a cache tensor truncated according to the end-side authorized length. This is the hidden state after incremental decoding (dimension [1,4096]). This is the cache length authorized on the client side.

[0113] The relevant formulas for distributed computing of the classification head are as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] in, , The input and output weight matrices of the edge-side lightweight MLP are [96, 4096] and [4096, 150000], respectively. , For end-side MLP bias terms; , The complete MLP input and output weight matrices in the cloud (dimensions are [4000, 4096] and [4096, 150000], respectively). , For cloud-based MLP bias terms;

[0118] Language adaptive distribution pruning rules: In Chinese monolingual scenarios, retain the dimensions corresponding to Chinese words in the vocabulary (approximately 40%, or 60,000 dimensions), and fill the dimensions of English words with -∞; in English monolingual scenarios, retain the dimensions corresponding to English words (approximately 30%, or 45,000 dimensions), and fill the dimensions of Chinese words with -∞; in mixed Chinese-English scenarios, retain the complete 150,000-dimensional distribution; the pruning formula is as follows:

[0119] $Logits_{clip}=\begin{cases};

[0120] Logits_{total}[i]&,\text{If the i-th item in the vocabulary is a word in the target language}\\;

[0121] -\infty&,\text{otherwise};

[0122] \end{cases}$;

[0123] ;

[0124] in, The probability distribution of the generated tokens;

[0125] Pure CPU-side hiddenstate calculation satisfies: receiving the k-th hiddenstate block After (dimension [D / N_block, 131200]), immediately load the subset of input weights corresponding to the full-layer LoRA fine-tuning model. Execute the calculation while asynchronously receiving the (k+1)th sub-hiddenstate. After all sub-blocks are calculated, they are concatenated in the block order to obtain Logits_end, which is then merged with the received trimmed Logits_cloud.

[0126] GPU-based edge computing satisfies the following: it can process the hidden state of multiple sessions in parallel through the batch processing capability of nano-vllm, and the speculative decoding prediction step size N can be configured to 4-6, reducing the number of interaction rounds by more than 40% compared with non-GPU edge computing.

[0127] Edge device computing satisfies the following: the full-layer LoRA adaptation layer of the ONNX model reduces inference latency through fusion computation graph, and the speculative decoding prediction step size N is configured as 1-2, balancing efficiency and device load.

[0128] (2) After receiving the hidden state (or sub-hidden state), the edge device first inputs it into the locally deployed speculative decoding module (the speculative decoding module is an MLP or Transformerlayer+MLP structure) and predicts the subsequent N tokens to be generated and their corresponding features in advance through speculative decoding related formulas; after receiving the sub-hidden state, the pure CPU edge device immediately starts local computation and receives the next sub-hidden state in parallel. After all sub-blocks are computed, the results are spliced ​​according to the merging rules; the GPU edge device accelerates speculative decoding and subsequent computation by loading model slices with nano-vllm; the edge device calls the model to perform related computations through ONNXRuntime;

[0129] The relevant formulas for speculative decoding are as follows:

[0130] ;

[0131] in, For speculative decoding of feature tensors, For speculative decoding encoding functions, if the speculative decoding module is an MLP structure, then If it is a Transformerlayer+MLP structure, then ; , , , For the weights and biases of the speculative decoding module in the MLP structure, , , This is the attention projection weight matrix for the speculative decoding module in the Transformerlayer+MLP structure.

[0132] ;

[0133] in, The output layer weight matrix for the speculative decoding module. For output layer bias terms, Output logits for speculative decoding.

[0134] ;

[0135] in, This is a token sequence predicted by speculative decoding.

[0136] (3) The concatenated complete hidden state is split according to the distributed computing parameters of the classification head. The first D_end dimension (D_end=96) features are extracted on the edge side and input into the local deployed [D_end,V] (V=150000) lightweight MLP. The edge local logits (Logits_end) are calculated by the cloud incremental decoding formula. The remaining D_cloud dimension (D_cloud=4000) features are extracted on the cloud and input into the cloud [D_cloud,V] complete MLP to calculate the cloud local logits (Logits_cloud). The Logits_cloud is then pruned according to the language type identifier.

[0137] The terminal receives the encrypted and quantized clipped Logits_cloud, adds it element-wise to the local Logits_end to obtain the complete Logits_total = Logits_end + Logits_cloud; based on the language type identifier, it fills the vocabulary positions of non-target languages ​​in Logits_total with negative infinity (-∞), and calculates the distribution probability P using the cloud incremental decoding formula; the incremental token, speculative decoding prediction result, and corresponding incremental features are sent to the cloud server together.

[0138] The cloud server first verifies the speculative decoding prediction result. If the verification passes, the prediction result is directly reused and the KVCache is updated. If the verification fails, incremental decoding is re-executed based on the incremental features on the client side to obtain the correct token and features, and then the KVCache is updated. Then, the next incremental decoding is executed according to the latest control parameters on the client side until the stopping condition is met.

[0139] e-side output: The edge device performs sampling based on the probability distribution P, decodes and post-processes the obtained token sequence, generates decoding results for users, and completes result display or downstream processing;

[0140] In this method, the KVCache is maintained by the cloud during inference, while the client controls the decision-making power regarding cache activation and usage scope through control parameters, without saving or synchronizing KVCache across devices. The cloud only transmits the hidden state in the form of feature tensors and the pruned logits throughout the process, without transmitting any complete distribution, plaintext data, or history_ids-like information. Furthermore, this method employs a dual encryption mechanism of "traditional encryption algorithm + hidden state protection": on the one hand, it uses AES series encryption algorithms to encrypt the transmitted feature tensors, speculative decoding prediction results, pruned logits, and other data, employing a lightweight AES-GCM algorithm to prevent theft and cracking during data transmission; on the other hand, it uses client-side... A hidden state protection barrier is constructed by deploying a fully LoRA-tuned model and a speculative decoding module. The parameters of the speculative decoding module are also optimized through LoRA fine-tuning and are only stored on the edge side, ensuring that the transmitted content cannot be reconstructed by enumerating the model or speculative decoding module parameters when the complete model weights are not obtained. At the same time, the edge-side speculative decoding module reduces the number of edge-cloud interaction rounds by predicting subsequent tokens to be generated in advance. The pure CPU edge side achieves latency hiding through block transmission and parallel computing. The GPU-containing edge side improves throughput through nano-vllm model segmentation and efficient inference capabilities. Edge devices achieve cross-device compatibility through the ONNX format. Distributed computing of the classification head reduces edge-side computing power consumption. Adaptive distributed compression of language reduces the amount of transmitted data and significantly improves collaborative inference efficiency.

[0141] The lightweight AES-GCM algorithm has the following encryption-related formulas:

[0142] ;

[0143] ;

[0144] ;

[0145] in, For quantization function, These are feature tensor elements (including sub-feature tensor elements) or clipped logits elements. The quantization bit width is adjustable (supports 4-bit / 8-bit / 16-bit adaptive adjustment). The data is after quantification; For pre-shared keys (128 / 256 bits). For a 12-byte random initialization vector, each sub-feature tensor and its corresponding sub-logits are assigned an independent IV and bound to a block identifier on the pure CPU side. For GPU-based and edge devices, independent IVs are assigned per session. For encrypted ciphertext, , These are the AES-GCM encryption and decryption functions, respectively.

[0146] Pure CPU-side decryption and computation coordination: After receiving the sub-feature tensor ciphertext and sub-logits ciphertext, decryption and dequantization operations are performed first, and model computation is started synchronously. The overlap rate between decryption time and computation time is ≥80%. The dequantization formula is as follows: ;

[0147] Edge device decryption and ONNX model collaboration: The decrypted feature tensors and logits are directly input into the ONNX model, skipping the secondary format conversion, and the overlap rate of decryption and model loading time is ≥70%.

[0148] Meanwhile, the model deployed on the edge meets the following conditions: For the open-source base model, full-layer LoRA fine-tuning is performed based on business scenario data before deployment. During the fine-tuning process, the backbone parameters of the base model's backbone network are fixed, and LoRA adapters are added to the Transformer layer, attention layer, MLP layer, adaptation layer, and lightweight classification head MLP, and the corresponding weights are updated. The full-layer LoRA fine-tuning weight update formula is as follows:

[0149] ;

[0150] ;

[0151] ;

[0152] in, Let A be a low-rank matrix (dimension [D,r]). Let B be a low-rank matrix (dimension [r,D]). For LoRA weight increment, The corresponding layer weights of the basic model, For loss function, As training samples, For tags, Learning rate (range of values) ), The gradient of the loss function. and These are the LoRA parameters before and after the update, respectively. For the model in training samples Forward propagation output, This indicates that the explicit gradient acts only on the A and B matrices of the LoRA adapter. This indicates that under the current LoRA weight configuration, the model's performance on training samples... The forward propagation prediction results.

[0153] The rank r configuration rule for full-layer LoRA fine-tuning is as follows: high computing power devices (including GPUs) use rank r = 8-16, low computing power devices and edge devices use rank r = 4-8, and pure CPU devices use rank r = 4.

[0154] The fine-tuned model weights are only stored on the client side and are not transmitted to the cloud or third parties; the self-developed closed-source model is directly deployed with its native full-layer model, and the transmission content is protected by keeping the model weights confidential.

[0155] Pure CPU-side model weight adaptation block calculation: The input and output weights of each layer of the full-layer LoRA fine-tuning model and the lightweight MLP weights of the classification head are split into N_block sub-weight matrices according to the number of blocks. The dimension of each sub-weight matrix matches the corresponding sub-feature tensor, and supports SIMD instruction set to accelerate matrix multiplication operations.

[0156] GPU-based model weight adaptation nano-vllm: After fusing the full-layer LoRA fine-tuning weights, the lightweight MLP weights of the classification head, and the basic model weights, it is split into slices with layered granularity to ensure that each model slice contains complete LoRA adaptation layer information and supports tensor parallel computation.

[0157] Edge device model weight adaptation to ONNX: Full-layer LoRA fine-tuning weights and lightweight MLP weights for classification heads are embedded into the ONNX model computation graph. Weight compression and computation graph optimization are enabled during export, reducing the model size by more than 60% compared to the native version.

[0158] In some embodiments, the latency hiding effect of pure CPU-side block transfer and computation satisfies: Let the transfer time of a single block feature tensor be... The computation time for a single block is Total time ,in The total time taken to transfer all blocks. Calculate the total time for all blocks to ensure that network latency is reduced to below 20%;

[0159] The GPU-enabled nano-vllm inference efficiency meets the following requirements: single-session inference latency is reduced by more than 50% compared to non-GPU-enabled devices, and multi-session parallel throughput of a single GPU device is increased by more than 3 times.

[0160] The ONNX model inference efficiency for edge devices meets the following requirements: inference latency ≤ 200ms, memory usage ≤ 30% of total device memory, and supports long-term continuous operation without memory leaks.

[0161] The distributed computing efficiency of the classification head meets the following requirements: the computational load on the edge is reduced to 2.34% of the original (96 / 4096), and the inference latency is reduced by more than 40% compared with the full edge computing.

[0162] The language-adaptive distributed compression achieves the following results: the amount of data transmitted in Chinese monolingual scenarios is reduced to 40% of the original amount, and in English monolingual scenarios it is reduced to 30%, with transmission latency reduced by more than 50% compared to fully distributed transmission.

[0163] In some embodiments, specific scenarios are exemplified as follows:

[0164] 1. High-privacy cross-device scenarios (industrial IoT data interaction, in-vehicle privacy dialogue)

[0165] On the client side, L_cache is dynamically decremented, reducing 5-10 cache entries per round of incremental inference;

[0166] The encryption has been upgraded to the AES-256-GCM algorithm, and the key derivation formula is as follows: ;in This is the final derived key. HKDF-SHA256 indicates an HMAC-based key derivation function that uses the SHA-256 hash function. It generates the key from three core inputs: Info is the initial master key (or input key material), Salt is a random salt value used to enhance security (making the derivation process nondeterministic and preventing rainbow table attacks), and Info is optional context information (used to bind the derived key to a specific protocol or application context).

[0167] Full-layer LoRA fine-tuning uses a higher rank r=16-24, and the feature tensor uses 16-bit asymmetric quantization, as shown in the formula. ;

[0168] Speculative decoding sets the prediction step size N=1 and enables secondary verification; the number of blocks on the pure CPU side is 2, and HMAC verification is added to each sub-block; the number of parallel sessions on the GPU side is 2, enhancing isolation; ONNX model encryption deployment is enabled on edge devices; the distributed computing of the classification head adopts dynamic dimension splitting, and the processing dimension on the edge can be configured to 48-128; language adaptive compression adds language filtering rules, supporting precise cropping of minor languages.

[0169] Extremely lightweight scenarios for low-computing-power edge devices (lightweight IoT gateways, legacy vehicle terminals):

[0170] The full-layer LoRA fine-tuning model uses parameter pruning (pruning rate ≤60%) + 4-bit quantization, reducing the model size by more than 85%.

[0171] The speculative decoding module is integrated with the MLP layer and the lightweight MLP computation process of the classification head, and the formula is as follows: ;in It is the hidden state of the output. It is the Gaussian error linear unit activation function. These are the hidden layer features of the input; the weights within the parentheses are task-specific weights. Basic multilayer perceptron weights and fine-tuning the end weights The sum of the three components constitutes the total, while the corresponding bias component is composed of a specific bias. Basic bias and end bias The sum of the three is required to form the composition.

[0172] ONNX model export uses extreme lightweight optimization, removing redundant operators, retaining the core computation graph, and supporting memory-mapped loading; the lightweight classification head MLP uses 8-bit quantization, reducing weight size by 75%; language adaptive compression uses 4-bit quantization by default, further reducing data transmission volume by 50%.

[0173] 2. Multi-tenant, multi-device cloud scenario:

[0174] The cloud allocates independent KVCache space, speculative decoding verification resources, classification header computing resources, and transmission bandwidth to each tenant and device type;

[0175] A three-layer isolation system of hardware, software and network is adopted. Model slices are allocated independently to GPU-side tenants, ONNX execution environments are allocated independently to edge device tenants, and block transmission buffers are allocated independently to pure CPU-side tenants.

[0176] Optimize data transmission strategies based on the communication characteristics of different device types to prevent cross-access of cached data, speculative decoding data, and logits data; maintain an independent language type configuration table for each tenant and support tenant-defined language trimming rules.

[0177] 3. Multi-device scenarios with ultra-large-scale feature tensors (feature tensor dimensions [1, 10240]):

[0178] GPU edge: nano-vllm is split into layers with granular attention head level, multi-session parallelism = 8, and multi-level cache scheduling is adopted; the classification head is distributed computing to dynamically adjust the dimension splitting ratio, edge processing dimension = 256, cloud processing dimension = 10240-256=9984.

[0179] On the pure CPU side: the number of blocks is dynamically adjusted to 6-8, and multi-level block scheduling is adopted. The first two blocks are transmitted to start the calculation, and subsequent blocks are pushed asynchronously according to the calculation progress.

[0180] Edge devices: The ONNX model adopts a block-based inference strategy to reduce the memory usage of a single inference operation;

[0181] In the cloud: multi-card segmented decoding is used for ultra-large-scale feature tensors, and the decoding results are adapted to the transmission format according to the device type; logits are truncated and transmitted in blocks, supporting breakpoint resume.

[0182] The implementation principle of this application's large-scale model edge-cloud collaborative inference method, which balances efficiency and privacy, is as follows: By implementing edge-cloud functional layering and access control, it addresses the problems of high privacy leakage risk, low collaborative efficiency, and poor multi-device compatibility in traditional solutions. The technical solution adopts an architecture combining lightweight edge processing and heavy cloud computing: the edge is responsible for input preprocessing, embedding calculation, speculative decoding, and low-dimensional feature calculation of the classification head; the cloud performs complete decoding layer inference, KV cache management, and high-dimensional feature calculation of the classification head. In terms of security, a dual encryption system of "traditional encryption algorithm + hidden state protection" is constructed. Transmitted data is encrypted using AES-GCM, and a full-layer LoRA fine-tuning model is deployed on the edge. Fine-tuning weights are only stored on the edge to prevent the hidden state from being enumerated and restored. The edge controls cache access permissions through KV cache control parameters to avoid the plaintext transmission of historical identifiers. In terms of efficiency, edge-side speculative decoding predicts subsequent tokens in advance, reducing interaction rounds; pure CPU edge-side uses block transmission and parallel computing to hide network latency; GPU edge-side improves throughput through nano-VLLM model segmentation; edge devices export ONNX format to achieve cross-platform compatibility; classification headers are split and calculated in 96-dimensional and 4000-dimensional formats, reducing edge-side computing power consumption; language adaptive pruning reduces the amount of data transmitted. Thus, under the same bandwidth conditions, throughput exceeds that of the native VLLM engine, edge-cloud collaboration latency is significantly reduced, and it supports full-scenario adaptation for pure CPU, GPU, and edge devices, achieving dual protection of efficiency and privacy.

[0183] Figure 1 This is a flowchart illustrating a large-scale edge-cloud collaborative inference method that balances efficiency and privacy in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0184] Based on the same technical concept, referring to Figure 2 This application also provides a large-scale model edge-cloud collaborative inference device that balances efficiency and privacy, employing the following technical solution: The device includes:

[0185] The data input module is used to preprocess user input data acquired from the edge device, generate an input representation, and configure inference control parameters. The inference control parameters include KVCache control parameters, which are used to indicate whether the cloud server enables and how long of the historical KVCache is used for this inference. At the same time, the control edge device configures device adaptation parameters according to its own hardware type. The device adaptation parameters include at least one of the following: block transfer parameters for pure CPU edge devices, model splitting parameters for GPU edge devices, and ONNX runtime parameters for edge devices.

[0186] The feature calculation module is used to perform embedding calculations based on the input representation to generate an initial feature tensor; adapt the initial feature tensor according to the device adaptation parameters, and upload the processed feature tensor, KVCache control parameters and language type identifier to the cloud;

[0187] The cloud receiving module is used to receive data based on the cloud server, and authorize the invocation or reconstruction of KVCache according to the KVCache control parameters. It combines the received feature tensor to perform decoding layer inference, generate the first hidden state, and return the first hidden state to the end device.

[0188] The data inference module is used in the current token generation step. The edge device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates the edge partial inference result and the speculative decoding prediction result. The edge uploads the incremental token and the speculative decoding prediction result to the cloud. The speculative decoding prediction result is verified and the KVCache is updated. At the same time, based on the high-dimensional part of the current hidden state, cloud partial inference is performed to generate the cloud partial inference result. The cloud sends back the cloud partial inference result after pruning and encryption. The edge merges the edge partial inference result and the cloud partial inference result to generate the complete inference result of the current step and outputs the incremental token.

[0189] The data decoding module is used to control the terminal device to decode and post-process the final generated token sequence based on the complete inference result and the incremental token, and output the inference result.

[0190] In some embodiments, the data input module is specifically used to automatically determine the language type based on the proportion of Chinese and English in the input prompt words, or to determine it based on the user's language preferences, including Chinese monolingual, English monolingual, or mixed Chinese and English.

[0191] KVCache control parameters include cache enable flag and cache usage length. Cache is disabled by default for new sessions and enabled by default for non-new sessions. The cache usage length is dynamically adjusted by the end device based on the session context, and its value ranges from 1 to the current total length of KVCache.

[0192] The device adaptation parameters include: For pure CPU-side devices, the block transmission parameters include the number of blocks, the block order identifier and the merging rules. The number of blocks is adaptively configured according to the feature tensor dimension and the CPU computing power, and the dimension of each sub-feature tensor is consistent.

[0193] For GPU-based edge devices, the model splitting parameters include the splitting granularity and the number of sessions in parallel processing per device. The splitting granularity supports splitting by Transformer level or attention head level, and the number of sessions in parallel processing is configured according to the GPU memory capacity.

[0194] For edge devices, ONNX runtime parameters include quantization bit width and hardware acceleration instruction set call configuration.

[0195] In some embodiments, the feature computation module is specifically used to perform embedding computation based on the input representation to generate an initial feature tensor. The embedding computation includes a weighted summation of word embeddings, position embeddings, and type embeddings.

[0196] For pure CPU-based devices, the initial feature tensor is divided into multiple sub-feature tensors by columns, and verification information is added to each sub-feature tensor to ensure data integrity.

[0197] For GPU-side devices, the model slices are loaded at a preset layer granularity using the nano-vllm engine, and a tensor parallel strategy is used to accelerate the embedding computation.

[0198] For edge devices, embedded computation is performed by calling the hardware acceleration instruction set through ONNXRuntime, and the quantization results in the input preprocessing are reused;

[0199] The processed feature tensor, KVCache control parameters, and language type identifier are uploaded to the cloud.

[0200] In some embodiments, the cloud receiving module is specifically used to receive data based on the cloud server, and authorize the invocation or reconstruction of KVCache according to the KVCache control parameters, and convert the feature tensor into the first hidden state through self-attention mechanism and feedforward network operation;

[0201] When the first hidden state is returned to the edge device via the cloud server, for pure CPU edge devices, the first hidden state is split into multiple sub-hidden states according to the block parameters uploaded by the edge device, and then streamed to the edge device in sequence, while supporting a retransmission mechanism; for GPU edge devices and edge devices, the complete first hidden state is returned directly.

[0202] In some embodiments, the data inference module is specifically used to extract the low-dimensional part of the current first hidden state based on the edge device according to the preset dimensional splitting rules, input it into the locally deployed lightweight MLP computation, and obtain the edge local inference result.

[0203] The cloud server extracts the high-dimensional part of the current first hidden state according to the preset dimension splitting rules, inputs it into the complete MLP computation deployed in the cloud, and obtains the cloud local inference result;

[0204] The complete inference result is obtained by adding the local inference result on the edge and the local inference result on the cloud element by element. The vocabulary dimension corresponding to the non-target language in the complete inference result is masked according to the language type identifier. The sampling probability is then obtained by the Softmax function, and an incremental token is generated based on the sampling probability.

[0205] Based on the current first hidden state, a speculative decoding prediction result is generated through the speculative decoding module deployed on the edge. The speculative decoding module is an MLP structure or a Transformer layer plus an MLP structure, and its prediction step size is dynamically configured according to the type of edge device.

[0206] The client uploads the incremental tokens and speculative decoding prediction results to the cloud.

[0207] The cloud server determines the consistency between the predicted token and the actual decoded token. If the verification is successful, the prediction result is reused directly, the decoding calculation is skipped, and the KVCache is updated.

[0208] In some embodiments, the data inference module is further configured to implement a lightweight computing module deployed locally on the edge device based on a model fine-tuned with full-layer LoRA, wherein full-layer LoRA fine-tuning includes:

[0209] A low-rank adapter is added to each layer of the base model. During fine-tuning, only the weights of the low-rank adapter are updated, while the backbone parameters of the base model are frozen. The weights of the low-rank adapter after fine-tuning are only stored on the edge device and are not transmitted to the cloud.

[0210] The rank is finely tuned based on the type of edge device, with higher rank used for GPU edge devices and lower rank used for edge devices and pure CPU edge devices.

[0211] For pure CPU-based devices, the fine-tuned weights are split into multiple sub-weight matrices to adapt to block-based computation.

[0212] For GPU-based devices, the fine-tuned weights are fused with the base model weights and then split into layers with granularity.

[0213] For edge devices, the fine-tuned weights are embedded into the ONNX model computation graph and quantized and compressed during export.

[0214] In some embodiments, the data inference module is further configured to perform quantization processing on the data to be uploaded and transmitted on the end device, converting the floating-point number into an integer with a specified bit width. The data to be transmitted includes the processed feature tensor, the incremental token, and the feature data corresponding to the speculative decoding prediction result.

[0215] The quantized data is encrypted using AES-GCM with a pre-shared key and a random initialization vector, and the ciphertext is then uploaded to the cloud.

[0216] The cloud-based partial inference results are subjected to the same quantization and encryption processing on the cloud server; at the receiving end, the same key and initialization vector are used for decryption and dequantization to restore the original data.

[0217] For pure CPU-based edge devices, each sub-feature tensor and its corresponding return data are assigned an independent initialization vector and bound to a block identifier; for GPU-based edge devices and edge devices, an independent initialization vector is assigned per session; the decryption and computation processes are performed in tandem, so that the decryption time overlaps with the computation time, thereby reducing latency.

[0218] This application also discloses a control device.

[0219] Specifically, the control device includes a memory and a processor, the memory storing a computer program that can be loaded by the processor and executed to implement the aforementioned large-scale model edge-cloud collaborative inference method that balances efficiency and privacy.

[0220] This application also discloses a computer-readable storage medium.

[0221] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the large-scale cloud-edge collaborative inference method that balances efficiency and privacy as described above. The computer-readable storage medium includes, for example, various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0222] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A large-scale edge-cloud collaborative inference method that balances efficiency and privacy, characterized in that, include: The system acquires user input data from the edge device, performs preprocessing to generate an input representation, and configures inference control parameters. These inference control parameters include KVCache control parameters, which instruct the cloud server whether to enable and for how long to use the historical KVCache for this inference. Simultaneously, the control edge device configures device adaptation parameters according to its own hardware type. These device adaptation parameters include at least one of the following: block transfer parameters for pure CPU edge devices, model splitting parameters for GPU edge devices, and ONNX runtime parameters for edge devices. Based on the input representation, embedding computation is performed to generate an initial feature tensor; The initial feature tensor is adapted according to the device adaptation parameters, and the processed feature tensor, KVCache control parameters and language type identifier are uploaded to the cloud. Based on the data received from the cloud server, and according to the KVCache control parameters, authorize the invocation or reconstruction of KVCache, combine the received feature tensor to perform decoding layer inference, generate the first hidden state, and return the first hidden state to the end device; In the current token generation step, the edge device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates an edge-side partial inference result and a speculative decoding prediction result; the edge uploads the incremental token and the speculative decoding prediction result to the cloud; the speculative decoding prediction result is verified and the KVCache is updated, and at the same time, cloud-based partial inference is performed based on the high-dimensional part of the current hidden state to generate a cloud-based partial inference result; the cloud sends back the cloud-based partial inference result after pruning and encryption; the edge merges the edge-side partial inference result and the cloud-based partial inference result to generate the complete inference result of the current step and outputs the incremental token; Based on the complete inference result and the incremental token, the control-side device decodes and post-processes the finally generated token sequence and outputs the inference result; In the current token generation step, the edge device receives the first hidden state returned by the cloud, performs partial inference based on the locally deployed lightweight computing module, and generates edge-side partial inference results and speculative decoding prediction results; the edge uploads the incremental token and speculative decoding prediction results to the cloud; the speculative decoding prediction results are verified and the KVCache is updated, including: Based on the edge device, the low-dimensional part of the current first hidden state is extracted according to the preset dimensional splitting rules, and input into the locally deployed lightweight MLP computation to obtain the edge local inference result. The cloud server extracts the high-dimensional part of the current first hidden state according to the preset dimension splitting rules, inputs it into the complete MLP computation deployed in the cloud, and obtains the cloud local inference result; The complete inference result is obtained by adding the local inference result on the edge and the local inference result on the cloud element by element. The vocabulary dimension corresponding to the non-target language in the complete inference result is masked according to the language type identifier. The sampling probability is then obtained by the Softmax function, and an incremental token is generated based on the sampling probability. Based on the current first hidden state, the speculative decoding prediction result is generated by the speculative decoding module deployed on the edge. The speculative decoding module is an MLP structure or a Transformer layer plus an MLP structure, and its prediction step size is dynamically configured according to the type of edge device. The client uploads the incremental token and the speculative decoding prediction result to the cloud. The cloud server determines the consistency between the predicted token and the actual decoded token. If the verification is successful, the prediction result is reused directly, the decoding calculation is skipped, and the KVCache is updated.

2. The large-scale edge-cloud collaborative inference method for balancing efficiency and privacy as described in claim 1, characterized in that, After the user input data acquired from the edge device is preprocessed, an input representation is generated, and inference control parameters are configured, the method further includes: The language type identifier is automatically determined based on the ratio of Chinese to English in the input prompts, or determined based on the user's language preferences, including Chinese monolingual, English monolingual, or mixed Chinese and English; KVCache control parameters include cache enable flag and cache usage length. Cache is disabled by default for new sessions and enabled by default for non-new sessions. The cache usage length is dynamically adjusted by the end device based on the session context, and its value ranges from 1 to the current total length of KVCache. The device adaptation parameters include: For pure CPU-side devices, the block transmission parameters include the number of blocks, the block order identifier and the merging rules. The number of blocks is adaptively configured according to the feature tensor dimension and the CPU computing power, and the dimension of each sub-feature tensor is consistent. For GPU-based edge devices, the model splitting parameters include the splitting granularity and the number of sessions in parallel processing per device. The splitting granularity supports splitting by Transformer level or attention head level, and the number of sessions in parallel processing is configured according to the GPU memory capacity. For edge devices, ONNX runtime parameters include quantization bit width and hardware acceleration instruction set call configuration.

3. The large-scale edge-cloud collaborative inference method for balancing efficiency and privacy as described in claim 1, characterized in that, The process involves performing embedding computation based on the input representation to generate an initial feature tensor; adapting the initial feature tensor according to device adaptation parameters; and uploading the processed feature tensor, KVCache control parameters, and language type identifier to the cloud, including: Embedding computation is performed based on the input representation to generate the initial feature tensor. The embedding computation includes a weighted summation of word embeddings, position embeddings, and type embeddings. For pure CPU-side devices, the initial feature tensor is divided into multiple sub-feature tensors by columns, and verification information is added to each sub-feature tensor to ensure data integrity. For GPU-side devices, the model slices are loaded at a preset layer granularity using the nano-vllm engine, and a tensor parallel strategy is used to accelerate the embedding computation. For edge devices, embedded computation is performed by calling the hardware acceleration instruction set through ONNXRuntime, and the quantization results in the input preprocessing are reused; The processed feature tensor, KVCache control parameters, and language type identifier are uploaded to the cloud.

4. The large-scale edge-cloud collaborative inference method for balancing efficiency and privacy as described in claim 1, characterized in that, The process of receiving data from a cloud server, authorizing the invocation or reconstruction of the KVCache according to KVCache control parameters, performing decoding layer inference in conjunction with the received feature tensor, generating a first hidden state, and returning the first hidden state to the end device includes: Based on receiving data from the cloud server and authorizing the invocation or reconstruction of KVCache according to KVCache control parameters, the feature tensor is converted into the first hidden state through self-attention mechanism and feedforward network operation. When the first hidden state is returned to the edge device via the cloud server, for a pure CPU edge device, the first hidden state is split into multiple sub-hidden states according to the block parameters uploaded by the edge device, and then transmitted sequentially to the edge device in a streaming manner, while supporting a retransmission mechanism; for GPU edge devices and edge devices, the complete first hidden state is returned directly.

5. The large-scale edge-cloud collaborative inference method for balancing efficiency and privacy as described in claim 1, characterized in that, The method further includes: The lightweight computing module deployed locally on the edge device is implemented based on a model that has undergone full-layer LoRA fine-tuning. Full-layer LoRA fine-tuning includes: A low-rank adapter is added to each layer of the base model. During fine-tuning, only the weights of the low-rank adapter are updated, while the backbone parameters of the base model are frozen. The weights of the low-rank adapter after fine-tuning are only stored on the edge device and are not transmitted to the cloud. The rank is finely tuned based on the type of edge device, with higher rank used for GPU edge devices and lower rank used for edge devices and pure CPU edge devices. For pure CPU-based devices, the fine-tuned weights are split into multiple sub-weight matrices to adapt to block-based computation. For GPU-based devices, the fine-tuned weights are fused with the base model weights and then split into layers with granularity. For edge devices, the fine-tuned weights are embedded into the ONNX model computation graph and quantized and compressed during export.

6. The large-scale edge-cloud collaborative inference method for balancing efficiency and privacy as described in claim 1, characterized in that, The cloud server transmits the partially inference results back to the endpoint after pruning and encrypting them, including: The end-side device performs quantization processing on the data to be uploaded and transmitted, converting floating-point numbers into integers of a specified bit width. The data to be transmitted includes the processed feature tensor, the incremental token, and the feature data corresponding to the speculative decoding prediction result. The quantized data is encrypted using AES-GCM with a pre-shared key and a random initialization vector, and the ciphertext is then uploaded to the cloud. The cloud-based partial inference results are subjected to the same quantization and encryption processing on the cloud server; at the receiving end, the same key and initialization vector are used for decryption and dequantization to restore the original data. For pure CPU-based edge devices, each sub-feature tensor and its corresponding return data are assigned an independent initialization vector and bound to a block identifier; for GPU-based edge devices and edge devices, an independent initialization vector is assigned per session; the decryption and computation processes are performed in tandem, so that the decryption time overlaps with the computation time, thereby reducing latency.

7. A large-scale edge-cloud collaborative inference device that balances efficiency and privacy, characterized in that, The device includes: The data input module is used to preprocess user input data acquired from the edge device, generate an input representation, and configure inference control parameters. The inference control parameters include KVCache control parameters, which are used to indicate whether the cloud server enables and how long of the historical KVCache is used for this inference. At the same time, the control edge device configures device adaptation parameters according to its own hardware type. The device adaptation parameters include at least one of the following: block transfer parameters for pure CPU edge devices, model splitting parameters for GPU edge devices, and ONNX runtime parameters for edge devices. The feature calculation module is used to perform embedding calculations based on the input representation to generate an initial feature tensor; to perform adaptation processing on the initial feature tensor according to the device adaptation parameters; and to upload the processed feature tensor, KVCache control parameters and language type identifier to the cloud. The cloud receiving module is used to receive data based on the cloud server, authorize the invocation or reconstruction of KVCache according to the KVCache control parameters, perform decoding layer inference in combination with the received feature tensor, generate the first hidden state, and return the first hidden state to the end device. The data inference module is used in the current token generation step. The edge device receives the first hidden state returned from the cloud, performs partial inference based on a locally deployed lightweight computing module, and generates an edge-side partial inference result and a speculative decoding prediction result. The edge device uploads the incremental token and the speculative decoding prediction result to the cloud. The speculative decoding prediction result is verified and the KVCache is updated. Simultaneously, based on the high-dimensional part of the current hidden state, cloud-based partial inference is performed to generate a cloud-based partial inference result. The cloud-based partial inference result is pruned, encrypted, and then sent back to the edge device. The edge device merges the edge-side partial inference result and the cloud-based partial inference result to generate the complete inference result for the current step and outputs the incremental token. The data decoding module is used to control the terminal device to decode and post-process the final generated token sequence based on the complete inference result and the incremental token, and output the inference result. Specifically, the data inference module is used to extract the low-dimensional part of the current first hidden state based on the edge device according to the preset dimensional splitting rules, input it into the locally deployed lightweight MLP computing, and obtain the edge local inference result. The cloud server extracts the high-dimensional part of the current first hidden state according to the preset dimension splitting rules, inputs it into the complete MLP computation deployed in the cloud, and obtains the cloud local inference result; The complete inference result is obtained by adding the local inference result on the edge and the local inference result on the cloud element by element. The vocabulary dimension corresponding to the non-target language in the complete inference result is masked according to the language type identifier. The sampling probability is then obtained by the Softmax function, and an incremental token is generated based on the sampling probability. Based on the current first hidden state, the speculative decoding prediction result is generated by the speculative decoding module deployed on the edge. The speculative decoding module is an MLP structure or a Transformer layer plus an MLP structure, and its prediction step size is dynamically configured according to the type of edge device. The client uploads the incremental token and the speculative decoding prediction result to the cloud. The cloud server determines the consistency between the predicted token and the actual decoded token. If the verification is successful, the prediction result is reused directly, the decoding calculation is skipped, and the KVCache is updated.

8. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 6.

Images