Calculation power network-oriented large and small model cooperative reasoning performance optimization method and device

Abstract

The invention discloses a computing power network-oriented big and small model cooperative reasoning performance optimization method and device. Through an innovative cooperation mechanism, the problems of semantic inconsistency, high communication overhead and the like caused by model difference in the prior art can be effectively solved. According to the method, on the premise that the output quality of a large model is remarkably maintained, the SLO achievement rate of a system request is greatly improved, and the overall response delay is optimized. Meanwhile, the method remarkably improves the comprehensive utilization efficiency of cloud-edge-end computing power resources in a heterogeneous computing environment, and reduces the overall computing and communication overhead. Through efficient collaboration, the system can realize higher throughput and energy efficiency ratio, and a high-performance and low-delay reliable solution is provided for deploying large and small model collaboration services in a computing power network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and specifically to a method and apparatus for optimizing the performance of collaborative reasoning between large and small models in computing networks. Background Technology

[0002] With the significant performance improvements of large language models (LMs) in natural language understanding and generation tasks, their parameter scale has expanded dramatically, leading to bottlenecks in the inference process such as high computational overhead, high latency, and high resource consumption. Especially with the increasing prevalence of computing network architectures, computing resources are heterogeneously distributed across the cloud, edge, and endpoint. Traditional single large model deployment schemes struggle to adapt to diverse node capabilities and cannot simultaneously meet the SLO (Solution-Like Response) targets of low latency and high throughput. In contrast, small language models (SLMs), while having performance limitations, offer advantages such as low computational complexity, low resource consumption, and short inference latency. Therefore, academia and industry are gradually exploring the Large-Small Model Collaborative Inference architecture, advocating for scheduling large models to core high-computing-power nodes and deploying small models to edge low-computing-power nodes. The aim is to achieve a balance between performance and efficiency through the dual complementarity of model capabilities and computing resources.

[0003] Existing collaborative systems for large and small models generally attempt to leverage the complementary advantages of models of different sizes to achieve resource conservation and faster response without significantly reducing output quality. The core idea is to have smaller models handle some inference tasks, while larger models process complex samples or regions with high uncertainty. Through this division of labor, the system as a whole can achieve higher throughput and energy efficiency than a single large model. However, existing research also has significant limitations. Because models of different sizes or families differ significantly in latent space distribution, parameter structure, and attention representation, directly passing intermediate features or key-value caches between models often leads to semantic shifts, contextual inconsistencies, or information loss. Therefore, most studies choose to allow models to communicate semantically only through output text—that is, the output of one model serves as the input of another. While this "text-level" collaboration ensures compatibility, it sacrifices rich information transmission at the internal semantic level and introduces additional generation latency and parsing overhead, thus reducing overall inference efficiency. Furthermore, in determining the timing of collaboration, existing systems typically rely on input complexity estimation, scene feature analysis, or uncertainty metrics to dynamically decide whether to call the large model. While such mechanisms offer a degree of flexibility, they typically introduce additional computational and control overhead and may cause system performance fluctuations due to judgment biases.

[0004] In summary, how to maintain semantic consistency between models while achieving a low-overhead, low-latency, and efficient collaboration mechanism between large and small models in a heterogeneous computing network environment is a core problem that current research urgently needs to solve. Summary of the Invention

[0005] This invention addresses the issues of language fragmentation and communication overhead in existing large-small model collaborative reasoning techniques, and discloses a method and apparatus for optimizing the performance of large-small model collaborative reasoning in computing networks. This method decouples the request reasoning process into two stages: pre-filling and decoding, which are executed collaboratively by the small and large models respectively. Furthermore, it achieves seamless migration of key-value caches between models through a trained projector, thereby significantly improving request SLO achievement rate and resource utilization while ensuring output quality. Specifically, this invention includes the following technical steps: A method for optimizing the collaborative inference performance of large and small models in computing power networks, the method comprising: S1. Based on the length of the input sequence of several arriving inference requests, a predefined scheduling strategy is used to select requests to be processed from the inference requests, and the small model performs block pre-filling processing on the input sequence of the selected requests to generate the corresponding key-value cache. S2. Input the key-value cache generated by the small model into the pre-trained projector. The projector maps the key-value cache from the semantic representation space of the small model to the semantic representation space of the large model to obtain the projected key-value cache. S3. Input the projected key-value cache into the large model, and the large model performs autoregressive decoding based on the projected key-value cache to generate the final output of the inference request.

[0006] Preferably, S1 includes: Based on the length of the input sequence of the arriving inference requests, the inference requests are sorted using a short sequence priority strategy, and the requests to be processed are selected according to the sorting results. Based on the maximum number of tokens processed in a single iteration of the small model, the input sequence of the selected request is divided into one or more consecutive blocks; The small model performs a forward propagation operation on the partitioned blocks to generate the corresponding key-value cache.

[0007] Preferably, S2 includes: The key-value cache generated by the small model is expanded and reorganized to obtain the first feature representation; The first feature representation is compressed using high-dimensional features to obtain the compressed feature representation. The compressed feature representation is subjected to nonlinear transformation and semantic enhancement to obtain an enhanced semantic feature representation. The enhanced semantic feature representation is dimensionally aligned and reshaped to obtain a projected key-value cache with the same shape as the key-value cache required by the large model.

[0008] Preferably, the projector is trained using a joint loss function that includes projection loss, prediction loss, and regularization loss; wherein, the projection loss is calculated based on the center kernel alignment index between the projected key-value cache and the target key-value cache, the prediction loss is calculated based on the cross-entropy loss, and the regularization loss is calculated based on L2 regularization.

[0009] Preferably, the step of calculating the projection loss based on the center kernel alignment index includes: Expand the projected key-value cache and the target key-value cache into standard matrix shapes respectively; Calculate the kernel matrices of the projected key-value cache and the target key-value cache respectively; The kernel matrix is ​​centered. The similarity between the center kernel alignment of the projected key-value cache and the target key-value cache is calculated based on the centered kernel matrix, and the projection loss is determined based on the similarity.

[0010] Preferably, S3 includes: Receive the projected key-value cache and store it in the corresponding asynchronous buffer queue according to its corresponding inference request identifier; When it is detected that all input sequence blocks of a certain inference request have been pre-filled and projected, retrieve all projected key-value caches corresponding to the request from the asynchronous buffer queue; The extracted projected key values ​​are cached and input into the large model, triggering the large model to perform autoregressive decoding to generate the output sequence.

[0011] Preferably, an asynchronous transmission mechanism is adopted, so that the key-value cache generated after the small model completes a single round of forward calculation can be immediately sent to the projector for processing, without waiting for all requests or batch calculations to be completed.

[0012] The present invention also provides a performance optimization device for collaborative inference of large and small models for computing power networks. The device is used to implement the above method and includes: a processing module, an input module and an output module. The processing module is used to select a request to be processed from the inference requests according to the length of the input sequence of the incoming inference requests using a predefined scheduling strategy, and the small model performs block pre-filling processing on the input sequence of the selected request to generate the corresponding key-value cache. The input module is used to input the key-value cache generated by the small model into the pre-trained projector. The projector maps the key-value cache from the semantic representation space of the small model to the semantic representation space of the large model to obtain the projected key-value cache. The output module is used to input the projected key-value cache into the large model, and the large model performs autoregressive decoding based on the projected key-value cache to generate the final output of the inference request.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention significantly improves the overall performance of collaborative inference systems with large and small models. While maintaining output quality comparable to the large model, it substantially increases system throughput and SLO (Solution-Time Response) achievement rate, effectively reducing end-to-end inference latency. Simultaneously, this method significantly optimizes the global utilization of heterogeneous computing resources, reduces the additional computational and communication overhead caused by inter-model collaboration, and achieves better energy efficiency. Ultimately, it provides an efficient and reliable solution for building high-performance, low-latency intelligent services in a computing network environment. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the model structure according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 like Figure 1 The diagram shown is a schematic representation of the method flow in this embodiment, and the steps include: S1. Based on the length of the input sequence of several arriving inference requests, a predefined scheduling strategy is used to select requests to be processed from the inference requests, and the small model performs block pre-filling processing on the input sequence of the selected requests to generate the corresponding key-value cache.

[0019] like Figure 2The diagram shown illustrates the model structure constructed in this embodiment. The scheduler selects requests for pre-filled block processing based on a short-sequence-first scheduling strategy, and the smaller model pre-fills blocks based on the maximum number of tokens processed in a single iteration. The specific implementation details of this step are as follows: (1) When a new request q Arrival in waiting queue In this case, the method employs a short-sequence-first scheduling strategy, sorting requests from shortest to longest based on the length of the input sequence. This prioritizes processing short-sequence requests within limited computing resources, thereby shortening the average response time and improving the overall system throughput. Specifically, if a new request... q The input length is Lq Then its insertion position in the waiting queue satisfies: .

[0020] That is, when the system inserts a new request, it will place it between requests with adjacent input lengths to achieve orderly updates of the dynamic scheduling queue.

[0021] (2) When the small model performs inference, define the maximum number of tokens (basic units) that the small model can process in a single iteration. A The number of tokens contained in a single block a and requests qj Length of the input token sequence .

[0022] Before entering the small model, the input sequence for each request is divided into multiple consecutive sub-blocks according to the block size: , in, Represents a sequence j Subscript Kj The number of tokens contained in the block.

[0023] Each block corresponds to one independent pre-filling operation. In a single iteration, a small model can process a maximum of no more than [number missing] blocks simultaneously. A There are Tokens, therefore the following condition is met: , in, Indicates the first i Requests are allocated during round-robin scheduling. qj The number of tokens; j Indicates the index of the request in the current scheduling set; m This represents the total number of all scheduled requests in the current scheduling set.

[0024] Based on the above definitions, the effective token processing set of the small model in a single iteration can be represented as: .

[0025] Through this "scheduling-blocking-parallelism" mechanism, the model can fully utilize the computing power of small models in a single iteration while ensuring fairness among all requests, thereby improving overall resource utilization and throughput efficiency.

[0026] S2. Input the key-value cache generated by the small model into the pre-trained projector. The projector maps the key-value cache from the semantic representation space of the small model to the semantic representation space of the large model, thus obtaining the projected key-value cache.

[0027] After S1-based scheduling and pre-filled blocks, the small model performs forward propagation on the selected blocks. After each forward propagation, the small model generates a corresponding key-value (KV) cache to store intermediate states from attention calculations. To enable the large model to directly utilize these KV caches generated by the small model for subsequent decoding, a projector is needed to perform semantic mapping and dimension alignment. This projection process achieves semantic alignment and compatibility between the source and target models in the KV cache representation space. The specific implementation details are as follows: (1) Projector Structure Design and Mapping Process. To achieve semantic and structural alignment of the KV cache, this embodiment designs and pre-trains a projector composed of a multilayer perceptron. During the training phase, the projector uses the KV cache generated by the source model as input and the KV cache generated by the target model as the alignment supervision signal. The source model KV cache and the target model KV cache are defined as follows: , in, These represent the batch size, the number of attention heads in the source and target models, the sequence length, and the model dimensions of the source and target models, respectively.

[0028] (2) The entire projection process is as follows: ① Dimensional Unfolding and Reorganization: First, the head dimension and feature dimensions of the source model are transposed and merged to unify the feature space representation. , in, This represents the KV cache after preprocessing the source KV cache; Indicates to Csrc Perform dimensional expansion and reorganization operations.

[0029] This step achieves the fusion of features from different attention heads, providing more complete contextual features for subsequent nonlinear transformations.

[0030] ② High-dimensional feature compression: Compress the flattened high-dimensional features to a unified hidden dimension. Extract key semantic information: , in, Indicates to Perform the first layer of MLP projection operation; .

[0031] ③ Semantic Enhancement and Nonlinear Transformation: Next, multiple hidden layers are stacked, and nonlinear activation functions and residual connection structures are combined to semantically enhance the compressed features. After multiple layers of mapping, a semantically enhanced feature representation is obtained. .

[0032] ④ Dimension Alignment and Reshaping: Finally, through linear mapping and dimension reshaping operations, the enhanced semantic representation is mapped back to the KV cache shape required by the target model: , Here, Linear represents a linear mapping.

[0033] Thus, the projector has achieved semantic mapping and structural alignment of the key-value cache from the source model to the target model. Through this projection process, the projector can learn how to transform the source model's key-value cache into the target model's semantic space, allowing the target model to directly utilize the projection results for decoding in subsequent inference.

[0034] (3) Quantization training and loss function design of the projector. During the training phase, the system uses both the source and target model-generated KV caches to train the projector parameters in a supervised manner, ensuring that its output semantically approximates the target KV representation. During the testing phase, the system retains only the source model path and generates a KV cache compatible with the target model through the projector. Therefore, the total loss function... It can be represented as: , in, These represent the projection loss, prediction loss, regularization loss, and the weight coefficients of the projection loss, prediction loss, and regularization loss, respectively.

[0035] Among them, projection loss This is used to quantify the semantic similarity between the projected KV cache and the target KV cache. Traditional methods only use cosine similarity to quantify the degree of similarity, but cosine similarity has significant limitations: it only measures the similarity of vector directions and ignores the feature space structure; it is sensitive to changes in feature scale and cannot adapt to the distribution differences between models of different sizes; and it cannot capture nonlinear relationships in high-dimensional representation spaces.

[0036] In contrast, the CKA metric has unique advantages: it can capture the nonlinear structural similarity of the representation space, is invariant to changes in feature scale, and can effectively measure the geometric structural consistency of high-dimensional feature spaces, making it more suitable for collaborative reasoning scenarios involving large and small models.

[0037] Therefore, the CKA metric is used in the implementation examples to quantify the semantic similarity between the projected key cache and the target key cache. The projected KV cache and the target KV cache are defined as follows: and .

[0038] The steps for calculating the CKA similarity are as follows: ① Dimensional Unfolding and Reorganization: This method treats each token as a sample, with its embedding vector serving as a feature. The projected KV cache and the target KV cache are then transposed and reorganized to obtain their standard matrix shapes. and .

[0039] ② Kernel Function Construction: Since the core attention mechanism of Transformer is based on dot product calculation, this method uses a linear kernel function to directly reflect the consistency of the two key-value buffers in the attention space. The kernel matrices corresponding to the two key-value buffers are calculated as follows: , in, T This represents the matrix transpose operation; L express Y The kernel matrix constructed by (target KV cache); P express X The kernel matrix constructed from (source KV cache).

[0040] ③ Centering operation: To eliminate the influence of the data mean and focus on the changing structure of the feature space, the kernel matrix is ​​centered. Then: , in, H For a centered matrix, I It is the identity matrix. It is a matrix of all ones. This represents the number of samples.

[0041] ④ Similarity calculation: Calculate and The similarity between them, CKA is essentially a standardized Hilbert-Schmidt independence criterion (HSIC). and Flatten each into a one-dimensional vector, we get and Then there is and similarity It can be represented as: , in, This is represented as a modulo operation. Then the projection loss... It can be represented as: , The loss function takes values ​​in the range [0,1]. When the projected KV cache is completely identical to the target KV cache, the CKA value is 1, and the projection loss is... The value is 0; when the two are completely uncorrelated, the CKA value is close to 0, indicating a projection loss. Approaching 1. By minimizing The projector can learn the semantic space mapping from the source model to the target model, enabling efficient migration and alignment of key-value caches between large and small models.

[0042] Predicting losses The goal is to enable the model to generate accurate tokens based on the projected key-value cache. Therefore, this method uses standard cross-entropy loss to measure the difference between the tokens generated by the model and the true target tokens. It can be represented as: , in, and They represent the first i Each generated token, before the generation sequence i Key-value cache for each token; The model predicts the first... i The logarithm of the conditional probability of each token.

[0043] Regularization loss This method encourages the source key-value buffer after projection to retain sufficient information to avoid information loss during the projection process. Therefore, it utilizes L2 regularization to penalize unnecessary complexity or overfitting during projection and ensures a smoother key-value buffer after projection. It can be represented as: , in, It is represented as a projection mapping function.

[0044] Through the above three joint optimizations, the projector learns how to accurately map the KV cache of the source model to the semantic space of the target model during the training phase; while during the inference phase, it can generate semantically consistent and structurally compatible KV representations by relying only on the KV cache of the source model, thereby achieving efficient collaborative inference between large and small models.

[0045] (4) In this embodiment, the Open Hermes Preferences dataset is selected as the training dataset. This dataset is an open, synthetic dataset for fine-tuning general language models, designed to provide training data for a variety of downstream tasks. It covers a wide range of task domains, helping the projector network to generalize and adapt knowledge across multiple tasks.

[0046] S3. Input the projected key-value cache into the large model, and the large model performs autoregressive decoding based on the projected key-value cache to generate the final output of the inference request.

[0047] After completing the projection mapping of S2, the system obtains a semantically aligned key-value cache, which can be directly input into the target model for subsequent autoregressive generation and decoding operations. The specific implementation details of this step are as follows: (1) Projection buffer reception and buffer queue management. When the projector outputs the projected KV buffer... Then, the large model first appends the cache to the corresponding asynchronous buffer queue based on the inference request ID. : , in, Indicates that the request number is k The KV cache after projection.

[0048] During the receiving process, the system maintains an independent key-value buffer for each request to enable parallel writing and subsequent ordered concatenation. When a request... q k Once the new round of pre-filling results is completed, the corresponding projection cache will be appended to the corresponding cache, thereby ensuring that the KV state of each request can be continuously accumulated in the time dimension.

[0049] (2) Request Status Detection and Autoregressive Decoding Trigger. Because S1 employs a block-based pre-filling strategy, the KV cache for a single request is not generated all at once, but rather iteratively arrives in multiple iterations. Therefore, after each iteration, the system determines which requests have completed the pre-filling of all blocks. When a request is detected... q After all block processing is complete, the system retrieves the corresponding projected key-value (KV) cache and sends it to the decoding module of the large model, triggering the autoregressive generation process. Once the large model receives the complete KV cache, it can independently perform autoregressive inference without relying on the computation of the small model, gradually generating the output sequence.

[0050] (3) Asynchronous KV Cache Transmission and System Latency Optimization. To reduce inter-model communication latency, this embodiment adopts an asynchronous projection and transmission mechanism. That is, after the small model completes each round of forward computation, its generated KV cache is immediately sent to the projector for mapping, without waiting for the entire batch or request to be fully computed. The advantage of this mechanism is that the projector and the small model can work in parallel, and the data transmission between the projector and the large model can overlap with the next round of pre-filling computation. The large model only needs to start decoding immediately after detecting the arrival of complete request data. Therefore, the waiting delay of the large model only exists during the initialization of the first round of asynchronous projection in the system, during which a very short time "bubble" will be generated. However, as the pipeline gradually enters a steady state, the computation of the small model, the projector, and the large model can achieve almost complete overlap, so that the overall end-to-end latency of the system is almost unaffected.

[0051] Example 2 This embodiment also provides a performance optimization device for collaborative inference between large and small models in a computing power network, including: a processing module, an input module, and an output module; the processing module is used to select requests to be processed from the inference requests according to the length of the input sequence of several arriving inference requests, using a predefined scheduling strategy, and the small model performs block pre-filling processing on the input sequence of the selected requests to generate corresponding key-value caches; the input module is used to input the key-value caches generated by the small model to a pre-trained projector, and the projector maps the key-value caches from the semantic representation space of the small model to the semantic representation space of the large model to obtain the projected key-value caches; the output module is used to input the projected key-value caches to the large model, and the large model performs autoregressive decoding based on the projected key-value caches to generate the final output of the inference requests.

[0052] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for optimizing the performance of collaborative inference in large and small models for computing power networks, characterized in that, The method includes: S1. Based on the length of the input sequence of several arriving inference requests, a predefined scheduling strategy is used to select requests to be processed from the inference requests, and the small model performs block pre-filling processing on the input sequence of the selected requests to generate the corresponding key-value cache. S2. Input the key-value cache generated by the small model into the pre-trained projector. The projector maps the key-value cache from the semantic representation space of the small model to the semantic representation space of the large model to obtain the projected key-value cache. S3. Input the projected key-value cache into the large model, and the large model performs autoregressive decoding based on the projected key-value cache to generate the final output of the inference request.

2. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 1, characterized in that, S1 includes: Based on the length of the input sequence of the arriving inference requests, the inference requests are sorted using a short sequence priority strategy, and the requests to be processed are selected according to the sorting results. Based on the maximum number of tokens processed in a single iteration of the small model, the input sequence of the selected request is divided into one or more consecutive blocks; The small model performs a forward propagation operation on the partitioned blocks to generate the corresponding key-value cache.

3. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 1, characterized in that, S2 includes: The key-value cache generated by the small model is expanded and reorganized to obtain the first feature representation; The first feature representation is compressed using high-dimensional features to obtain the compressed feature representation. The compressed feature representation is subjected to nonlinear transformation and semantic enhancement to obtain an enhanced semantic feature representation. The enhanced semantic feature representation is dimensionally aligned and reshaped to obtain a projected key-value cache with the same shape as the key-value cache required by the large model.

4. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 3, characterized in that, The projector is trained using a joint loss function that includes projection loss, prediction loss, and regularization loss. The projection loss is calculated based on the center kernel alignment metric between the projected key-value cache and the target key-value cache, the prediction loss is calculated based on the cross-entropy loss, and the regularization loss is calculated based on L2 regularization.

5. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 4, characterized in that, The steps for calculating projection loss based on the center kernel alignment index include: Expand the projected key-value cache and the target key-value cache into standard matrix shapes respectively; Calculate the kernel matrices of the projected key-value cache and the target key-value cache respectively; The kernel matrix is ​​centered. The similarity between the center kernel alignment of the projected key-value cache and the target key-value cache is calculated based on the centered kernel matrix, and the projection loss is determined based on the similarity.

6. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 1, characterized in that, S3 includes: Receive the projected key-value cache and store it in the corresponding asynchronous buffer queue according to its corresponding inference request identifier; When it is detected that all input sequence blocks of a certain inference request have been pre-filled and projected, retrieve all projected key-value caches corresponding to the request from the asynchronous buffer queue; The extracted projected key values ​​are cached and input into the large model, triggering the large model to perform autoregressive decoding to generate the output sequence.

7. The method for optimizing the performance of collaborative inference in large and small models for computing power networks according to claim 6, characterized in that, An asynchronous transmission mechanism is used so that the key-value cache generated after the small model completes a single round of forward computation can be immediately sent to the projector for processing without waiting for all requests or batch computations to complete.

8. A performance optimization device for collaborative inference of large and small models for computing power networks, the device being used to implement the method described in any one of claims 1-7, characterized in that, include: Processing module, input module, and output module; The processing module is used to select a request to be processed from the inference requests according to the length of the input sequence of the incoming inference requests using a predefined scheduling strategy, and the small model performs block pre-filling processing on the input sequence of the selected request to generate the corresponding key-value cache. The input module is used to input the key-value cache generated by the small model into the pre-trained projector. The projector maps the key-value cache from the semantic representation space of the small model to the semantic representation space of the large model to obtain the projected key-value cache. The output module is used to input the projected key-value cache into the large model, and the large model performs autoregressive decoding based on the projected key-value cache to generate the final output of the inference request.

Images