Multi-terminal device collaborative reasoning method and system for large model perception segmentation

Abstract

The invention relates to the technical field of model reasoning, and discloses a multi-terminal device collaborative reasoning method for large model perception segmentation, which comprises the following steps of: performing fine-grained analysis on a computational graph of a target large language model, and identifying each operator type and calculation-communication characteristics thereof; automatically generating a differentiated weight division strategy according to the type of the operator, the memory usage, the calculation intensity and the attention mechanism variant; mapping the divided weight fragments and the calculation tasks to a plurality of edge devices, and constructing a minimum execution subgraph of each device; in the reasoning process, a host centralization aggregation coordination and calculation-communication overlapping mechanism is adopted, and the synchronization overhead is minimized; and executing autoregressive reasoning, generating output token by token, and optimizing subsequent reasoning efficiency through a dynamic weight cache mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model inference, in particular to a large model perception segmentation multi-terminal device collaborative inference method and system. BACKGROUND

[0002] Large language models are gradually being deployed to edge devices to support privacy protection and low-latency response for local intelligent applications due to their strong language understanding and generation capabilities. However, due to the limited memory and computing resources of edge devices such as mobile phones, embedded development boards, and IoT nodes, current mainstream large models cannot efficiently run on a single device, especially for models with more than 100 billion parameters, which often cannot be loaded due to insufficient memory.

[0003] To break through the resource bottleneck of a single device, researchers have attempted to coordinate multiple edge devices to jointly perform LLM inference tasks. Current mainstream collaborative inference techniques mainly draw on parallel strategies in data center scenarios, mainly including two categories: 1) pipeline parallelism, which divides the model by layer and assigns it to different devices, and passes the intermediate activation values layer by layer during forward propagation; 2) tensor parallelism, which splits the weight matrix within a single layer by row or column, and each device calculates a partial result before aggregation. These methods perform well in GPU clusters and batch inference scenarios, but face serious challenges in resource-constrained, token-by-token autoregressive decoding edge environments.

[0004] Specifically, the existing technology has the following key problems: First, pipeline parallelism has a serious device idle problem during token-by-token decoding. Since each device must wait for the output of the previous layer before starting calculation, the computational resource utilization is low, making it difficult to reduce single token latency; second, tensor parallelism introduces excessive synchronization overhead on edge devices. Each token generated requires multiple fine-grained communications and synchronization between devices, while the limited bandwidth of edge networks and inefficient communication stacks make the synchronization time longer than the local computation time in normal cases, becoming a performance bottleneck; finally, existing methods generally use a unified, operator-independent partitioning strategy, ignoring the significant differences in computing-communication characteristics and KV cache semantics (MHA / MQA / GQA) among different modules (such as self-attention, feedforward network, output projection) in the Transformer architecture, resulting in uneven load distribution, excessive communication volume, and even breaking the correctness of the attention mechanism after partitioning.

[0005] To solve the above problems, some studies attempt to introduce hybrid parallel or computation-communication overlap optimization methods, but still cannot fundamentally solve the synchronization problem under token-by-token reasoning, and lack adaptive support for different attention variants. Therefore, a new collaborative reasoning framework is needed to achieve efficient, scalable, and low-latency large language model inference on resource-constrained edge device clusters. SUMMARY

[0006] The present application provides a large model-aware split multi-terminal device collaborative reasoning method and system to solve the positioning problem of existing collaborative reasoning frameworks.

[0007] To achieve the above purpose, the present application realizes the following technical solutions: In a first aspect, the present application provides a large model-aware split multi-terminal device collaborative reasoning method, comprising: Performing fine-grained analysis on the computation graph of the target large language model to identify the types of each operator and their computation-communication characteristics; According to the types, memory occupation, computation intensity, and attention mechanism variants of the operators, automatically generating differentiated weight division strategies; Mapping the divided weight slices and computation tasks to multiple edge devices to construct the minimum execution subgraph for each device; In the reasoning process, use host-centralized aggregation coordination and computation-communication overlap mechanisms to minimize synchronization overhead; Performing autoregressive reasoning to generate output token by token, and optimizing subsequent reasoning efficiency through a dynamic weight caching mechanism.

[0008] Optionally, the weight division strategy includes: Column slicing or shared replication of Query, Key, and Value weights in the self-attention module according to the attention mechanism type; Column slicing of the upper projection and gated projection weights in the feedforward network, and row slicing of the lower projection weights; Column slicing of the output projection layer weights according to the vocabulary dimension; Marking lightweight operators as locally reserved and only executing them on the host device.

[0009] Optionally, the attention mechanism variants include multi-head attention mechanism, multi-query attention mechanism, and grouped query attention mechanism, and the mechanism division methods are as follows: For multi-head attention mechanism, column slicing is performed on Q, K, and V weights, and row slicing is performed on output projection weights; For multi-query attention mechanism, copy the full amount of K and V weights to each device, and only perform column slicing on Q weights; For the grouped query attention mechanism, the K and V heads are grouped and shared according to the ratio of the number of devices to the number of head groups, and the Q heads are evenly distributed.

[0010] Optionally, the construction of the minimum execution subgraph includes: Generate an execution graph for each device that only contains its local operators; Explicitly mark synchronization operation nodes in the graph, including AllReduce and AllGather; Eliminate lightweight operators that do not participate in distributed execution to avoid unnecessary communication dependencies.

[0011] Optionally, the compute-communication overlap mechanism includes: The host starts local computation immediately after distributing the input, without waiting for the worker node to respond; Use pre-allocated buffers and I / O thread pools to asynchronously receive local results from worker nodes; Trigger aggregation operations immediately after the results are collected and advance the next stage of computation.

[0012] Optionally, the method further includes: After the first inference is completed, each device caches the weight slices according to the memory conditions; In subsequent inferences, the host retrieves and skips the transmission of cached weights through weight identification.

[0013] In a second aspect, the embodiments of the present application provide a multi-terminal device cooperative inference system for large model-aware partitioning, which implements the method of any one of the first aspect, and includes: A model operator-aware partitioning module for parsing model structure and generating partitioning strategies; A device-aware load balancing mapping module for mapping task slices to a heterogeneous device cluster; A synchronization overhead minimization inference module for performing inference and optimizing communication synchronization; A collaborative device weight caching module for managing the caching and reuse of weight slices.

[0014] Optionally, the model operator-aware partitioning module includes: A model weight analysis submodule for identifying divisible weights and locally retained operators; An attention mechanism adaptive partitioning submodule for dynamically generating partitioning templates according to attention types; A feedforward network collaborative partitioning submodule for complementary partitioning of FFN weights by rows and columns; An output projection vocabulary partitioning submodule for partitioning the output layer by vocabulary dimension; A lightweight operator local retention submodule for marking and retaining lightweight operators for execution on the host.

[0015] Optionally, the synchronization overhead minimization inference module comprises: a minimization subgraph generation submodule for constructing a device local execution graph; a host centralized aggregation coordination submodule for centralized processing of AllReduce and AllGather operations; a computation-communication overlap execution submodule for implementing asynchronous distribution and parallel computation; a collaborative device weight cache submodule for managing storage and retrieval of weight cache.

[0016] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method of any one of the first aspect.

[0017] Advantages: The large model perception segmentation multi-terminal device collaborative inference method provided by the application, through the fine-grained division strategy of the operator perception, automatically selects the optimal row-column or vocabulary dimension segmentation mode according to the computation-communication characteristics of different self-attention architectures MHA, MQA, GQA, and different modules such as feedforward network (FFN) and output projection, and combines device heterogeneity for load balancing mapping; at the same time, through the synchronization overhead minimization execution mechanism, a lightweight subgraph is constructed, host centralized aggregation coordination is adopted, and computation-communication overlap and dynamic load scheduling are deeply integrated, which significantly hides the cross-device communication delay. The application not only supports large model collaborative inference beyond the memory capacity of a single device, but also realizes inference acceleration of up to 5.4 times in a real edge cluster, which is significantly better than the traditional single-machine inference or pipeline parallel baseline, and is suitable for practical application scenarios such as privacy-sensitive, low-latency requirement local intelligent assistants, AR / VR interaction, edge AI services, etc., enabling the hundred-billion-parameter-level large language model to truly have the ability to run efficiently in the cloud outside the edge environment. It can effectively solve the core problems of large synchronization overhead, high communication redundancy, and coarse-grained operator division faced by existing parallel inference schemes in edge scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of the large model perception segmentation multi-terminal device collaborative inference of the preferred embodiment of the application; Figure 2 A framework diagram of the multi-device collaborative inference system designed by the application; Figure 3 A model weight division and collaborative inference flowchart designed by the application; Figure 4 A performance comparison diagram of I / O, computation overlap, and traditional I / O, computation synchronization proposed by the application; Figure 5Inference latency comparison of deploying and coordinating inference of the LLAMA3-120b Q2_K quantized model in parallel with the pipeline using 8 Raspberry Pi 5s in a preferred example of the present application. DETAILED DESCRIPTION

[0019] The technical solutions of the present application will be described below in a clear and complete manner. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0020] Unless otherwise defined, the technical terms or scientific terms used in the present application shall have the usual meanings understood by those of ordinary skill in the art to which the present application belongs. The terms "first", "second", and similar words used in the present application do not represent any order, number, or importance, but are only used to distinguish different components. Similarly, "one" or "a" and similar words do not represent a quantity limitation, but represent the existence of at least one. The terms "connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0021] Please refer to Figure 1 The embodiment of the present application provides a multi-terminal device cooperative inference method for large model perception segmentation, comprising: Performing fine-grained analysis on the computation graph of the target large language model to identify the types of each operator and their computation-communication characteristics; According to the types, memory occupancy, computation intensity, and attention mechanism variants of the operators, automatically generating differentiated weight division strategies; Mapping the divided weight slices and computation tasks to multiple edge devices to construct the minimum execution subgraphs of each device; In the inference process, a host-centered aggregation coordination and computation-communication overlap mechanism is used to minimize synchronization overhead; Performing autoregressive inference to generate output token by token, and optimizing the efficiency of subsequent inference through a dynamic weight caching mechanism.

[0022] Optionally, the weight division strategy comprises: Column-cutting or shared copying the Query, Key, and Value weights in the self-attention module according to the attention mechanism type; Column-cutting the up-projection and gated projection weights in the feedforward network, and row-cutting the down-projection weights; The output projection layer weight is column split according to the vocabulary dimension; Mark the lightweight operator as locally reserved, and only execute on the host device.

[0023] In the above embodiment, in the model loading stage, all divisible weight types are identified by traversing all weights of the LLM, including Self-Attention, FFN and OutputProjection, and the calculation intensity, memory occupation and intermediate tensor size are evaluated. Based on the preset threshold such as single operator delay ratio or parameter quantity ratio, the calculation intensive operator is marked as "cooperative execution is needed", and the lightweight operator such as TokenEmbedding, LayerNorm, and rotated position encoding is marked as "locally reserved" to avoid unnecessary cross-device communication.

[0024] Optionally, the attention mechanism variant includes a multi-head attention mechanism, a multi-query attention mechanism and a grouped query attention mechanism, and the mechanism division mode is respectively: For the multi-head attention mechanism, the Q, K and V weights are column split, and the output projection weight is row split; For the multi-query attention mechanism, the K and V weights are fully copied to each device, and only the Q weight is column split; For the grouped query attention mechanism, the K and V heads are grouped and shared according to the ratio of the number of devices to the number of head groups, and the Q heads are evenly distributed.

[0025] In the above embodiment, first, the metadata of the model is read to automatically identify whether the KV sharing strategy adopted by the current model is MHA, MQA or GQA. Then, according to the identification result, the corresponding division template is dynamically generated: For MHA (multi-head attention), the Query (Q), Key (K) and Value (V) weight matrices are all column split, so that each device independently calculates part of the attention heads;

[0026] The attention output projection matrix is row split, so that the local output can be aggregated by AllReduce sum:

[0027] During calculation, each device first calculates the attention result of the corresponding attention head independently:

[0028]

[0029] Then the output of the attention head is calculated with the corresponding output projection weight to obtain the output of the corresponding attention head:

[0030] For MQA (multi-query attention), the shared K / V weight is replicated to all devices, and only the Q weight is column partitioned to maintain semantic correctness and reduce communication volume; For GQA (grouped query attention), the K / V head is shared within the device group in proportion to the number of devices and head groups, and the Q head is evenly distributed to balance computation and communication.

[0031] Optionally, the construction of the minimum execution subgraph includes: Generate an execution graph for each device that only contains its local operators; Explicitly mark synchronization operation nodes in the graph, including AllReduce and AllGather; Remove lightweight operators that do not participate in distributed execution to avoid unnecessary communication dependencies.

[0032] In the above embodiment, for the three sub-operators contained in the FFN Adopt a complementary partitioning strategy: Use column partitioning to allow each device to independently calculate the high-dimensional intermediate activation; Use row partitioning to allow the local down-projection result to be directly aggregated into the complete output through AllReduce summation; This design avoids cross-device broadcasting of intermediate activations, compresses communication volume to only one reduction operation, and significantly reduces synchronization overhead.

[0033] Optionally, the compute-communication overlap mechanism includes: The host starts local computation immediately after distributing the input, without waiting for the worker node to respond; Use pre-allocated buffers and I / O thread pools to asynchronously receive local results from worker nodes; Trigger aggregation operations and advance the next stage of computation immediately after the results are collected.

[0034] In the above embodiment, the output projection layer at the end of the model is specially optimized. Since the size of the vocabulary is usually much larger than the model hidden dimension, the weight memory occupation of this layer is significant. Specifically, this sub-module divides the output projection matrix into N non-overlapping sub-matrices in the column direction (i.e., the vocabulary dimension) Each device only holds part of the weight corresponding to the vocabulary. During inference, each device calculates local logits in parallel: ​

[0035] Subsequently, all devices stitch the local logits along the vocabulary dimension by an AllGather operation to form a complete logits vector for final token prediction. This design evenly distributes the output layer memory overhead to each device, avoiding single-device memory bottlenecks, while the communication volume is only the size of the logits vector (not the weights themselves), significantly better than the full replication strategy.

[0036] Optionally, the method further comprises: After the first inference is completed, each device caches the weight shards according to memory conditions; In subsequent inferences, the host retrieves and skips the transmission of cached weights through weight identifiers.

[0037] In the above embodiments, by identifying and retaining lightweight operators that have minimal contribution to overall latency but would introduce significant communication overhead if executed in a distributed manner, their local execution on the host device is forced.

[0038] Specifically, the retained operators include: Token Embedding: Although its weight matrix size is comparable to the output projection, its forward calculation is only performed once in the prefill stage, and there is no KV cache dependency in the decode stage, so full loading into the host memory is sufficient without the need for segmentation; Layer Normalization: The calculation is element-wise normalization, with minimal parameter size (only scaling and offset vectors), and requires a complete input tensor, so distributed execution requires additional AllGather, which is not worth the cost; Rotary Position Encoding (RoPE): It is a deterministic function with no parameters, and the calculation overhead is negligible, so local execution has no communication cost.

[0039] By marking the above operators as "locally retained", the system excludes them from the distributed execution plan when constructing the device subgraph, and only executes them on the host, thereby eliminating unnecessary cross-device synchronization points.

[0040] After completing the partition templates for each operator, the weight shards and calculation tasks are mapped to the actual set of edge devices participating in collaboration. The core is to jointly consider device heterogeneity (memory, computing power, bandwidth) and operator computation-communication characteristics to achieve global load balancing.

[0041] The embodiments of the present application also provide a multi-terminal device collaborative inference system for large model-aware segmentation, comprising: A model operator-aware partition module for parsing the model structure and generating a partition strategy; A device-aware load balancing mapping module for mapping task shards to a heterogeneous device cluster; A synchronization overhead minimization inference module is configured to perform inference and optimize communication synchronization. A collaborative device weight caching module is configured to manage caching and reuse of weight shards.

[0042] Optionally, the model operator-aware partitioning module includes: A model weight analysis submodule is configured to identify partitionable weights and local reserved operators. An attention mechanism adaptive partitioning submodule is configured to dynamically generate a partitioning template according to an attention type. A feedforward network collaborative partitioning submodule is configured to perform row-column complementary cutting of FFN weights. An output projection vocabulary partitioning submodule is configured to partition an output layer according to a vocabulary dimension. A lightweight operator local reservation submodule is configured to mark and reserve lightweight operators for execution on a host.

[0043] Optionally, the synchronization overhead minimization inference module includes: A minimum subgraph generation submodule is configured to construct a device-local execution graph. A host-centralized aggregation coordination submodule is configured to centrally process AllReduce and AllGather operations. A compute-communication overlap execution submodule is configured to implement asynchronous distribution and parallel computation. A collaborative device weight caching submodule is configured to manage storage and retrieval of weight caches.

[0044] A computer-readable storage medium having a computer program stored thereon, when the program is executed by a processor, implements any one of the methods of the multi-terminal device collaborative inference method based on large model-aware partitioning.

[0045] Specifically, in the embodiment, an LLAMA3-120b Q2_K quantization model (total model size about 46.2 GB) is deployed and run on 8 Raspberry Pi 5 development boards (Raspberry Pi 5, equipped with 8 GB RAM, four-core ARM Cortex-A76 CPU) to illustrate the specific implementation steps of the present application. The entire implementation process is divided into two stages: offline planning and online inference, and the overall workflow is as shown in Figure 2 .

[0046] I. Offline planning stage Step 1: Model weight partitioning based on operator awareness Before the model is loaded, the model operator perception division module of the application first analyzes the computation graph structure of LLaMA3-120B, identifies the types of each operator and its KV sharing strategy. The model adopts group query attention (GQA), each layer contains 8 KV heads, a total of 8 devices, so each device is allocated 1 KV head.

[0047] Self-attention module: Query weight of each layer Column-wise split is adopted, and each device is allocated a 8192x1024 submatrix; Key and value weight Due to the GQA sharing mechanism, it is evenly divided according to the number of KV heads, and each device is allocated a 8192x128 submatrix; Attention output projection weight Row-wise split is adopted, and each device is allocated a 1024x8192 submatrix.

[0048] Feedforward network (FFN): The FFN of the model does not contain a bias term, and contains an upper projection , a gating projection (both are 8192x28672) and a lower projection

[0049] The application adopts column-wise split for and , row-wise split for , and each device obtains a weight fragment of 8192x3584, 8192x3584 and 3584x8192 respectively.

[0050] Output projection layer: Vocabulary projection matrix , column-wise split according to the vocabulary dimension, and each device is allocated a 8192x4000 submatrix.

[0051] Local reservation operator: Token Embedding, each layer LayerNorm and other lightweight operators have small computational overhead and require complete input, so they are executed on the host in full, without being split.

[0052] The size and file offset of all weight fragments are calculated in the planning stage in advance. For example, the The offset is 8192 x 1024 x 2 x element_size. Since the total model size (46.2 GB) is much larger than the single-device memory (8 GB), the host reads each tensor shard on demand by system call read when loading, and distributes it to the corresponding device or keeps it locally through RPC.

[0053] Step 2: Constructing the minimized device subgraph and collaborative computation graph After the weight division and transmission are completed, the system enters the computation graph construction phase. The core of the present application is to generate a minimized execution subgraph for each device, which only contains the actual operator nodes it needs to execute and the necessary synchronization primitives, thereby avoiding the high latency and memory overhead caused by full graph broadcast in traditional RPC frameworks.

[0054] The specific construction process is as follows: 1. Host subgraph construction: The subgraph of the coordinating host (i.e., the designated master device) contains two types of nodes: Local exclusive operators: Token Embedding, all LayerNorm, RoPE rotation position encoding, and other lightweight operators, as they do not need to be distributed for execution and are directly retained on the host; Aggregation nodes: After each Self-Attention, FFN, and Output Projection module, a synchronization primitive is inserted - an AllReduce node for Self-Attention and FFN output (used to sum the local results), and an AllGather node for Output Projection (used to concatenate the local logits).

[0055] 2. Work node subgraph construction: Each work node (i.e., the 7 participating Raspberry Pis) only contains the computation nodes corresponding to the weights assigned to it: Self-Attention shard: receives hidden state input from the host, performs local Q / K / V projection (according to GQA rules, K / V is single-headed, Q is partially headed), and calculates the local attention output; FFN shard: performs local , column split calculation, and then gets the local FFN output through row split; Output Projection shard: performs local vocabulary projection on the FFN output to generate logits shards with dimensions [1, 4000]; All subgraphs do not contain any aggregation logic, only return the computation result to the host through RPC.

[0056] 3. Synchronization primitives are explicitly modeled: The system explicitly marks two types of synchronization operations in the computation graph: AllReduce: for additive intermediate results (e.g., Self-Attention output, FFN output), each device computes a local partial, and the host sums and broadcasts the complete result; AllGather: for non-additive but concatenable results (e.g., Output Projection logits), each device returns local logits, and the host concatenates them into [1, 32000] complete logits along the vocabulary dimension.

[0057] 4. Dependency alignment and pipeline: All subgraphs strictly follow the Transformer layer order: Embedding → LayerNorm → Self-Attention → LayerNorm → FFN → LayerNorm → … → Output Projection.

[0058] The input of Self-Attention and FFN in each layer comes from the complete output of the previous layer (which has been aggregated in the pre AllReduce), ensuring semantic correctness. The execution flow of the computation graph is shown in Figure 3 .

[0059] Step three: Collaborative inference execution and computation-communication overlap The inference phase uses a self-recursive approach to execute token by token. The processing flow for each token is as follows, fully embodying the synchronization minimization and computation-communication overlap mechanism of the present invention: 1. Host local preprocessing: The host first performs Token Embedding and initial LayerNorm to generate the input tensor for the first layer Self-Attention

[0060] 2. Asynchronous distribution and parallel start: As shown in Figure 4 , the host distributes X in parallel to all 7 worker nodes through non-blocking RPC, and immediately starts the forward computation of the local Self-Attention shard (without waiting for the worker node to confirm receipt), achieving "send and compute".

[0061] 3. Worker node parallel computation: Self-Attention shard computation (including Q / K / V projection, attention score, local output); FFN shard computation (up / gate / down projection); Output Projection shard computation (if it is the last layer); After computation, return the result (e.g., local attention output, local FFN output, or local logits) to the host through RPC.

[0062] 4. Asynchronous result collection and aggregation: The host maintains an I / O thread pool, and each thread is responsible for listening to the return data stream of a worker node. When the local result arrives, the thread writes it directly into the pre-allocated aggregation buffer, avoiding the delay jitter introduced by dynamic memory allocation.

[0063] When all Self-Attention local results arrive, the host performs AllReduce to get the complete attention output; similarly, the FFN output is also aggregated through AllReduce; finally, the local logits of the Output Projection are formed into complete logits through AllGather (concatenation).

[0064] Through the above mechanism, the invention successfully runs LLaMA3-120B-Q2 (46.2GB) on 8 Raspberry Pi 5s, with the llama.cpp self-provided pipeline behavior as the baseline, and the final per-token inference delay is significantly lower than the baseline (up to 4.6x acceleration in actual measurement), verifying the efficiency and feasibility of the collaborative inference framework in resource-constrained edge environments.

[0065] The above describes the preferred embodiments of the invention in detail. It should be understood that those skilled in the art can make many modifications and changes without creative labor according to the concept of the invention. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the existing technology according to the concept of the invention shall be within the protection scope determined by the claims.

Claims

1. A large model perception segmentation multi-terminal device cooperative inference method, characterized in that, The method comprises the following steps: Fine-grained analysis of the computational graph of the target large language model, identifying the types of each operator and their computation-communication characteristics; According to the type, memory occupation, computing intensity and attention mechanism variant of the operator, a differentiated weight division strategy is automatically generated; Map the divided weight slices and computing tasks to multiple edge devices to construct the minimum execution subgraph of each device; In the inference process, the host-centered aggregation coordination and computation-communication overlap mechanism is adopted to minimize the synchronization overhead; Perform autoregressive inference to generate output token by token, and optimize the efficiency of subsequent inference through a dynamic weight caching mechanism.

2. The method of claim 1, wherein the method further comprises: The weight division strategy includes: Column slicing or shared replication of Query, Key, and Value weights in the self-attention module according to the attention mechanism type; Column slicing of the upper projection and gated projection weights in the feedforward network, and row slicing of the lower projection weights; Column slicing of the output projection layer weights according to the vocabulary dimension; Mark lightweight operators as locally reserved and only execute them on the host device.

3. The method of claim 2, wherein the method further comprises: The attention mechanism variants include multi-head attention mechanism, multi-query attention mechanism, and grouped query attention mechanism, and the mechanism division methods are as follows: For multi-head attention mechanism, Q, K, and V weights are column sliced, and output projection weights are row sliced; For multi-query attention mechanism, copy the K and V weights to each device in their entirety, and only slice the Q weights column-wise; For the grouped query attention mechanism, group-share K and V heads according to the ratio of the number of devices to the number of head groups, and evenly distribute Q heads.

4. The method of claim 1, wherein the method further comprises: The construction of the minimum execution subgraph includes: Generate an execution graph for each device that only contains its local operators; Explicitly mark synchronization operation nodes in the graph, including AllReduce and AllGather; Remove lightweight operators that do not participate in distributed execution to avoid unnecessary communication dependencies.

5. The method of claim 1, wherein the method further comprises: The computation-communication overlap mechanism includes: The host starts local computation immediately after distributing the input, without waiting for the worker nodes to respond; Use pre-allocated buffers and I / O thread pools to asynchronously receive local results from worker nodes; Trigger aggregation operations and advance the next stage of computation immediately after all results are collected.

6. The method of claim 1, wherein the method further comprises: The method further comprises: After the first inference is completed, each device caches the weight slices according to the memory conditions; In subsequent inferences, the host retrieves and skips the transmission of cached weights through weight identification.

7. A multi-terminal device collaborative inference system for implementing large model-aware partitioning of any one of claims 1-6. The method comprises the following steps: Model operator-aware division module for parsing model structure and generating division strategy; Device-aware load balancing mapping module for mapping task slices to a heterogeneous device cluster; Synchronization overhead minimization inference module for performing inference and optimizing communication synchronization; Collaborative device weight caching module for managing weight slice caching and reuse.

8. The large model-aware partitioned multi-terminal device collaborative reasoning system of claim 7, wherein, The model operator-aware division module includes: Model weight analysis submodule for identifying divisible weights and locally reserved operators; Attention mechanism adaptive division submodule for dynamically generating division templates according to attention types; Feedforward network collaborative division submodule for complementary row and column slicing of FFN weights; Output projection vocabulary slicing submodule for slicing output layers according to vocabulary dimensions; A lightweight operator local reservation submodule is configured to mark and reserve the lightweight operator in the host execution.

9. The large model-aware partitioned multi-terminal device collaborative reasoning system of claim 7, wherein, The synchronization overhead minimization inference module comprises: A minimization subgraph generation submodule is configured to construct a device local execution graph; A host centralized aggregation coordination submodule is configured to centrally process AllReduce and AllGather operations; A computation-communication overlap execution submodule is configured to implement asynchronous distribution and parallel computation; A cooperative device weight cache submodule is configured to manage storage and retrieval of the weight cache.

Images