A cloud-edge collaborative large language model hierarchical perception quantization and distributed deployment method

Abstract

The application discloses a large language model layered perception quantization and distributed deployment method under cloud edge cooperation. The method is as follows: first, a large language model configuration model and a cloud edge node configuration model are established, and the calculation amount, activation dimension, node computing power, memory and link bandwidth of each layer are obtained; second, a layered quantization evaluation model is constructed, and a feasible bit width set of each layer is selected according to the relative reconstruction error and threshold constraint; an end-to-end inference delay model is established, the delay is decomposed into calculation delay and cross-node communication delay, and a joint optimization problem is constructed to minimize the inference delay and meet the memory and bandwidth constraints; third, a bit width-aware contract matching algorithm is used for solving, a layer and node preference list is constructed, and a stable matching is obtained by iterating the proposal and acceptance / rejection; finally, distributed sequential inference is performed according to the matching result to complete the deployment. The application can reduce the inference delay and resource consumption under the premise of ensuring the inference quality.

Description

Technical Field

[0001] This invention belongs to the field of large model deployment technology, specifically involving a method for hierarchical perception quantization and distributed deployment of large language models under cloud-edge collaboration. Background Technology

[0002] Cloud-edge collaborative inference, capable of providing intelligent services closer to the user, reducing interaction latency and improving availability, has become an important form of deployment for large language models. Large language models are typically built on a Transformer decoder architecture, consisting of an embedding layer, consecutively stacked decoding layers, and an output layer. Inference requires execution layer by layer and the transfer of activation data between layers. Given the large number of model parameters, high memory consumption, and computationally intensive nature of these models, centralized inference in the cloud often introduces additional network transmission latency; while deploying the entire model directly to edge nodes is easily limited by edge computing power and memory capacity, making it difficult to meet real-time response requirements. Therefore, layered collaborative inference schemes that split the model into layers and distribute them across multiple heterogeneous cloud-edge nodes for execution are gaining increasing attention.

[0003] As model size continues to increase, resource demands are rising rapidly. However, in hierarchical deployments involving heterogeneous nodes, the computing power, memory capacity, and link bandwidth of each node differ significantly. Cross-node deployment between adjacent layers incurs activation transmission overhead, making end-to-end inference latency significantly affected not only by computation time but also by communication time. To reduce resource overhead, existing technologies often employ low-bit quantization to reduce memory usage and improve throughput. However, quantization introduces precision loss, and different decoding layers have varying sensitivities to quantization errors. Using a uniform bit width or coarse-grained strategy may lead to a decrease in inference quality; conversely, increasing the bit width to ensure quality weakens compression benefits. Some methods obtain feasible bit width sets through layer-by-layer error evaluation and threshold screening, but these are not considered in conjunction with layer-node deployment decisions.

[0004] Therefore, under the premise of simultaneously satisfying inference quality, node capacity and inter-layer dependency constraints, how to comprehensively consider the impact of quantization bit width on computation and communication, construct a joint decision-making mechanism oriented towards end-to-end latency, and obtain feasible and stable layer-node allocation results are still problems that need to be solved by existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a hierarchical quantization and distributed deployment method that can effectively reduce end-to-end inference latency and deployment resource overhead in large language model deployment scenarios for heterogeneous cloud-edge collaborative inference, while ensuring inference quality, and simultaneously taking into account node memory capacity constraints and link bandwidth constraints. The technical solution to achieve the purpose of this invention is: a hierarchical perceptual quantization and distributed deployment method for large language models under cloud-edge collaboration, comprising the following steps:

[0006] Step 1: Establish the configuration model for the large language model to be deployed and the configuration model for the cloud-edge nodes;

[0007] Step 2: Establish a hierarchical quantitative evaluation model and an end-to-end inference latency model;

[0008] Step 3: With the goal of minimizing end-to-end inference latency, establish a joint optimization problem of deployment and quantization under the conditions of satisfying node resource constraints and link bandwidth constraints;

[0009] Step 4: Solve the joint optimization problem by using a bit-width-aware contract matching algorithm based on matching theory to generate deployment and quantization schemes. In this algorithm, the binary combination of nodes and bit width is defined as a contract and a preference relationship is constructed. Stable matching results are obtained through an iterative process of proposal, acceptance, and rejection.

[0010] Step 5: Perform distributed sequential inference based on the stable matching result, so that each decoding layer loads the corresponding bit width quantization weight on its matching node to complete the calculation, and transmits intermediate activation data when adjacent decoding layers match different nodes to complete end-to-end inference.

[0011] Furthermore, the establishment of the configuration model for the large language model to be deployed and the cloud-edge node configuration model described in step 1 is as follows:

[0012] Determine the set of decoding layers for a large language model and inter-layer dependencies, and for any decoding layer Obtain its computational cost and the scale of intermediate activation dimensions At the same time, determine the set of candidate deployment nodes. and for any node Obtain its effective computing power With available video memory capacity And for any node pair Obtain link bandwidth ;in, Indicates the first The floating-point computation cost of the layer under single-token conditions. Indicates the first The activation dimension scale in the middle of the layer, Represents a node Effective computing power, Represents a node Available video memory capacity, Represents a node With nodes Available bandwidth of the link between them.

[0013] Furthermore, the hierarchical quantitative evaluation model and the end-to-end inference latency model described in step 2:

[0014] This invention addresses the weights of each decoding layer l in a large language model. In candidate bit width Quantization weights are obtained The relative reconstruction error is used as the error index, and its definition is as follows:

[0015] Furthermore, given a preset error threshold Construct the feasible bit-width set of the l-th layer. Its definition is as follows:

[0016]

[0017] In equation (1), The model number is represented by the first... The original weight matrix of the layer, Indicates the first Layer uses bit width Quantized weight matrix, Denotes the Frobenius norm. Indicates the first Layer in position width The relative quantization reconstruction error is given in equation (2). For the error threshold, For the candidate bit width set, For the first The feasible bit width set of the layer under the condition of satisfying error constraints; the feasible bit width set is used to constrain the search space of subsequent deployment and quantization decisions, thereby establishing a controllable boundary between compression ratio and inference quality.

[0018] This invention further establishes an end-to-end inference latency model, decomposing the end-to-end inference latency into two parts: computation latency and communication latency. The l-th layer uses a quantization bit width... The computational latency cost term deployed on node e is expressed as:

[0019] In equation (3), Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates the first The floating-point computational cost of a single decoding layer under single-token conditions; This represents the number of tokens used in the reasoning. Represents a node Effective computing power; Indicates the first The quantization bit width selected for the layer, This represents the bit width scaling factor based on 16 bits.

[0020] The communication latency cost when deploying across nodes in adjacent layers is represented as follows:

[0021]

[0022] In equation (4), Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the size of the intermediate activation dimension; Represents a node With nodes Inter-link bandwidth; This represents the number of tokens used in the reasoning. Indicates the first The quantization bit width selected for the layer.

[0023] Furthermore, the specific steps for establishing the deployment-quantization joint optimization problem model described in step 3 are as follows:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in, Represents the set of model decoding layers. Represents the set of candidate edge deployment nodes. Indicates the first The selected quantization bit width for the layer Indicates the first The feasible bit-width set obtained after filtering by the error threshold; decision variables Indicates the first Is the layer deployed on the node? If deployed, then ,otherwise Decision variables Indicates adjacent layers Deployed on nodes With nodes Does cross-node communication occur during the process? If so, ,otherwise Indicates the first Layer by width At the node The computational delay cost term executed on the above, Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the first Layer in position width The video memory usage is as follows. Represents a node Available video memory capacity; constraint (6) ensures that each layer It must be deployed on one and only one node, that is, for any satisfy Constraint (7) ensures communication variables through inequality linkage. Consistent with the deployment choices of the two adjacent layers, i.e., only when the first layer... Layer deployment in And the Layer deployment in Only when allowed Set to 1; constraint (8) is a node capacity constraint, ensuring that nodes are assigned to the same node. The total memory usage of each layer at the corresponding bit width does not exceed the available capacity MemCap of the node; constraint (9) is a bit width feasibility constraint to ensure that the bit width involved in each layer is within the limit. It must come from its feasible bit-width set Constraint (10) is a variable value constraint, which specifies... and It is a binary variable;

[0031] Step 4 above includes:

[0032] To uniformly express the selection of deployment nodes and quantization bit width for each layer, the concept of contract is introduced. For any node-bit width combination, the contract c=(e,b) is defined, which means that a certain layer is deployed to node e and inference is performed using bit width b, where e represents the edge candidate deployment node and b represents the quantization bit width.

[0033] For any model layer l, construct its set of optional contracts as follows:

[0034]

[0035] in, Indicates the first The contract set of the layer; Represents the set of candidate edge deployment nodes; Indicates the first The feasible bit-width set obtained after filtering by the error threshold;

[0036] A bilateral preference model is established to characterize the preference relationship between model layers and nodes for contracts. In the layer preference model, contracts are... For the The synthesis cost function of a layer is defined as:

[0037]

[0038] Based on this, the hierarchy preference order is defined as follows:

[0039]

[0040] in, Indicates the first The overall cost of choosing contract c at layer level; Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates starting from the node at the previous level. To candidate nodes The communication latency cost incurred in transmitting activation data; Indicates the first The preference relationship between layers for contracts;

[0041] In the node preference model, nodes Accepting the contract The preference cost is defined as:

[0042]

[0043] Based on this, the node preference order is defined as follows:

[0044]

[0045] in This represents the inbound communication pressure cost borne by node e when it accepts contract c; This represents the preference relationship of node e for the contract.

[0046] The bit-width-aware congruence matching algorithm based on matching theory described in step 4 is as follows:

[0047] Initialize all model layers to an unmatched state. When an unmatched model layer exists and its preference list is not empty, that model layer initiates a proposal request to its most preferred contract in order of preference, where the contract consists of a combination of candidate deployment nodes and quantization bit width. The node to which the proposal is made collects all proposals requested to be deployed to that node and filters the proposals based on the node's available resource constraints. When all collected proposals satisfy the node's resource constraints, the node temporarily accepts all proposals; otherwise, the node only retains the best proposal that satisfies the constraints and rejects the rest. The rejected model layer removes the rejected contract from its preference list and continues to initiate a proposal to the next preferred contract. Repeat the above proposal, temporary acceptance, and rejection process until there are no unmatched model layers and the preference list is not empty. Output the final accepted contract corresponding to each model layer, thereby obtaining a stable mapping relationship between model layers and contracts.

[0048] Compared with the prior art, the significant advantages of this invention are: (1) In the heterogeneous cloud-edge collaborative inference environment, by jointly considering the constraints of node computing power, memory capacity and link bandwidth, a feasible distributed deployment of the large language model decoding layer is realized, solving the problems of difficult deployment and high latency caused by limited resources on the edge side. (2) This invention aims to minimize end-to-end inference latency, while simultaneously characterizing computation latency and cross-node communication latency. It introduces layer-by-layer quantization error evaluation and threshold screening to obtain a feasible bit width set, and proposes a bit width-aware contract matching method. Through proposal-temporary acceptance-rejection iteration, a stable matching result is obtained, thereby reducing the overall latency and improving deployment stability while ensuring inference quality. Attached Figure Description

[0049] Figure 1 is a framework diagram of the hierarchical perception quantization and distributed deployment system for large language models under cloud-edge collaboration in this invention.

[0050] Figure 2 is a diagram of the bit-width-aware contract matching algorithm based on matching theory in this invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings.

[0052] This invention discloses a method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration, comprising the following steps:

[0053] Combination Figure 1 The configuration model for the large language model to be deployed and the cloud-edge node configuration model mentioned in step 1 are as follows:

[0054] Determine the set of decoding layers for a large language model and inter-layer dependencies, and for any decoding layer Obtain its computational cost and the scale of intermediate activation dimensions At the same time, determine the set of candidate deployment nodes. and for any node Obtain its effective computing power With available video memory capacity And for any node pair Obtain link bandwidth ;in, Indicates the first The floating-point computation cost of the layer under single-token conditions. Indicates the first The activation dimension scale in the middle of the layer, Represents a node Effective computing power, Represents a node Available video memory capacity, Represents a node With nodes Available bandwidth of the link between them.

[0055] Step 2 describes the hierarchical quantitative evaluation model and the establishment of an end-to-end inference latency model:

[0056] This invention addresses the weights of each decoding layer l in a large language model. In candidate bit width Quantization weights are obtained The relative reconstruction error is used as the error index, and its definition is as follows:

[0057]

[0058] Furthermore, given a preset error threshold Construct the feasible bit-width set of the l-th layer. Its definition is as follows:

[0059]

[0060] In equation (1), The model number is represented by the first... The original weight matrix of the layer, Indicates the first Layer uses bit width Quantized weight matrix, Denotes the Frobenius norm. Indicates the first Layer in position width The relative quantization reconstruction error is given in equation (2). For the error threshold, For the candidate bit width set, For the first The feasible bit width set of the layer under the condition of satisfying error constraints; the feasible bit width set is used to constrain the search space of subsequent deployment and quantization decisions, thereby establishing a controllable boundary between compression ratio and inference quality.

[0061] This invention further establishes an end-to-end inference latency model, decomposing the end-to-end inference latency into two parts: computation latency and communication latency. The l-th layer uses a quantization bit width... The computational latency cost term deployed on node e is expressed as:

[0062]

[0063] In equation (3), Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates the first The floating-point computational cost of a single decoding layer under single-token conditions; This represents the number of tokens used in the reasoning. Represents a node Effective computing power; Indicates the first The quantization bit width selected for the layer, This represents the bit width scaling factor based on 16 bits.

[0064] The communication latency cost when deploying across nodes in adjacent layers is represented as follows:

[0065]

[0066] In equation (4), Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the size of the intermediate activation dimension; Represents a node With nodes Inter-link bandwidth; This represents the number of tokens used in the reasoning. Indicates the first The quantization bit width selected for the layer.

[0067] The specific steps for establishing the deployment-quantization joint optimization problem model in step 3 are as follows:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] in, Represents the set of model decoding layers. Represents the set of candidate edge deployment nodes. Indicates the first The selected quantization bit width for the layer Indicates the first The feasible bit-width set obtained after filtering by the error threshold; decision variables Indicates the first Is the layer deployed on the node? If deployed, then ,otherwise Decision variables Indicates adjacent layers Deployed on nodes With nodes Does cross-node communication occur during the process? If so, ,otherwise Indicates the first Layer by width At the node The computational delay cost term executed on the above, Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the first Layer in position width The video memory usage is as follows. Represents a node Available video memory capacity; constraint (6) ensures that each layer It must be deployed on one and only one node, that is, for any satisfy Constraint (7) ensures communication variables through inequality linkage. Consistent with the deployment choices of the two adjacent layers, i.e., only when the first layer... Layer deployment in And the Layer deployment in Only when allowed Set to 1; constraint (8) is a node capacity constraint, ensuring that nodes are assigned to the same node. The total memory usage of each layer at the corresponding bit width does not exceed the available capacity MemCap of the node; constraint (9) is a bit width feasibility constraint to ensure that the bit width involved in each layer is within the limit. It must come from its feasible bit-width set Constraint (10) is a variable value constraint, which specifies... and It is a binary variable;

[0075] Combination Figure 2 Step 4 above includes:

[0076] To uniformly express the selection of deployment nodes and quantization bit width for each layer, the concept of contract is introduced. For any node-bit width combination, the contract c=(e,b) is defined, which means that a certain layer is deployed to node e and inference is performed using bit width b, where e represents the edge candidate deployment node and b represents the quantization bit width.

[0077] For any model layer l, construct its set of optional contracts as follows:

[0078]

[0079] in, Indicates the first The contract set of the layer; Represents the set of candidate edge deployment nodes; Indicates the first The feasible bit-width set obtained after filtering by the error threshold;

[0080] A bilateral preference model is established to characterize the preference relationship between model layers and nodes for contracts. In the layer preference model, contracts are... For the The preference cost function of a layer is defined as:

[0081]

[0082] Based on this, the hierarchy preference order is defined as follows:

[0083]

[0084] in, Indicates the first The preference cost of choosing contract c at layer level; Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates starting from the node at the previous level. To candidate nodes The communication latency cost incurred in transmitting activation data; Indicates the first The preference relationship between layers for contracts;

[0085] In the node preference model, nodes Accepting the contract The preference cost is defined as:

[0086]

[0087] Based on this, the node preference order is defined as follows:

[0088]

[0089] in This represents the inbound communication pressure cost borne by node e when it accepts contract c; This represents the preference relationship of node e for the contract.

[0090] The contract matching algorithm process described in step 4 is as follows:

[0091] Initialize all model layers to an unmatched state. When an unmatched model layer exists and its preference list is not empty, that model layer initiates a proposal request to its most preferred contract in order of preference, where the contract consists of a combination of candidate deployment nodes and quantization bit width. The node receiving the proposal collects all proposals requested to be deployed to that node and filters the proposals based on the node's available resource constraints. When all collected proposals satisfy the node's resource constraints, the node temporarily accepts all proposals; otherwise, the node only retains the best proposal that satisfies the constraints and rejects the rest. The rejected model layer removes the rejected contract from its preference list and continues to initiate a proposal to the next preferred contract. Repeat the above proposal, temporary acceptance, and rejection process until there are no unmatched model layers with non-empty preference lists. Output the final accepted contract corresponding to each model layer, thus obtaining a stable mapping relationship between model layers and contracts. The algorithm pseudocode is defined as follows:

[0092]

[0093] The above description outlines the implementation process and advantages of this invention. Those skilled in the art should understand that various changes and modifications can be made to this invention without departing from its principles, and all such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration, characterized in that, Includes the following steps: Step 1: Establish the configuration model for the large language model to be deployed and the configuration model for the cloud-edge nodes; Step 2: Establish a hierarchical quantitative evaluation model and an end-to-end inference latency model; Step 3: With the goal of minimizing end-to-end inference latency, establish a deployment-quantization joint optimization problem model under the conditions of satisfying node resource constraints and link bandwidth constraints. The optimization problem is expressed as: st C1: C2: C3: C4: C5: in, Represents the set of model decoding layers. Represents the set of candidate edge deployment nodes. Indicates the first The selected quantization bit width for the layer Indicates the first The feasible bit-width set obtained after filtering by the error threshold; decision variables Indicates the first Is the layer deployed on the node? If deployed, then ,otherwise Decision variables Indicates adjacent layers Deployed on nodes With nodes Does cross-node communication occur during the process? If so, ,otherwise Indicates the first Layer by width At the node The computational delay cost term executed on the above, Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the first Layer in position width The video memory usage is as follows. Represents a node Available video memory capacity; constraint C1 ensures that each layer It must be deployed on one and only one node, that is, for any satisfy Constraint C2 ensures communication variables through inequality linkage. Consistent with the deployment choices of the two adjacent layers, i.e., only when the first layer... Layer deployment in And the Layer deployment in Only when allowed Set to 1; Constraint C3 is a node capacity constraint, ensuring that nodes are assigned to the same node. The total video memory usage of each layer at its corresponding bit width does not exceed the available capacity MemCap of that node; constraint C4 is a bit width feasibility constraint, ensuring that the bit width involved in each layer... It must come from its feasible bit-width set Constraint C5 is a variable value constraint, specifying... and It is a binary variable; Step 4: Solve the joint optimization problem. Use a bit-width-aware contract matching algorithm to generate deployment and quantization schemes. To uniformly express the deployment node and quantization bit-width selection of each layer, the concept of contract is introduced. For any node-bit-width combination, the contract c=(e,b) is defined, which means that a certain layer is deployed to node e and inference is performed using bit-width b, where e represents the edge candidate deployment node and b represents the quantization bit-width. For any model layer l, construct its set of optional contracts as follows: in, Indicates the first The contract set of the layer; Represents the set of candidate edge deployment nodes; Indicates the first The feasible bit-width set obtained after filtering by the error threshold; A bilateral preference model is established to characterize the preference relationship between model layers and nodes for contracts. In the layer preference model, contracts are... For the The preference cost function of a layer is defined as: Based on this, the hierarchy preference order is defined as follows: in, Indicates the first The preference cost of choosing contract c at layer level; Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates starting from the node at the previous level. To candidate nodes The communication latency cost incurred in transmitting activation data; Indicates the first The preference relationship between layers for contracts; In the node preference model, nodes Accepting the contract The preference cost is defined as: Based on this, the node preference order is defined as follows: in This represents the inbound communication pressure cost borne by node e when it accepts contract c; This represents the preference relationship of node e for the contract; During the matching process, model layers that do not match and whose preference lists are not empty send proposals to the edge candidate deployment nodes corresponding to their most preferred contracts in the order of preference. The edge candidate deployment nodes temporarily accept or reject the proposals based on node resource constraints. The rejected model layers continue to send proposals to the next preferred contract. Through the iterative process of proposal, temporary acceptance and rejection, a stable matching result is obtained. Step 5: Perform distributed sequential inference based on the stable matching result, so that each decoding layer loads the corresponding bit-width quantization weight on its matching node to complete the calculation, and transmits intermediate activation data when adjacent decoding layers match different nodes to complete end-to-end inference.

2. The method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration as described in claim 1, characterized in that... The steps in step 1, which involve establishing the configuration model for the large language model to be deployed and the cloud-edge node configuration model, are as follows: Determine the set of decoding layers for a large language model and inter-layer dependencies, and for any decoding layer Obtain its computational cost and the scale of intermediate activation dimensions At the same time, determine the set of candidate deployment nodes. and for any node Obtain its effective computing power With available video memory capacity And for any node pair Obtain link bandwidth ;in, Indicates the first The floating-point computation cost of the layer under single-token conditions. Indicates the first The activation dimension scale in the middle of the layer, Represents a node Effective computing power, Represents a node Available video memory capacity, Represents a node With nodes Available bandwidth of the link between them.

3. The method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration as described in claim 1, characterized in that... The hierarchical quantitative evaluation model described in step 2 is as follows: For the Layer weights In candidate bit width Quantization weights are obtained And using relative reconstruction error as the error index, the error index is defined as follows: : Indicates the first The original weight matrix of the layer, Indicates the first Layer in position width The quantization weight matrix below, Denotes the Frobenius norm; Construct the feasible bit-width set of the l-th layer based on a preset error threshold. : Represents the set of candidate quantization bit widths. This represents the error threshold, used to filter out candidate bit widths that do not meet the accuracy requirements.

4. The method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration as described in claim 1, characterized in that... The specific steps for establishing the end-to-end inference latency model in step 2 are as follows: The end-to-end inference latency is expressed as the sum of computation latency and communication latency, where the l-th layer uses the quantization bit width. The computational latency cost term deployed on node e is expressed as: in, Indicates the first Layer by width At the node The computational delay cost item for the above execution; Indicates the first The floating-point computational cost of a single decoding layer under single-token conditions; This represents the number of tokens used in the reasoning. Represents a node Effective computing power; Indicates the first The quantization bit width selected for the layer, This represents the bit width scaling factor based on 16 bits; The communication latency cost when deploying across nodes in adjacent layers is represented as follows: in, Indicates the first Layer output activation at node With nodes The communication delay cost incurred during transmission between them; Indicates the size of the intermediate activation dimension; Represents a node With nodes Inter-link bandwidth; This represents the number of tokens used in the reasoning. Indicates the first The quantization bit width selected for the layer.

5. The method for hierarchical perceptual quantization and distributed deployment of large language models under cloud-edge collaboration as described in claim 1, characterized in that... Step 4, the specific process of the bit-width-aware contract matching algorithm, is as follows: Initialize all model layers to an unmatched state. When an unmatched model layer exists and its preference list is not empty, that model layer initiates a proposal request to its most preferred contract in order of preference, where the contract consists of a combination of deployment nodes and quantization bit width. The node to which the proposal is being made collects all proposals requested to be deployed to it and filters the proposals based on the node's available resource constraints and the currently temporarily accepted set of proposals. When all collected proposals satisfy the node's resource constraints, the node temporarily accepts all proposals; otherwise, the node only retains the best proposal that satisfies the constraints and rejects the rest. The rejected model layer removes the rejected contract from its preference list and continues to initiate a proposal to the next preferred contract. Repeat the above proposal, temporary acceptance, and rejection process until there are no unmatched model layers and the preference list is not empty. Output the final accepted contract corresponding to each model layer, thereby obtaining a stable mapping relationship between layers and contracts.

Images