Inference model partitioning method for serverless application scenarios based on self-attention

Abstract

The present invention discloses an inference model partitioning method for a serverless application scenario based on self-attention, the method comprising: step 1: extracting performance data of operators in each sub-inference unit ISU of a composite inference model CIM, and merging and preprocessing the operator performance data; step 2: constructing a resource prediction model based on XGBoost; step 3: hierarchically merging and summarizing the data output by the resource prediction model based on XGBoost to obtain performance data of the sub-inference units; step 4: based on the self-attention mechanism, analyzing the performance data of the sub-inference units to obtain the inter-layer dependency of the composite inference model, and generating a partitioning strategy; step 5: partitioning the composite inference model according to the partitioning strategy, encapsulating the sub-module corresponding to each partition as a serverless function, and completing the independent deployment partition.

Description

Technical Field

[0001] The present invention belongs to the field of computer artificial intelligence, and in particular relates to an inference model partitioning method for serverless application scenarios based on self-attention. Background Art

[0002] With the development of cloud computing, serverless computing has gained widespread popularity due to its pay-as-you-go and automatic scalability features. In the current era of widespread artificial intelligence and the Internet of Things, machine learning is finding increasingly widespread application. However, machine learning requires enormous computing power and fluctuates during the process, especially for deep learning inference services. Serverless computing, with its pay-as-you-go and automatic scalability features, is a perfect fit for this scenario, reducing costs and improving resource utilization.

[0003] However, there are several problems with serverless deep learning inference:

[0004] 1) Difficulty in deployment: Serverless platforms limit the size of deployment packages. Large inference models are difficult to deploy as a whole on serverless platforms because their size exceeds the platform limit. Even if small inference models can be deployed as a whole, fluctuations in resource demand will lead to low resource utilization and cost waste.

[0005] 2) Difficulty in Partitioning: Large deep learning inference models are difficult to deploy as a whole in a serverless environment. Therefore, the model needs to be partitioned, with each component encapsulated as a serverless function and then deployed. However, this partitioning requires consideration of multiple factors, including serverless platform limitations, resource utilization, model execution performance, and overall cost, making the mathematical modeling process complex.

[0006] Existing research on serverless deep learning inference model partitioning uses dynamic programming as the main partitioning method, and uses the model's external performance factors as auxiliary judgment basis, such as resource occupancy, execution time and total cost constraints, and cooperates with other partitioning methods to improve model partitioning, such as enumeration method, reinforcement learning and prediction model.

[0007] However, these classification methods based on external performance factors still have the following shortcomings:

[0008] 1) Limited model flexibility: Partitioning methods based on external performance factors typically rely on the model's performance under specific inputs (such as memory usage and execution time). These factors are often static and lack adaptability to dynamic changes within the model during inference. This means that the partitioning strategy may not be effectively adjusted when faced with different input data or changes in model parameters, resulting in limited model flexibility and adaptability. This is especially true in serverless environments, where models need to adjust dynamically based on runtime resource changes. Therefore, partitioning methods that rely solely on external performance factors cannot meet the requirements of real-time performance and dynamic optimization.

[0009] 2) Ignoring Internal Model Dependencies: Existing methods primarily rely on external performance to determine layer partitioning strategies. However, these external factors do not fully reflect the computational dependencies within the inference model. Deep learning models exhibit complex computational dependencies and information flows between layers, yet these intrinsic characteristics are often overlooked. Traditional partitioning methods based on external factors may fail to accurately capture these inter-layer dependencies, leading to illogical partitioning, increased cross-node communication overhead, and reduced inference efficiency. Therefore, relying solely on external performance factors for partitioning makes it difficult to achieve more accurate and efficient resource allocation. Summary of the Invention

[0010] In response to the shortcomings of the current existing technology, this paper proposes an inference model partitioning method for serverless application scenarios based on self-attention. This method aims to improve the flexibility and adaptability of model partitioning and optimize the internal computational dependencies of the model, so as to more efficiently allocate resources and reduce computing and communication overhead.

[0011] To this end, the present invention provides the following technical solutions:

[0012] On the one hand, an inference model partitioning method for serverless application scenarios based on a self-attention mechanism includes:

[0013] Step 1: Extract the performance data of operators in each inference subunit (ISU) of the composite inference model (CIM), merge the operator performance data, and preprocess it.

[0014] Step 2: Build a resource prediction model based on XGBoost;

[0015] The input data of the XGBoost-based resource prediction model consists of the operator-level feature dataset OLFD (Operator-Level Feature Dataset, OLFD) and the sub-inference unit ISU layer feature dataset ILFD (IL-ISU Layer-Level Feature Dataset, ILFD). OLFD is the preprocessed data obtained in step 1, and ILFD is the hierarchical feature data of the internal layer of ISU (IL-ISU) of the sub-inference unit extracted and calculated from OLFD. The output data is the operator-level predicted dataset (OLPD-X), which includes the predicted resource usage time and execution time of each operator.

[0016] Step 3: Hierarchically merge and summarize the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units;

[0017] Step 4: Based on the self-attention mechanism, the performance data of the sub-inference units is analyzed to obtain the inter-layer dependencies of the composite inference model and generate a partitioning strategy;

[0018] Step 5: Partition the composite inference model based on the partitioning strategy, encapsulate the submodule corresponding to each partition into a serverless function, and complete the independent deployment of the partition.

[0019] In this solution, the composite inference model (CIM) consists of multiple composite inference layers (CILs). Each composite inference layer (CIL) includes two types of computing units: the inference sub-unit (ISU) and the general computing layer (GCL). Each sub-inference unit (ISU) includes multiple internal sub-unit layers (ILs) (ISUs). Partitioning essentially combines multiple CILs into a partitioned inference sub-module (PIM). Since the computational complexity of the GCL is generally smaller than that of the ISU, its impact on the overall partitioning scheme is far less critical than that of the ISU. Therefore, when calculating the performance data of the CIL, it is only necessary to calculate the performance data of all the ISUs contained in the CIL.

[0020] Starting from the internal dependencies and resource usage of the model, combined with the XGBoost prediction model and self-attention mechanism, the adaptability and execution efficiency of serverless inference model partitions are improved.

[0021] Furthermore, the operator-level feature dataset OLFD includes the input / output data size at the operator level, the resource occupancy and execution time monitoring value of the operator, and the complexity score of the operator; the hierarchical feature dataset ILFD of the sub-inference unit ISU includes two parts: structural features and computing resource features. The structural features include the layer type of IL-ISU, the number of layer parameters of IL-ISU, and the number of input / output dimensions of IL-ISU. The computing resource features include the input / output data size of IL-ISU, the shape ratio of the input and output tensors of IL-ISU, the total data volume of the input and output of IL-ISU, the input-output product of IL-ISU, the average size of each dimension, and the complexity score of IL-ISU.

[0022] The input data is operator-level data, and the output is also operator-level prediction data. Then, it is merged into hierarchical data based on the connection structure between operators, and finally summarized into the performance data of the underlying model.

[0023] Furthermore, the merging of the operator performance data refers to integrating the operator performance data into the layer performance data of the sub-unit internal layer IL-ISU and the overall performance data of the sub-inference unit ISU according to the operator combination structure. The operator combination structure includes three types: sequential structure, parallel structure and hybrid structure. Among them, the hybrid structure refers to the superposition and combination of the sequential structure and the parallel structure.

[0024] The operator performance data structure is as follows:

[0025] .

[0026] in, represents the first Operator, Representation operator The name of the sub-reasoning unit ISU to which it belongs, Representation operator The name of the subunit internal layer IL-ISU of the sub-inference unit ISU, Represents the input operator The data size, Representation operator The number of parameters of the internal layer IL-ISU of the subunit, Representation operator Memory usage during operation, Representation operator The length of execution time during the run;

[0027] When the operator combination structure in the sub-unit internal layer IL-ISU is a sequential structure, the memory usage of the sub-unit internal layer IL-ISU is The maximum memory usage among all operators in this layer, and the execution time of the IL-ISU layer within the sub-unit The sum of the execution time of all operators in this layer;

[0028] When the operator combination structure in the sub-unit internal layer IL-ISU is a parallel structure, the memory usage of the sub-unit internal layer IL-ISU is is the maximum value of the combined memory usage at each position, and the execution time of the IL-ISU layer within the subunit It is the sum of the execution time of the combination at each position, and different branches at the same position Combined memory usage of , combined execution time , n represents the number of parallel branches, represents the execution time of the jth operator in the nth branch, Indicates the The first of the branches Operator, operator The memory usage and execution time of and ;

[0029] When the operator combination structure in the IL-ISU layer of the sub-unit is a hybrid structure, the memory usage of the IL-ISU layer of the sub-unit is The maximum memory usage of the sequential part and the parallel part, the execution time of the sub-unit internal layer IL-ISU is the sum of the execution time of the sequential part and the parallel part;

[0030] For those with The performance data of the sub-reasoning unit ISU of the layer IL-ISU within the layer sub-unit is expressed as the collection of the performance data of the layer IL-ISU within each sub-unit.

[0031] Furthermore, the process of building a resource prediction model based on XGBoost is as follows:

[0032] Data preparation: Extract the preprocessed data obtained in step 1;

[0033] Objective function setting: The objective function is to minimize the mean square error of each layer's memory usage and execution time;

[0034] These are two independent objective functions, training two independent models to predict memory usage and execution time respectively;

[0035] Model training: Use the data in the training set to train the XGBoost model with the set objective function to obtain a trained XGBoost-based resource prediction model;

[0036] The process of building a resource prediction model based on XGBoost is as follows:

[0037] The data in the training set are the pre-processed data (OLFD) and feature data (ILFD) extracted from each sub-inference unit ISU in the known composite inference model CIM according to step 1;

[0038] During the training process, grid search or Bayesian optimization is used to adjust the XGBoost model hyperparameters until the prediction accuracy of the XGBoost model meets the set requirements.

[0039] Furthermore, hierarchical merging and aggregation of the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units refers to:

[0040] According to the operator combination structure, the hierarchical prediction datasets (ILPD) of the sub-unit internal layer IL-ISU are first merged, and then the hierarchical prediction datasets of the sub-unit internal layer IL-ISU are merged to obtain the prediction dataset (IOPD) of the sub-inference unit ISU. The prediction data of all sub-inference units ISU belonging to the same composite inference layer CIL are unioned as the hierarchical performance data of the composite inference layer CIL.

[0041] The hierarchical prediction data sets of the sub-unit internal layer IL-ISU are merged into the overall performance data of the sub-inference unit ISU according to the following formula:

[0042] .

[0043] in, Indicates the memory usage of the 1st, 2nd, ..., Lth layer subunit internal layer IL-ISU, represents the execution time of the inner layer IL-ISU of the 1st, 2nd, ..., Lth layer sub-units, M and T represent the memory usage and execution time of the sub-inference unit ISU, respectively.

[0044] Furthermore, the specific process of analyzing the performance data of the sub-reasoning units based on the self-attention mechanism to obtain the inter-layer dependencies of the composite reasoning model and generate the partitioning strategy is as follows:

[0045] S1: Construct the inter-layer dependency feature vector of the composite reasoning layer CIL;

[0046] Based on the obtained composite reasoning layer CIL hierarchical data, an inter-layer dependency feature vector is constructed for each composite reasoning layer CIL, which includes the resource prediction data of the composite reasoning layer CIL itself, the structural characteristics of the composite reasoning layer CIL, and the inter-layer connection relationship of the composite reasoning layer CIL;

[0047] The structural characteristics of the composite inference layer (CIL) include the CIL layer type, the input and output data sizes of the CIL layer, and the number of CIL layer parameters. The connection relationship between the composite inference layer CIL layers is determined by extracting the computational graph and generating a dependency matrix using tools provided by PyTorch or TensorFlow.

[0048] S2: Calculate the attention weights between the composite inference layer CIL layers;

[0049] The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model to calculate the attention weights between the layers of the composite inference layer CIL. The specific process includes the following:

[0050] a) Generation of Query, Key, and Value;

[0051] The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model through linear transformation to generate the query vector Query, key vector Key and value vector Value between each composite inference layer CIL layer;

[0052] These vectors are automatically generated from the attention mechanism and are used to calculate the similarity between layers.

[0053] b) Calculation of attention weights between a composite inference layer CIL and other composite inference layers CIL;

[0054] .

[0055] in, is the feature dimension, Represents the similarity between the query and the key, V represents the value vector between a composite reasoning layer CIL and other composite reasoning layer CILs;

[0056] c) Generate attention matrix: The attention weights of each composite inference layer CIL and other composite inference layers CIL form a matrix, which represents the strength of the inter-layer dependency;

[0057] S3: Identify key layers and dependency paths;

[0058] According to the attention weights between each composite reasoning layer CIL and other composite reasoning layer CILs in the attention matrix, the inter-layer dependency of composite reasoning layer CILs within the CIM is obtained, and the closely dependent composite reasoning layer CIL layer groups and critical paths are determined. The closely dependent composite reasoning layer CILs and the composite reasoning layer CILs on the critical path are divided into the same partition.

[0059] Furthermore, the specific process of determining the layer groups and critical paths of the closely dependent composite reasoning layer CIL is as follows:

[0060] a) Identification of closely dependent layer groups: The attention weights in the attention matrix are used to determine the dependency strength between layers in the composite inference layer (CIL). Layers that exceed the attention weight threshold are considered to be closely dependent layer groups.

[0061] The elements in the attention matrix represent the attention weights between CIL layers. When the value of an element exceeds the attention weight threshold, it indicates that there is a strong computational dependency between the CIL layers corresponding to the element.

[0062] The attention weight threshold can be determined by:

[0063] ① Rule of thumb: Start with 0.5 as the initial threshold.

[0064] ②Data-driven: Analyze the distribution of attention weights and select points that can effectively distinguish high and low dependencies.

[0065] ③Task requirements: Adjust the threshold according to the requirements of the specific task.

[0066] ④ Experimental verification: Verify the impact of different thresholds on model performance through experiments and select the appropriate threshold.

[0067] b) Critical path identification: By analyzing the global attention distribution of the attention matrix, the sum of the attention weights between the composite inference layers (CIL) that the data passes through from the input to the composite inference model to the output is calculated, and the path with the highest sum of attention weights is selected as the critical path.

[0068] Furthermore, in the attention weight matrix, find the bottleneck layer CIL whose attention weights with other CIL layers exceed the bottleneck layer threshold, and adjust the partition according to the bottleneck layer CIL;

[0069] The specific constraints are as follows:

[0070] ①The number of bottlenecks in each partition shall not exceed , The bottleneck layer threshold is the maximum number of bottleneck layers allowed in a single partition, and is initially set to 2.

[0071] ② According to the IOPD data set, the total memory usage of the bottleneck layer shall not exceed , the pre-allocated memory for each partition is .

[0072] Bottleneck layer concentration constraint: For bottleneck layers with high computing resource consumption, avoid concentrating multiple bottleneck layers in the same partition to prevent the computing load of a specific partition from being too high, thereby affecting the overall inference performance.

[0073] Furthermore, the steps to optimize the partitioning strategy are as follows:

[0074] a) Monitor partition operation in a simulated environment;

[0075] ① Monitor the memory usage and execution time of the partition blocks to ensure that the load of the partition blocks does not exceed 90% and the standard deviation of the memory usage does not exceed 10%. At the same time, check whether the execution time exceeds the predicted maximum time and whether the standard deviation of the execution time exceeds 10%.

[0076] The numerical settings are all for judging the initial starting point, and they will be adjusted and optimized according to the specific situation;

[0077] ② Monitor the communication volume and frequency between partition blocks to ensure that the communication volume does not exceed 10% and the communication frequency does not exceed 10 times per second;

[0078] b) Evaluate partition memory usage and execution time;

[0079] ① Check whether the memory usage of the partition block exceeds 90%. If so, split the current partition block to reduce the memory usage;

[0080] ② Calculate whether the standard deviation of the memory usage of the partition block exceeds 10%. If so, split the current high memory usage partition block and merge the low memory usage partition to reduce memory usage fluctuations;

[0081] c) assess the communication situation;

[0082] ① Check whether the communication volume between partition block PIMs exceeds 10% of the total data volume. If so, reduce the communication volume by merging partitions or optimizing data transmission.

[0083] ② Check whether the communication frequency between partition block PIMs exceeds 10 times per second. If so, merge partitions to reduce the communication frequency.

[0084] The split and merge operations are as follows:

[0085] ① Split: According to the attention matrix, the CIL part of the continuous composite reasoning layer with the strongest dependency in the partition to be split is retained as one partition, and the remaining CIL parts of the composite reasoning layer are merged into the adjacent partitions;

[0086] ②Merge: According to the attention matrix, the partitions to be merged are merged into the partitions where the composite inference layer CIL with which they have a stronger dependency is located in the adjacent partitions.

[0087] In another aspect, a computer device comprises:

[0088] one or more processors;

[0089] a memory storing one or more computer programs;

[0090] The processor calls the computer program to implement:

[0091] The above steps are for an inference model partitioning method for a serverless application scenario based on a self-attention mechanism.

[0092] The technical solution of the present invention provides an inference model partitioning method for a serverless application scenario based on a self-attention mechanism, including step 1: extracting the performance data of operators in each sub-inference unit ISU of a composite inference model CIM; step 2: constructing a resource prediction model based on XGBoost; step 3: hierarchically merging and summarizing the data output by the resource prediction model based on XGBoost to obtain the performance data of the sub-inference unit; step 4: based on the self-attention mechanism, analyzing the performance data of the sub-inference unit to obtain the inter-layer dependency of the composite inference model and generating a partitioning strategy; step 5: partitioning the composite inference model according to the partitioning strategy, encapsulating the sub-module corresponding to each partition as a serverless function, and completing the independent deployment of the partition.

[0093] Compared with the existing technology, it has the following advantages:

[0094] The constructed XGBoost resource prediction model generates new trees by gradually optimizing the gradient information of the objective function to reduce the value of the loss function and ensure the high accuracy of the model; L1 and L2 regularization terms are introduced to avoid model overfitting; and custom objective functions are supported: users can design objective functions according to specific needs to improve adaptability to different tasks.

[0095] In the case of partitioned inference models, the self-attention mechanism can analyze the computational dependencies and information flow between model layers by modeling the dependencies within the model's internal structure. Based on this analysis, the partitioning strategy can be optimized to assign closely related layers to the same partition, reducing communication overhead while balancing resource allocation.

[0096] The self-attention mechanism generates an attention matrix and calculates the strength of connections between model layers, thereby identifying dependencies between layers. These dependencies can be used to intuitively analyze which layers need to work closely together and how to partition them to optimize execution efficiency.

[0097] This method aims to improve the flexibility and adaptability of model partitioning and optimize the internal computational dependencies of the model to more efficiently allocate resources and reduce computational and communication overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a schematic diagram of the process of the technical solution of the present invention;

[0099] Figure 2 This is the LSTM model operator structure diagram. DETAILED DESCRIPTION

[0100] The embodiment of the present invention provides a method for partitioning an inference model for a serverless application scenario based on a self-attention mechanism, such as Figure 1 As shown, including:

[0101] Step 1: Extract the performance data of operators in each inference subunit (ISU) of the composite inference model (CIM), merge the operator performance data, and preprocess it.

[0102] Step 2: Build a resource prediction model based on XGBoost;

[0103] The input data of the XGBoost-based resource prediction model consists of the operator-level feature dataset OLFD (Operator-Level Feature Dataset, OLFD) and the sub-inference unit ISU layer feature dataset ILFD (IL-ISU Layer-Level Feature Dataset, ILFD). OLFD is the preprocessed data obtained in step 1, and ILFD is the hierarchical feature data of the internal layer of ISU (IL-ISU) of the sub-inference unit extracted and calculated from OLFD. The output data is the operator-level predicted dataset (OLPD-X), which includes the predicted resource usage time and execution time of each operator.

[0104] The operator-level feature dataset OLFD includes the input / output data size at the operator level, the resource occupancy and execution time monitoring value of the operator, and the complexity score of the operator; the sub-inference unit ISU level feature dataset ILFD includes two parts: structural features and computing resource features. The structural features include the layer type of IL-ISU, the number of layer parameters of IL-ISU, and the number of input / output dimensions of IL-ISU. The computing resource features include the input / output data size of IL-ISU, the shape ratio of the input and output tensors of IL-ISU, the total data volume of the input and output of IL-ISU, the input-output product of IL-ISU, the average size of each dimension, and the complexity score of IL-ISU.

[0105] The input data is operator-level data, and the output is also operator-level prediction data. Then, it is merged into hierarchical data based on the connection structure between operators, and finally summarized into the performance data of the underlying model.

[0106] Merging the performance data of operators means integrating the operator performance data into the layer performance data of the sub-unit internal layer IL-ISU and the overall performance data of the sub-inference unit ISU according to the operator combination structure. The operator combination structure is as follows: Figure 2As shown, there are three types: sequential structure, parallel structure and hybrid structure, among which the hybrid structure refers to the superposition and combination of sequential structure and parallel structure;

[0107] The performance data structure of the operator is as follows:

[0108] .

[0109] in, represents the first Operator, Representation operator The name of the sub-reasoning unit ISU to which it belongs, Representation operator The name of the subunit internal layer IL-ISU of the sub-inference unit ISU, Represents the input operator The data size, Representation operator The number of parameters of the internal layer IL-ISU of the subunit, Representation operator Memory usage during operation, Representation operator The length of execution time during the run;

[0110] When the operator combination structure in the sub-unit internal layer IL-ISU is a sequential structure, the memory usage of the sub-unit internal layer IL-ISU is The maximum memory usage among all operators in this layer, and the execution time of the IL-ISU layer within the sub-unit The sum of the execution time of all operators in this layer;

[0111] That is, the operator performance data is integrated into IL-ISU level performance data according to the following formula: Indicates a certain IL-ISU layer within the ISU:

[0112] .

[0113] When the operator combination structure in the sub-unit internal layer IL-ISU is a parallel structure, the memory usage of the sub-unit internal layer IL-ISU is is the maximum value of the combined memory usage at each position, and the execution time of the IL-ISU layer within the subunit It is the sum of the execution time of the combination at each position, and different branches at the same position Combined memory usage of , combined execution time , n represents the number of parallel branches, represents the execution time of the jth operator in the nth branch, Indicates the The first of the branches Operator, operator The memory usage and execution time of and ;

[0114] That is, the operator performance data is integrated into IL-ISU level performance data according to the following formula:

[0115] .

[0116] k represents the longest branch length among the parallel branches;

[0117] When the operator combination structure in the IL-ISU layer of the sub-unit is a hybrid structure, the memory usage of the IL-ISU layer of the sub-unit is The maximum memory usage of the sequential part and the parallel part, the execution time of the sub-unit internal layer IL-ISU is the sum of the execution time of the sequential part and the parallel part;

[0118] That is, the operator performance data is integrated into IL-ISU level performance data according to the following formula:

[0119] .

[0120] For those with The performance data of the sub-reasoning unit ISU of the layer IL-ISU within the layer sub-unit is expressed as the collection of the performance data of the layer IL-ISU within each sub-unit.

[0121] The process of building a resource prediction model based on XGBoost is as follows:

[0122] Data preparation: Extract the preprocessed data obtained in step 1;

[0123] Objective function setting: The objective function is to minimize the mean square error of each layer's memory usage and execution time;

[0124] These are two independent objective functions, training two independent models to predict memory usage and execution time respectively;

[0125] Model training: Use the data in the training set to train the XGBoost model with the set objective function to obtain a trained XGBoost-based resource prediction model;

[0126] The process of building a resource prediction model based on XGBoost is as follows:

[0127] The data in the training set are the pre-processed data (OLFD) and feature data (ILFD) extracted from each sub-inference unit ISU in the known composite inference model CIM according to step 1;

[0128] During the training process, grid search or Bayesian optimization is used to adjust the XGBoost model hyperparameters until the prediction accuracy of the XGBoost model meets the set requirements.

[0129] Step 3: Hierarchically merge and summarize the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units;

[0130] Hierarchical merging and aggregation of the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units means:

[0131] According to the operator combination structure, the hierarchical prediction datasets (ILPD) of the sub-unit internal layer IL-ISU are first merged, and then the hierarchical prediction datasets of the sub-unit internal layer IL-ISU are merged to obtain the prediction dataset (IOPD) of the sub-inference unit ISU. The prediction data of all sub-inference units ISU belonging to the same composite inference layer CIL are unioned as the hierarchical performance data of the composite inference layer CIL.

[0132] The hierarchical prediction data sets of the sub-unit internal layer IL-ISU are merged into the overall performance data of the sub-inference unit ISU according to the following formula:

[0133] .

[0134] in, Indicates the memory usage of the 1st, 2nd, ..., Lth layer subunit internal layer IL-ISU, represents the execution time of the inner layer IL-ISU of the 1st, 2nd, ..., Lth layer sub-units, M and T represent the memory usage and execution time of the sub-inference unit ISU, respectively.

[0135] Step 4: Based on the self-attention mechanism, the performance data of the sub-inference units is analyzed to obtain the inter-layer dependencies of the composite inference model and generate a partitioning strategy;

[0136] The specific process of analyzing the performance data of the sub-reasoning units based on the self-attention mechanism to obtain the inter-layer dependencies of the composite reasoning model and generate the partitioning strategy is as follows:

[0137] S1: Construct the inter-layer dependency feature vector of the composite reasoning layer CIL;

[0138] Based on the obtained composite reasoning layer CIL hierarchical data, an inter-layer dependency feature vector is constructed for each composite reasoning layer CIL, which includes the resource prediction data of the composite reasoning layer CIL itself, the structural characteristics of the composite reasoning layer CIL, and the inter-layer connection relationship of the composite reasoning layer CIL;

[0139] The structural characteristics of the composite inference layer (CIL) include the CIL layer type, the input and output data sizes of the CIL layer, and the number of CIL layer parameters. The connection relationship between the composite inference layer CIL layers is determined by extracting the computational graph and generating a dependency matrix using tools provided by PyTorch or TensorFlow.

[0140] S2: Calculate the attention weights between the composite inference layer CIL layers;

[0141] The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model to calculate the attention weights between the layers of the composite inference layer CIL. The specific process includes the following:

[0142] a) Generation of Query, Key, and Value;

[0143] The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model through linear transformation to generate the query vector Query, key vector Key and value vector Value between each composite inference layer CIL layer;

[0144] These vectors are automatically generated from the attention mechanism and are used to calculate the similarity between layers.

[0145] b) Calculation of attention weights between a composite inference layer CIL and other composite inference layers CIL;

[0146] .

[0147] in, is the feature dimension, Represents the similarity between the query and the key, V represents the value vector between a composite reasoning layer CIL and other composite reasoning layer CILs;

[0148] c) Generate attention matrix: The attention weights of each composite inference layer CIL and other composite inference layers CIL form a matrix, which represents the strength of the inter-layer dependency;

[0149] S3: Identify key layers and dependency paths;

[0150] According to the attention weights between each composite reasoning layer CIL and other composite reasoning layer CILs in the attention matrix, the inter-layer dependency of composite reasoning layer CILs within the CIM is obtained, and the closely dependent composite reasoning layer CIL layer groups and critical paths are determined. The closely dependent composite reasoning layer CILs and the composite reasoning layer CILs on the critical path are divided into the same partition.

[0151] Assume the attention matrix For one The matrix of Presentation Layer Layer The attention weight of .

[0152] .

[0153] The weight path accumulation formula is: ,in Indicates the current path.

[0154] Set a high weight initial threshold , the high-weight layer can be expressed as .

[0155] Layer The total attention weight connected to the high-weight layer is .

[0156] The CIL connecting multiple high-weight layers is the bottleneck layer, which can be expressed as , where the initial connectivity threshold ,express After sorting from smallest to largest, As the boundary, the first 75% of the value is Small, the last 25% of the value is smaller than big.

[0157] The specific process of determining the layer groups and critical paths of the closely dependent composite reasoning layer CIL is as follows:

[0158] b) Identification of closely dependent layer groups: The dependency strength between layers of the composite inference layer (CIL) is determined by the attention weights in the attention matrix. Layers exceeding the attention weight threshold are considered to be closely dependent layer groups.

[0159] The elements in the attention matrix represent the attention weights between CIL layers. When the value of an element exceeds the attention weight threshold, it indicates that there is a strong computational dependency between the CIL layers corresponding to the element.

[0160] The attention weight threshold can be determined by:

[0161] ① Rule of thumb: Start with 0.5 as the initial threshold.

[0162] ②Data-driven: Analyze the distribution of attention weights and select points that can effectively distinguish high and low dependencies.

[0163] ③Task requirements: Adjust the threshold according to the requirements of the specific task.

[0164] ④ Experimental verification: Verify the impact of different thresholds on model performance through experiments and select the appropriate threshold.

[0165] b) Critical path identification: By analyzing the global attention distribution of the attention matrix, the sum of the attention weights between the composite inference layers (CIL) that the data passes through from the input to the composite inference model to the output is calculated, and the path with the highest sum of attention weights is selected as the critical path.

[0166] In the attention weight matrix, find the bottleneck layer CIL whose attention weights with other CIL layers exceed the bottleneck layer threshold, and adjust the partition according to the bottleneck layer CIL;

[0167] The specific constraints are as follows:

[0168] ①The number of bottlenecks in each partition shall not exceed , The bottleneck layer threshold is the maximum number of bottleneck layers allowed in a single partition, and is initially set to 2.

[0169] ② According to the IOPD data set, the total memory usage of the bottleneck layer shall not exceed , the pre-allocated memory for each partition is .

[0170] Bottleneck layer concentration constraint: For bottleneck layers with high computing resource consumption, avoid concentrating multiple bottleneck layers in the same partition to prevent the computing load of a specific partition from being too high, thereby affecting the overall inference performance.

[0171] Step 5: Partition the composite inference model based on the partitioning strategy, encapsulate the submodule corresponding to each partition into a serverless function, and complete the independent deployment of the partition.

[0172] The steps to optimize the partitioning strategy are as follows:

[0173] a) Monitor partition operation in a simulated environment;

[0174] ① Monitor the memory usage and execution time of the partition blocks to ensure that the load of the partition does not exceed 90% and the standard deviation of the memory usage does not exceed 10%. At the same time, check whether the execution time exceeds the predicted maximum time and whether the standard deviation of the execution time exceeds 10%.

[0175] The numerical settings are all for judging the initial starting point, and they will be adjusted and optimized according to the specific situation;

[0176] ② Monitor the communication volume and frequency between partition blocks to ensure that the communication volume does not exceed 10% and the communication frequency does not exceed 10 times per second;

[0177] b) Evaluate partition memory usage and execution time;

[0178] ① Check whether the partition block memory usage exceeds 90%. If it exceeds, split the current partition to reduce memory usage;

[0179] ② Calculate whether the standard deviation of the partition block memory usage exceeds 10%. If so, split the current high memory usage partition and merge the low memory usage partition to reduce memory usage fluctuations;

[0180] c) assess the communication situation;

[0181] ① Check whether the communication volume between partition block PIMs exceeds 10% of the total data volume. If so, reduce the communication volume by merging partitions or optimizing data transmission.

[0182] ② Check whether the communication frequency between partition block PIMs exceeds 10 times per second. If so, merge partitions to reduce the communication frequency.

[0183] The split and merge operations are as follows:

[0184] ① Split: According to the attention matrix, the CIL part of the continuous composite reasoning layer with the strongest dependency in the partition to be split is retained as one partition, and the remaining CIL parts of the composite reasoning layer are merged into the adjacent partitions;

[0185] ②Merge: According to the attention matrix, the partitions to be merged are merged into the partitions where the composite inference layer CIL with which they have a stronger dependency is located in the adjacent partitions.

[0186] In this solution, the composite inference model (CIM) consists of multiple composite inference layers (CILs). Each composite inference layer (CIL) includes two types of computing units: the ISU (sub-inference unit) and the GCL (general computing layer). Each sub-inference unit includes multiple internal IL-ISU layers. Partitioning essentially combines multiple CILs into a partitioned inference submodule (PIM). Since the computational complexity of the GCL is typically smaller than that of the ISU, its impact on the overall partitioning scheme is far less critical than that of the ISU. Therefore, when calculating the performance data of the CIL, it is sufficient to only calculate the performance data of all the ISUs contained in the CIL.

[0187] Example 2

[0188] A computer device comprising:

[0189] one or more processors;

[0190] a memory storing one or more computer programs;

[0191] The processor calls the computer program to implement:

[0192] The above steps are for an inference model partitioning method for a serverless application scenario based on a self-attention mechanism.

[0193] For the specific implementation process of each step, please refer to the description of the above method.

[0194] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0195] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0196] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0197] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A self-attention-based inference model partitioning method for serverless application scenarios, characterized in that: include: Step 1: Extract the performance data of operators in each sub-inference unit ISU of the composite inference model CIM, merge and preprocess the performance data of the operators; Step 2: Build a resource prediction model based on XGBoost; The input data of the XGBoost-based resource prediction model consists of the operator-level feature dataset OLFD and the sub-inference unit ISU hierarchical feature dataset ILFD, where OLFD is the preprocessed data obtained in step 1, and ILFD is the hierarchical feature data of the sub-unit internal layer LI-ISU of the sub-inference unit extracted and calculated from OLFD; the output data is the operator-level prediction dataset, including the resource occupancy time and execution time prediction values ​​of each operator; Step 3: Hierarchically merge and summarize the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units; Step 4: Based on the self-attention mechanism, analyze the performance data of the sub-inference units to obtain the inter-layer dependencies of the composite inference model and generate a partitioning strategy; Step 5: Partition the composite inference model based on the partitioning strategy, encapsulate the submodules corresponding to each partition into serverless functions, and complete independent deployment of the partitions; The specific process of analyzing the performance data of the sub-reasoning units based on the self-attention mechanism to obtain the inter-layer dependencies of the composite reasoning model and generate the partitioning strategy is as follows: S1: Construct the inter-layer dependency feature vector of the composite reasoning layer CIL; Based on the obtained composite reasoning layer CIL hierarchical data, an inter-layer dependency feature vector is constructed for each composite reasoning layer CIL, which includes the resource prediction data of the composite reasoning layer CIL itself, the structural characteristics of the composite reasoning layer CIL, and the inter-layer connection relationship of the composite reasoning layer CIL; The structural characteristics of the composite inference layer (CIL) include the CIL layer type, the input and output data sizes of the CIL layer, and the number of CIL layer parameters. The connection relationship between the composite inference layer CIL layers is determined by extracting the computational graph and generating a dependency matrix using tools provided by PyTorch or TensorFlow. S2: Calculate the attention weights between the composite inference layer CIL layers; The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model to calculate the attention weights between the layers of the composite inference layer CIL. The specific process includes the following: a) Generation of Query, Key, and Value; The inter-layer dependency feature vector of the composite inference layer CIL is input into the self-attention mechanism model through linear transformation to generate the query vector Query, key vector Key and value vector Value between each composite inference layer CIL layer; b) Calculation of attention weights between a composite inference layer CIL and other composite inference layers CIL; ； in, is the feature dimension, Represents the similarity between the query and the key, V represents the value vector between a composite reasoning layer CIL and other composite reasoning layer CILs; c) Generate attention matrix: The attention weights of each composite inference layer CIL and other composite inference layers CIL form a matrix, which represents the strength of the inter-layer dependency; S3: Identify key layers and dependency paths; According to the attention weights between each composite reasoning layer CIL and other composite reasoning layer CILs in the attention matrix, the inter-layer dependency of composite reasoning layer CILs within the CIM is obtained, the layer groups and critical paths of closely dependent composite reasoning layer CILs are determined, and the closely dependent composite reasoning layer CILs and composite reasoning layer CILs on the critical path are divided into the same partition.

2. The method according to claim 1, characterized in that The operator-level feature dataset OLFD includes the input / output data size at the operator level, the resource occupancy and execution time monitoring value of the operator, and the complexity score of the operator; the sub-inference unit ISU level feature dataset ILFD includes two parts: structural features and computing resource features. The structural features include the layer type of IL-ISU, the number of layer parameters of IL-ISU, and the number of input / output dimensions of IL-ISU. The computing resource features include the input / output data size of IL-ISU, the shape ratio of the input and output tensors of IL-ISU, the total data volume of the input and output of IL-ISU, the input-output product of IL-ISU, the average size of each dimension, and the complexity score of IL-ISU.

3. The method according to claim 1, characterized in that Merging the operator performance data refers to integrating the operator performance data into the layer performance data of the sub-unit internal layer IL-ISU and the overall performance data of the sub-inference unit ISU according to the operator combination structure. The operator combination structure includes three types: sequential structure, parallel structure and hybrid structure. Among them, the hybrid structure refers to the superposition and combination of the sequential structure and the parallel structure. The performance data structure of the operator is as follows: ； in, represents the first Operator, Representation operator The name of the sub-reasoning unit ISU to which it belongs, Representation operator The name of the subunit internal layer IL-ISU of the sub-inference unit ISU, Represents the input operator The data size, Representation operator The number of parameters of the internal layer IL-ISU of the subunit, Representation operator Memory usage during operation, Representation operator The length of execution time during the run; When the operator combination structure in the sub-unit internal layer IL-ISU is a sequential structure, the memory usage of the sub-unit internal layer IL-ISU is The maximum memory usage among all operators in this layer, and the execution time of the IL-ISU layer within the sub-unit The sum of the execution time of all operators in this layer; When the operator combination structure in the sub-unit internal layer IL-ISU is a parallel structure, the memory usage of the sub-unit internal layer IL-ISU is is the maximum value of the combined memory usage at each position, and the execution time of the IL-ISU layer within the subunit It is the sum of the execution time of the combination at each position, and different branches at the same position Combined memory usage of , combined execution time , n represents the number of parallel branches, represents the execution time of the jth operator in the nth branch, Indicates the The first of the branches Operator, operator The memory usage and execution time of and ; When the operator combination structure in the IL-ISU layer of the sub-unit is a hybrid structure, the memory usage of the IL-ISU layer of the sub-unit is The maximum memory usage of the sequential part and the parallel part, the execution time of the sub-unit internal layer IL-ISU is the sum of the execution time of the sequential part and the parallel part; For those with The performance data of the sub-inference unit ISU of the layer IL-ISU within the layer sub-unit is expressed as the collection of the performance data of the layer IL-ISU within each sub-unit.

4. The method according to claim 1, wherein The process of building a resource prediction model based on XGBoost is as follows: Data preparation: Extract the preprocessed data obtained in step 1; Objective function setting: The objective function is to minimize the mean square error of each layer's memory usage and execution time; Model training: Use the data in the training set to train the XGBoost model with the set objective function to obtain a trained XGBoost-based resource prediction model; The data in the training set are the pre-processed data and feature data extracted from each sub-inference unit ISU in the known composite inference model CIM according to step 1; During the training process, grid search or Bayesian optimization is used to adjust the XGBoost model hyperparameters until the prediction accuracy of the XGBoost model meets the set requirements.

5. The method according to claim 3, characterized in that Hierarchical merging and aggregation of the data output by the XGBoost-based resource prediction model to obtain the performance data of the sub-inference units means: First, the hierarchical prediction data sets of the sub-unit internal layer IL-ISU are merged according to the operator combination structure. Then, the hierarchical prediction data sets of the sub-unit internal layer IL-ISU are merged to obtain the prediction data set of the sub-inference unit ISU. The prediction data of all sub-inference units ISU belonging to the same composite inference layer CIL are unioned as the hierarchical performance data of the composite inference layer CIL. The hierarchical prediction data sets of the sub-unit internal layer IL-ISU are merged into the overall performance data of the sub-inference unit ISU according to the following formula: ； in, Indicates the memory usage of the 1st, 2nd, ..., Lth layer subunit internal layer IL-ISU, represents the execution time of the inner layer IL-ISU of the 1st, 2nd, ..., Lth layer sub-units, M and T represent the memory usage and execution time of the sub-inference unit ISU, respectively.

6. The method according to claim 5, characterized in that The specific process of determining the layer groups and critical paths of the closely dependent composite reasoning layer CIL is as follows: a) Identification of closely dependent layer groups: The attention weights in the attention matrix are used to determine the dependency strength between layers in the composite inference layer (CIL). Layers that exceed the attention weight threshold are considered to be closely dependent layer groups. b) Critical path identification: By analyzing the global attention distribution of the attention matrix, the sum of the attention weights between the composite inference layers (CIL) that the data passes through from the input to the composite inference model to the output is calculated, and the path with the highest sum of attention weights is selected as the critical path.

7. The method according to claim 6, characterized in that In the attention weight matrix, find the composite inference layer CIL whose attention weights with other composite inference layers CIL exceed the bottleneck layer threshold, and use it as the bottleneck layer, and adjust the partition according to the bottleneck layer; The specific constraints are as follows: ①The number of bottleneck layers in each partition shall not exceed , The bottleneck layer threshold is the maximum number of bottleneck layers allowed in a single partition, and is initially set to 2. ② According to the IOPD data set, the total memory usage of the bottleneck layer shall not exceed , the pre-allocated memory for each partition is .

8. The method according to claim 1, characterized in that The steps to optimize the partitioning strategy are as follows: a) Monitor partition operation in a simulated environment; ① Monitor the memory usage and execution time of the partition blocks to ensure that the load of the partition blocks does not exceed 90% and the standard deviation of the memory usage does not exceed 10%. At the same time, check whether the execution time exceeds the predicted maximum time and whether the standard deviation of the execution time exceeds 10%. ② Monitor the communication volume and frequency between partition blocks to ensure that the communication volume does not exceed 10% and the communication frequency does not exceed 10 times per second; b) Evaluate partition memory usage and execution time; ① Check whether the memory usage of the partition block exceeds 90%. If so, split the current partition block to reduce the memory usage; ② Calculate whether the standard deviation of the memory usage of the partition block exceeds 10%. If so, split the current partition block with high memory usage and merge the partition block with low memory usage to reduce memory usage fluctuations; c) assess the communication situation; ① Check whether the communication volume between partition block PIMs exceeds 10% of the total data volume. If so, reduce the communication volume by merging partitions or optimizing data transmission. ② Check whether the communication frequency between partition block PIMs exceeds 10 times per second. If so, merge partitions to reduce the communication frequency. The split and merge operations are as follows: ① Split: According to the attention matrix, the CIL part of the continuous composite reasoning layer with the strongest dependency in the partition to be split is retained as one partition, and the remaining CIL parts of the composite reasoning layer are merged into the adjacent partitions; ②Merge: According to the attention matrix, the partitions to be merged are merged into the partitions where the composite inference layer CIL with which they have a stronger dependency is located in the adjacent partitions.

9. A computer device, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The steps of the method according to any one of claims 1 to 8.

Images