Multi-outlet model reasoning acceleration method based on parameter exchange mechanism in terminal environment

Abstract

The invention discloses a multi-outlet model reasoning acceleration method based on a parameter exchange mechanism in a terminal environment. The method comprises an Encoder-only Transform model parameter grouping quantification method, a parameter asynchronous loading problem model and an IO planning algorithm based on a priority queue. According to the method, firstly, importance differences of model parameters are considered, the model parameters are grouped according to the dimensionality of the attention head, and different groups of parameters are quantized to different bit widths by adopting a Gaussian abnormal value sensing method, so that parameter loading delay is reduced, and the influence on model performance is reduced. An IO planning model is established for a hierarchical pipeline reasoning mode based on a parameter exchange mechanism, and the influence of different parameters and different quantization bits on the performance of the model is analyzed. And finally, designing a heuristic algorithm, distributing IO resources for model parameters by using the priority queue, setting different quantization bit widths, and obtaining a quantization mode which has the minimum influence on model performance and enables parameter loading and calculation to be balanced.

Description

Technical Field

[0001] This invention relates to the field of edge computing technology, and in particular to a method for accelerating multi-exit model inference based on a parameter exchange mechanism in a terminal environment. Background Technology

[0002] In recent years, Transformer models have been widely used in computer vision due to their powerful feature extraction and modeling capabilities. However, these models are complex in structure, have a large number of parameters, and are computationally expensive, making them difficult to deploy directly on edge devices with limited computing power. In intelligent power video analysis scenarios, terminal devices (such as station cameras, drone terminals, and inspection robots) typically have limited computing and storage resources, making it particularly difficult to deploy Transformer deep learning models containing thousands or even hundreds of millions of parameters. Furthermore, the operating environment of terminal devices is very complex. For example, smartphones not only need to run deep learning model inference tasks but also need to run operating systems and various applications. Even if memory resources can meet the requirements of inference tasks for some "simplified" models, storing parameters in memory for extended periods will encroach on the space of other applications, thus affecting their normal operation and potentially leading to forced termination by the operating system.

[0003] To address these issues, existing applications typically utilize heterogeneous storage resources on the device, employing memory-swapping technology to perform inference task computation and model parameter loading in parallel. Specifically, when inference tasks are not required, the application stores model parameters in a large external storage location, retaining only a small portion in a memory buffer. When an inference task is needed, the application immediately begins calculating the pre-loaded parameters and simultaneously loads subsequent model parameters in parallel, achieving rapid task response. Parallel parameter loading and computation ensures that models with a large number of parameters do not waste valuable memory resources during idle time and are quickly deployed to memory for computation when inference tasks are required. However, due to the limited I / O bandwidth of terminal devices, even using parallel parameter loading for Transformer model inference incurs additional latency overhead due to the large number of parameters. Furthermore, the parameter loading latency and computation latency of Transformer model inference on terminal devices are not balanced. Research indicates that executing Transformer model inference in parallel parameter loading mode causes frequent device computation pauses, with end-to-end latency limited by the I / O overhead of parameter loading. To fully utilize the heterogeneous storage resources of terminal devices and reduce end-to-end latency in model inference, it is necessary to consider optimizing the latency overhead of loading model parameters from external storage to memory. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment.

[0005] This invention is achieved through the following technical solution:

[0006] A method for accelerating inference of a multi-exit model based on a parameter exchange mechanism in a terminal environment includes the following steps:

[0007] S1. Group the parameters of the Multi-exit Encoder-only Transformer model and determine the quantization method for grouping the model parameters;

[0008] S2. Based on model parameter quantization, define a problem model for asynchronous loading of Multi-exit Encoder-only Transformer model parameters;

[0009] S3. Design a heuristic IO planning algorithm based on priority queues to solve the asynchronous loading problem model with the parameters.

[0010] Step S1 specifically includes the following steps:

[0011] The parameters of the Multi-exit Encoder-only Transformer model are divided into different groups, and the Gaussian outlier detection method is used to quantize each group of parameters to different bit widths. The quantized values, centroids, outliers, and the subscript data corresponding to the outliers of each layer of parameters are compressed and packaged so that the parameters of different layers can be combined together during inference.

[0012] The specific details of step S1 are as follows:

[0013] The self-attention parameters in the Multi-exit Encoder-only Transformer model are divided according to the attention head dimension, and the parameters of the fully connected feedforward network are also divided accordingly. Since different AttentionHeads will pay attention to different temporal patterns in the input feature sequence, their contribution to the model's representation ability and subsequent exit classification accuracy varies, and therefore they have different importance. Based on the importance level of each group of parameters, they are quantized to different bit widths to further improve the compression efficiency of the model while ensuring the accuracy of multi-exit inference.

[0014] First, the parameters of the Multi-exit Encoder-only Transformer model are fitted with a Gaussian distribution. Parameters conforming to this distribution are sorted in ascending order of value and divided into several parameter segments. Then, the mean of the parameters within each segment is calculated as the centroid of that segment, and the original values ​​of all parameters within the segment are replaced with quantization index values ​​pointing to the corresponding centroid. For outlier parameters that do not satisfy the Gaussian distribution assumption, their indices and original values ​​in the original parameter array are recorded for subsequent recovery. Finally, after obtaining all quantization values, centroids, outlier parameters, and indices, the information is encoded and compressed using Python's bitarray library to generate the corresponding binary file.

[0015] Step S2 specifically includes the following steps:

[0016] Measure the average loading time, average computation time, and parameter loading time for each layer of the Multi-exit Encoder-only Transformer model at different quantization bit widths;

[0017] Set a parameter loading target, that is, ensure that the loading delay of each layer of parameters is less than the calculation delay of the parameters of the previous layer;

[0018] Based on the multi-exit architecture of the Multi-exit Encoder-only Transformer model, we measure the impact of quantizing parameters at a certain layer on the accuracy of all subsequent exits.

[0019] The measurements of the average loading time, average computation time, and parameter loading time for each layer of the Multi-exit Encoder-only Transformer model, as well as the parameter loading time under different quantization bit widths, are detailed below:

[0020] Assuming the Multi-exit Encoder-only Transformer model includes Layer Encoder and There are 1 export classifier, and the position of each export classifier is... ,in, It is the location of the model's main exit, i.e. ;

[0021] We performed the inference task of the Multi-exit Encoder-only Transformer model multiple times on a device using a publicly available dataset, and obtained the average computation time of the input mapping layer, the encoder layer, and the exit classifier layer. ;

[0022] Measure the average loading time and the loading time of the exit classifier for a set of parameters under different quantization bit settings. and ,in, These are the quantization bits of the parameter group. , It is the highest quantization bit;

[0023] Let one layer of Encoder contain Given the nth Attention Head, then Quantization bit settings for different parameter groups in the layer encoder Its loading time is: ;

[0024] in, Indicates the first In the Encoder layer, the first (total) The quantization bit width of each attention head; Indicates the first The total I / O time for the layer encoder to load data from disk to memory; Indicates the first In the layer, the first Each Head is quantified as After that, the parameter corresponding to this Head is the I / O time for loading from disk to memory.

[0025] To ensure that parallel parameter loading does not affect the calculation process, the loading latency needs to be less than the calculation latency of the previous layer, that is: .

[0026] Based on the multi-exit architecture of the Multi-exit Encoder-only Transformer model, the impact of quantizing parameters at a certain layer on the accuracy of all subsequent exits is measured as follows:

[0027] For the For the export classifier, its performance will be affected by the model's... Layer to the first The impact of layer quantization parameters: quantization settings for parameters of all layers in a given model. , No. The accuracy scores of each export classifier are: ,in, and It consists of the input features and their corresponding labels or predicted values ​​in the dataset. It is the computation process function of a Multi-exits Encoder-only Transformer; Indicates the first Quantization bit width settings for each parameter group in the layer encoder; This represents the total number of encoder layers in the Multi-exit Encoder-only Transformer model.

[0028] Find a set of quantization bit settings Under the constraint of satisfying the parallel loading delay of parameters, the performance loss of all exit classifiers should be minimized as much as possible, that is: ,in, When Multi-exitTransformer is not quantized, the first... The accuracy score of each export, among which... This represents the quantization bit width setting that minimizes the loss function in an objective optimization problem. This represents the total number of export classifiers in the multi-export model.

[0029] Step S3 specifically includes the following steps:

[0030] The priority queue-based heuristic I / O planning algorithm iterates in layers:

[0031] Using a parameter priority queue, the priority of all parameters in the layer is quantified and set, with one parameter priority queue for each group of parameters.

[0032] Using a layer priority queue, the priority of each group of parameters in the layer is determined, and all parameter groups share a single layer priority queue.

[0033] The heuristic IO planning algorithm based on priority queues traverses all parameters and quantization settings within the layer to obtain the accuracy scores of all subsequent exit classifiers, forming a score array that is pushed into the layer priority queue.

[0034] Dequeue the first element of all parameter group priority queues and push it into the layer priority queue for sorting and comparison. At this point, the first element of the layer priority queue is the optimal setting.

[0035] A multi-exit model inference acceleration system based on a parameter exchange mechanism in a terminal environment includes:

[0036] The model quantization module groups the parameters of the Multi-exit Encoder-only Transformer model, determines the quantization method for the grouped model parameters, and provides the quantization mode for the model parameters.

[0037] Asynchronous loading of problem model building modules; defining a Multi-exit Encoder-only Transformer model parameter asynchronous loading problem model based on model parameter quantization mode;

[0038] The model solving module designs a heuristic IO planning algorithm based on priority queues to solve the asynchronous loading problem model with parameters.

[0039] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-exit model inference acceleration method based on parameter exchange mechanism in a terminal environment.

[0040] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multi-exit model inference acceleration method based on a parameter exchange mechanism in a terminal environment.

[0041] To optimize parameter loading latency, this invention uses a Multi-exit Encoder-only Transformer as the base model. Considering the characteristics of multi-exit models and the importance of different parameters, and taking into account the device's computational and I / O resources, it utilizes quantization techniques to group and compress parameters, planning parameter I / O latency to reduce interference from parallel parameter loading on the computation process. For example, when quantizing parameters from 32 bits to 2 bits, instead of loading 16 32-bit floating-point numbers, only 4 32-bit floating-point numbers and a 32-bit array are needed, significantly reducing I / O latency during parameter loading. Existing model quantization research mainly focuses on quantization compression of traditional deep learning models. However, the Multi-exit Encoder-only Transformer model has multiple exit classifiers, and shallow parameters can be shared by different exit classifiers. Quantizing the same parameter will have different effects on different exit classifiers. To maintain model performance, it is necessary to optimize the quantization bit width of parameters and evaluate the impact on all exit classifiers that depend on that parameter.

[0042] This invention, by designing an efficient parameter exchange and dynamic loading mechanism within the model and combining it with a hierarchical inference strategy with a multi-exit structure, can significantly reduce terminal inference latency and storage usage while ensuring recognition accuracy, thereby improving the overall efficiency and reliability of the cloud-edge-device collaborative inference system.

[0043] The advantages of this invention are: In its operation, this invention solves the problem of imbalance between asynchronous model loading latency and computation latency during model inference tasks using a parameter exchange mechanism on terminal devices. This invention utilizes a Gaussian outlier awareness method to quantize the parameters of the Encoder-only Transformer model, groups the model parameters according to the attention head dimension, sets appropriate quantization bits based on parameter importance, and constructs a multi-exit Encoder-only Transformer asynchronous parameter loading model to ensure that the loading latency of the model parameters does not exceed the computation latency of the previous layer's parameters, thereby improving the model inference efficiency under the parameter exchange loading mechanism.

[0044] This invention uses a heuristic approach to solve the asynchronous parameter loading model of a multi-exit Encoder-only Transformer. It employs a priority queue to plan the quantization bits of model parameters, ensuring that loading latency meets requirements while maintaining model performance as much as possible. The algorithm plans the model parameters layer by layer from shallow to deep. After planning at each layer, it immediately quantizes the parameter group according to the plan, updating the impact on deeper layer parameters. During the quantization bit planning process at each layer, the algorithm first quantizes all parameter groups at the current layer to the minimum 2-bit width. Then, it iterates through each parameter group, examining the accuracy score of each subsequent exit classifier under different quantization bit settings. Subsequently, it combines the accuracy scores of all exits into an array, using this score array as the basis for evaluating the quantization result. The entire algorithm flow is implemented using a priority queue, which stores the aforementioned score array. The first score array dequeued represents the optimal quantization setting under the current condition. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of parameter quantization bit planning based on a priority queue;

[0046] Figure 2 This is a schematic diagram illustrating a specific implementation of the method of the present invention. Detailed Implementation

[0047] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0048] like Figure 1 , 2 As shown, this invention provides a method for accelerating inference of a multi-exit Encoder-only Transformer model based on a parameter exchange mechanism in a terminal environment, comprising the following steps:

[0049] Step 1: This invention establishes a Multi-exit Encoder-only Transformer model structure comprising 12 encoder layers, 1 input mapping layer, and 1 output classifier. Since the output classifier has a very small number of parameters, and the encoder layers constitute the majority of the model, this invention primarily focuses on quantization and compression of the encoder layers. Each encoder layer includes a Multi-Head Attention module and a Feed Forward Network module. Because different Attention Heads focus on different temporal patterns in the input feature sequence, their contributions to the model's representation ability and subsequent output classification accuracy differ, thus exhibiting different levels of importance. To reduce parameter loading latency, this invention groups the parameters of each encoder layer according to the Head, quantizing each group of parameters to the same bit width. Different groups of parameters can be quantized to different bit widths based on the importance of different Heads. Each group of parameters includes Attention Head parameters and a portion of the parameters from the FFN module.

[0050] This invention employs Gaussian outlier-aware quantization to compress each parameter group. Since the purpose of this invention's quantization compression model is to minimize the storage size of model parameters and reduce I / O latency for loading parameters, rather than to accelerate model computation, the computation process still primarily uses 32-bit floating-point numbers. Based on Gaussian outlier-aware quantization, this invention can set different quantization bits according to the importance of different parameter groups. During computation, all parameters maintain a 32-bit floating-point representation, thus seamlessly connecting parameter groups with different quantization bits into a complete encoder layer. Furthermore, this quantization method does not require further model fine-tuning or other additional hardware support, and can be well adapted to various existing terminal devices.

[0051] After the parameters of the Encoder-only Transformer model are quantized, each layer of parameters includes quantized values, centroids, outliers, and indices. This invention uses the Python library bitarray to compress the above arrays into binary files, where the quantized values ​​are stored as binary data of a specific bit width.

[0052] Step 2: On the terminal device to be deployed, use a public dataset to measure the average loading time, average computation time, and average parameter loading time of the Encoder-only Transformer model for each layer.

[0053] In the Multi-exit Encoder-only Transformer hierarchical pipeline inference process, to ensure that the device's computation does not stall, this invention uses the computation time of the previous layer as the I / O latency budget for the current layer. The model parameter size is compressed using the aforementioned grouping quantization method, reducing loading time and ensuring that all parameters for each layer are loaded into memory before computation begins. For pre-loaded parameters, since their loading latency is not within the end-to-end latency of model inference, this invention does not perform quantization compression.

[0054] Besides reducing parameter loading latency, parameter quantization also leads to a loss of accuracy in Multi-exit Encoder-only Transformers. Therefore, models addressing the asynchronous parameter loading problem in Encoder-only Transformers need to consider the impact of parameter quantization on the exit and its accuracy score at each layer. This invention uses a validation dataset to measure the impact of quantization on exit accuracy.

[0055] Step 3: After constructing the asynchronous parameter loading model, this invention solves the above model using heuristic ideas and proposes an IO planning algorithm based on priority queues. The algorithm is implemented based on two priority queues—a layer priority queue and a parameter group priority queue: during the planning process, each group of parameters and each quantization bit is traversed, the model accuracy change is measured using the dataset, and the parameters are added to the priority queue according to specific rules for sorting, ultimately obtaining the quantization bit planning for all parameters of the model.

[0056] Specifically, step 1 includes:

[0057] Step 1-1: Assume the length of the feature sequence collected by the terminal device is... The feature dimension at each time step is The output dimension of each Attention head is Then the input to the Attention Head is For AttentionHead, we have:

[0058]

[0059] in, These are the QKV parameter matrices of the Attention Head. This is the output parameter matrix. The input of the FFN is obtained by summing the outputs of all Attention Heads, i.e., the final output of Multi-HeadAttention is:

[0060]

[0061] in, It represents the number of Attention Heads in each Encoder layer.

[0062] Assume the hidden feature dimension of FFN is The output of Multi-Head Attention is denoted as Then for FFN, we have:

[0063]

[0064] in, and These are the intermediate layer parameters and output layer parameters of the FFN module, respectively. This invention increases the hidden feature dimension of the FFN. Based on the number of Heads, each group of parameters includes a portion of the weight matrix of the FFN module.

[0065] Steps 1-2: Specifically, the key idea of ​​Gaussian outlier-aware quantization is to divide the model parameters (99.9%) that mostly follow a Gaussian distribution into... Each parameter block can be replaced by its calculated average value (centroid). Therefore, these parameters can be represented by... The bit value is used as the replacement, with different values ​​pointing to different centroids. For the very few parameters (0.1%) that do not conform to a Gaussian distribution, their original values ​​are retained. When storing the quantized parameters, only the bit value needs to be saved. A centroid, a few outliers, and a bit array are sufficient. When performing model inference tasks, each... The bit value is replaced with the corresponding centroid, and the abnormal value is restored to its original position.

[0066] Steps 1-3: After quantizing the Encoder-only Transformer model using the Gaussian outlier detection method, the quantized data includes quantized values, centroids, outliers, and their corresponding indices. Centroids and outliers are stored as 32-bit floating-point data, while the indices of outliers are stored as 16-bit integer data. The quantized values ​​are stored as different binary data depending on the quantization bit width. Assuming the model parameters are quantized to 2 bits, the quantized data only contains 4 centroids; therefore, each quantized value only requires 2 bits, meaning one 32-bit binary data can store 16 quantized values.

[0067] Furthermore, step 2 includes the following steps:

[0068] Step 2-1: Assume the Multi-exit Encoder-only Transformer includes Layer Encoder and There are 1 export classifier, and the position of each export classifier is... .in, It is the location of the model's main exit, i.e. To characterize the device's computing and I / O capabilities, this invention executes the Multi-exitEncoder-only Transformer inference task multiple times on the device, obtaining the average computation time of the input mapping layer, Encoder layer, and exit classifier layer. Simultaneously, this invention utilizes a Gaussian outlier-aware quantization method to obtain the average loading time and the loading time of the exit classifier for a set of parameters under different quantization bit settings. and Where q is the quantization bit of the parameter group, , It is the highest quantization bit.

[0069] Step 2-2:

[0070] Let one layer of Encoder contain Given the nth Attention Head, then Quantization bit settings for different parameter groups in the layer encoder Its loading time is:

[0071]

[0072] To ensure that parallel parameter loading does not affect the calculation process, the loading latency needs to be less than the calculation latency of the previous layer, that is:

[0073]

[0074] in, Indicates the first In the Encoder layer, the first (total) The quantization bit width of each attention head; Indicates the first The total I / O time for the layer encoder to load data from disk to memory; Indicates the first In the layer, the first Each Head is quantified as After that, the parameter corresponding to this Head is the I / O time for loading from disk to memory.

[0075] Steps 2-3: Besides reducing parameter loading latency, parameter quantization also leads to a loss of accuracy for the Multi-exit Encoder-only Transformer. For the... For the export classifier, its performance will be affected by the model's... Layer to the first The impact of layer quantization parameters. Quantization settings for parameters of all layers in a given model. , No. The accuracy scores of each export classifier are:

[0076]

[0077] in, and It consists of the input features and their corresponding labels or predicted values ​​in the dataset. This is the computation process function of the Multi-exitsEncoder-only Transformer. The goal of the I / O planning in this invention is to find a set of quantization bit settings. Under the constraint of satisfying the parallel loading delay of parameters, the performance loss of all exit classifiers should be minimized as much as possible, that is:

[0078]

[0079]

[0080]

[0081] in, When the Multi-exit Encoder-only Transformer is not quantized, the first... The accuracy score of each export. This represents the quantization bit width setting that minimizes the loss function in an objective optimization problem. This represents the total number of export classifiers in the multi-export model. It is the highest quantization bit.

[0082] Furthermore, step 3 includes the following steps:

[0083] Step 3-1: During the quantization bit planning process at each layer, the algorithm first quantizes all parameter groups in the current layer to the minimum 2-bit width. Then, it iterates through each parameter group, examining the accuracy score of each subsequent exit classifier under different quantization bit settings. Prior experimental analysis shows that quantizing a parameter group has different effects on different subsequent exit classifiers. Therefore, the algorithm combines the accuracy scores of all subsequent exits into an array, using this score array as the basis for evaluating the quantization result. Specifically, assuming the current layer is the 1st... The layer parameters are planned, and the first exit is the final exit. Quantize the parameter set to a specific width Then, the accuracy scores of all subsequent exits of the model can be obtained using the validation dataset, thus generating an accuracy score array. Different quantization bit widths yield different accuracy score arrays. Furthermore, when evaluating the performance of different quantization bit widths for the parameter set, the accuracy score at the current branch exit has the highest priority, decreasing as the exit position deepens. Specifically, when selecting the quantization bit width for the current parameter set, the accuracy scores of the exit classifiers are compared sequentially in the score array order: first comparing the exit... Choose the quantization bit width with the largest accuracy score; if the output... If the accuracy scores are the same, then compare the next exit. The score; and so on, if the accuracy score is up to the last exit. If all are the same, the quantization bit with the lowest bit width is selected. The above comparison process is implemented through a parameter group priority queue. The queue stores the above score arrays in descending order, and the array at the head of the queue is the optimal quantization setting for the current parameter group, as shown in Algorithm 1.

[0084]

[0085] Step 3-2: After obtaining the priority queues of all parameter groups in the current layer, to meet the requirements of the hierarchical pipeline, the algorithm plans the IO latency overhead of the current layer based on the computation time of the parameters in the previous layer. This process is also implemented using priority queues: the head array of each parameter group's priority queue is stored in the layer priority queue, and the optimal quantization settings of all parameter groups continue to compete in the layer priority queue. Therefore, the head of the layer priority queue is the optimal quantization setting in the entire model layer. If loading a parameter group with that quantization bit width would exceed the latency budget, the quantization setting is discarded, and the next score array is selected from the corresponding parameter group's priority queue and stored in the layer priority queue. If the current IO budget can meet the setting requirements, the planning table is updated. This process is repeated until the layer priority queue is empty.

[0086] Based on the two different priority queues mentioned above, the algorithm can select the appropriate quantization bit settings for all exits of the Multi-exit Encoder-only Transformer, maximizing the accuracy score of each exit classifier and maintaining the performance of all exits while meeting the IO latency budget requirements. The IO planning process is shown in Algorithm 2.

[0087]

[0088]

[0089] In practical implementation, the inference acceleration method of this invention can be deployed in typical power visual monitoring scenarios, such as substations, distribution rooms, and transmission line inspection terminals. Various edge terminal devices (such as fixed monitoring cameras, drone terminals, and inspection robots) can achieve real-time intelligent analysis of on-site video streams based on the parameter exchange mechanism and multi-exit Encoder-only Transformer model described in this invention. In substation inspection scenarios, terminal devices can utilize the accelerated inference mechanism of this invention to quickly identify abnormal targets or equipment defects locally, and trigger cloud-based collaborative analysis based on the inference results, achieving low-latency anomaly alarms and status reporting. This application not only improves the automation level of inspection work but also significantly enhances the efficiency of safety monitoring and maintenance in complex operating environments.

[0090] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment, characterized in that: Includes the following steps: S1. Group the parameters of the Multi-exit Encoder-only Transformer model and determine the quantization method for grouping the model parameters; S2. Based on model parameter quantization, define a problem model for asynchronous loading of Multi-exit Encoder-only Transformer model parameters; S3. Design a heuristic IO planning algorithm based on priority queues to solve the asynchronous loading problem model with the parameters.

2. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 1, characterized in that: Step S1 specifically includes the following steps: The parameters of the Multi-exit Encoder-only Transformer model are divided into different groups, and the Gaussian outlier detection method is used to quantize each group of parameters to different bit widths. The quantized values, centroids, outliers, and the subscript data corresponding to the outliers of each layer of parameters are compressed and packaged so that the parameters of different layers can be combined together during inference.

3. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 2, characterized in that: The specific details of step S1 are as follows: The self-attention parameters in the Multi-exit Encoder-only Transformer model are divided according to the attention head dimension, and the parameters of the fully connected feedforward network are also divided accordingly. Since different attention heads will pay attention to different temporal patterns in the input feature sequence, their contribution to the model's representation ability and subsequent exit classification accuracy varies, and therefore they have different importance. Based on the importance level of each group of parameters, they are quantized to different bit widths. First, the parameters of the Multi-exit Encoder-only Transformer model are fitted with a Gaussian distribution. The parameters that conform to this distribution are sorted in ascending order of value and then divided into several parameter segments on an average basis. Subsequently, the mean value of the parameters within each parameter segment is calculated as the centroid of that segment, and the original values ​​of all parameters within the segment are replaced with quantization index values ​​pointing to the corresponding centroid. For outlier parameters that do not satisfy the Gaussian distribution assumption, their indices and original values ​​in the original parameter array are recorded for subsequent recovery. Finally, after obtaining all quantization values, centroids, outlier parameters, and indices, the above information is encoded and compressed using Python's bitarray library to generate the corresponding binary file.

4. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 2, characterized in that: Step S2 specifically includes the following steps: Measure the average loading time, average computation time, and parameter loading time for each layer of the Multi-exit Encoder-only Transformer model at different quantization bit widths; Set a parameter loading target, that is, ensure that the loading delay of each layer of parameters is less than the calculation delay of the parameters of the previous layer; Based on the multi-exit architecture of the Multi-exit Encoder-only Transformer model, we measure the impact of quantizing parameters at a certain layer on the accuracy of all subsequent exits.

5. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 4, characterized in that: The measurements of the average loading time, average computation time, and parameter loading time for each layer of the Multi-exit Encoder-only Transformer model, as well as the parameter loading time under different quantization bit widths, are detailed below: Assuming the Multi-exit Encoder-only Transformer model includes Layer Encoder and There are 1 export classifier, and the position of each export classifier is... ,in, It is the location of the model's main exit, i.e. ; We performed the inference task of the Multi-exit Encoder-only Transformer model multiple times on a device using a publicly available dataset, and obtained the average computation time of the input mapping layer, the encoder layer, and the exit classifier layer. ; Measure the average loading time and the loading time of the exit classifier for a set of parameters under different quantization bit settings. and ,in, These are the quantization bits of the parameter group. , It is the highest quantization bit; Let one layer of Encoder contain Given the nth Attention Head, then Quantization bit settings for different parameter groups in the layer encoder Its loading time is: ; in, Indicates the first In the Encoder layer, the first The quantization bit width of each Attention Head is... One Attention Head; Indicates the first The total I / O time for the layer encoder to load data from disk to memory; Indicates the first In the layer, the first Each Attention Head is quantized into After that, the I / O time for loading the parameters corresponding to the Attention Head from disk to memory; To ensure that parallel parameter loading does not affect the calculation process, the loading latency needs to be less than the calculation latency of the previous layer, that is: .

6. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 5, characterized in that: Based on the multi-exit architecture of the Multi-exit Encoder-only Transformer model, the impact of quantizing parameters at a certain layer on the accuracy of all subsequent exits is measured as follows: For the For the export classifier, its performance will be affected by the model's... Layer to the first The impact of layer quantization parameters: quantization settings for parameters of all layers in a given model. , No. The accuracy scores of each export classifier are: ,in, and It consists of the input features and their corresponding labels or predicted values ​​in the dataset. It is the computation process function of the Multi-exits Encoder-only Transformer. Indicates the first Quantization bit width settings for each parameter group in the layer encoder; This represents the total number of encoder layers in a Multi-exit Encoder-only Transformer model. Find a set of quantization bit settings Under the constraint of satisfying the parallel loading delay of parameters, the performance loss of all exit classifiers should be minimized as much as possible, that is: ,in, When Multi-exitTransformer is not quantized, the first... The accuracy score of each export, among which... This represents the quantization bit width setting that minimizes the loss function in an objective optimization problem. This represents the total number of export classifiers in the multi-export model.

7. The method for accelerating multi-exit model inference based on parameter exchange mechanism in a terminal environment according to claim 1, characterized in that: Step S3 specifically includes the following steps: The priority queue-based heuristic I / O planning algorithm iterates in layers: Using a parameter priority queue, the priority of all parameters in the layer is quantified and set, with one parameter priority queue for each group of parameters. Using a layer priority queue, the priority of each group of parameters in the layer is determined, and all parameter groups share a single layer priority queue. The heuristic IO planning algorithm based on priority queues traverses all parameters and quantization settings within the layer to obtain the accuracy scores of all subsequent exit classifiers, forming a score array that is pushed into the layer priority queue. Dequeue the first element of all parameter group priority queues and push it into the layer priority queue for sorting and comparison. At this point, the first element of the layer priority queue is the optimal setting.

8. A multi-exit model inference acceleration system based on parameter exchange mechanism in a terminal environment, characterized in that: Including: The model quantization module groups the parameters of the Multi-exit Encoder-only Transformer model, determines the quantization method for the grouped model parameters, and provides the quantization mode for the model parameters. Asynchronous loading of problem model building modules; defining Multi-exit Encoder-onlyTransformer model parameters based on model parameter quantization mode for asynchronous loading of problem models. The model solving module designs a heuristic IO planning algorithm based on priority queues to solve the asynchronous loading problem model with parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multi-exit model inference acceleration method based on parameter exchange mechanism in any one of claims 1-7 in the terminal environment.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-exit model inference acceleration method based on parameter exchange mechanism in any one of claims 1-7 in the terminal environment.

Images