A neural network collaborative inference method for resource-constrained heterogeneous devices

Abstract

The application discloses a neural network cooperative inference method for resource-limited heterogeneous devices, and belongs to the edge computing field. The method is as follows: an inference time delay predictor for the resource-limited heterogeneous devices is constructed, a Bayesian optimization splitting method is adopted, an original neural network model requiring inference is split to obtain multiple sub-models suitable for the computing performance and hardware resource constraints of the resource-limited heterogeneous devices; all the sub-models are sequentially trained by using a progressive enhancement distillation method, all the trained sub-models are deployed on the corresponding resource-limited heterogeneous devices, when an inference task request is received, multiple idle resource-limited heterogeneous devices are combined to perform the computation of the sub-models in parallel, the computation results are transmitted to a central node through a local area network between the resource-limited devices, the final inference result is obtained after the computation results are gathered, the computing cost and storage resources of the multiple resource-limited heterogeneous devices are saved, and the inference time delay and energy loss of the multiple resource-limited heterogeneous devices are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural network inference method, in particular to a neural network collaborative inference method for resource-constrained heterogeneous devices, and belongs to the field of edge computing. BACKGROUND

[0002] With the rapid development of high-performance computing, 5G / 6G and big data, machine learning technologies represented by deep neural networks (DNN) have far exceeded traditional methods and even achieved superhuman performance in pattern recognition, decision planning and other fields. In recent years, some low-power and resource-constrained mobile devices can help users complete intelligent tasks efficiently and accurately in real time due to the integration of intelligent personal assistant systems based on machine learning technologies, such as Apple Siri, Microsoft Cortana, etc. Therefore, it is of great significance to deploy DNN on resource-constrained devices such as development boards, mobile phones and satellites to realize intelligent applications. However, resource-constrained devices have strict limitations on power consumption and latency, and their hardware resources and computing performance are much smaller than those of computing servers, so DNN with large parameter quantity and high complexity cannot be directly deployed on resource-constrained devices to realize intelligent applications.

[0003] To solve the above problems, researchers usually compress DNN using model compression methods before deploying it on resource-constrained devices. Common model compression methods include weight quantization, model pruning and lightweight structure design techniques. However, these techniques have the following problems: 1) Different model compression methods need to be designed for different resource-constrained devices, which cannot adapt to different computing performance and hardware resources of resource-constrained devices, 2) Existing model compression methods cannot balance the prediction accuracy and inference speed of inference tasks, and cause huge energy consumption, so better neural network inference methods for resource-constrained devices need to be designed to achieve better performance. SUMMARY

[0004] To address the problem that current methods cannot adapt to different computing performance and hardware resources of resource-constrained heterogeneous devices and cannot balance the prediction accuracy and inference speed of inference tasks, the main purpose of the present application is to propose a neural network collaborative inference method for resource-constrained heterogeneous devices. Using Bayesian optimization method, different sub-models are obtained to adapt to the computing performance and hardware resources of multiple resource-constrained heterogeneous devices, realizing collaborative inference acceleration of multiple resource-constrained heterogeneous devices, saving computing overhead and storage resources of multiple resource-constrained heterogeneous devices, and reducing inference latency and energy consumption of multiple resource-constrained heterogeneous devices.

[0005] The main purpose of the present application is achieved through the following technical solutions:

[0006] The application discloses a neural network collaborative inference method for resource-limited heterogeneous devices.

[0007] The neural network collaborative inference method for resource-limited heterogeneous devices comprises the following steps:

[0008] Step one: collect inference time delay data of a neural network on each resource-limited heterogeneous device, and train a plurality of inference time delay predictors using the inference time delay data of each resource-limited heterogeneous device;

[0009] When there are m resource-limited heterogeneous devices, a Transformer model is used as a backbone model of the neural network, the number of layers l, the embedding layer dimension d, the number of self-attention heads h of each layer and the dimension D of the hidden layer neurons of each layer of the multi-layer perceptron of the Transformer model are changed, and the inference time delay t on each resource-limited heterogeneous device is tested to obtain m inference time delay data sets {l, d, h, D, t} for different resource-limited heterogeneous devices.

[0010] A multi-layer perceptron is used as an inference time delay predictor p(·), and the inference time delay data sets of different resource-limited heterogeneous devices are used to train the inference time delay predictor, wherein the predictor is a multi-layer perceptron, a linear rectification function (Rectified Linear Unit, ReLU) is used as an activation function, the input of the inference time delay predictor is l, d, and wherein, the average value of the number of self-attention heads of each layer is represented by, the average value of the dimension of the hidden layer neurons of each layer of the multi-layer perceptron is represented by, and the output of the inference time delay predictor is the inference time delay The trained inference time delay predictor can efficiently and accurately predict the inference time delay of the neural network.

[0011] Step two: establish a resource-limited heterogeneous device joint inference time delay and accuracy loss model, set available split points in a to-be-split original neural network according to the constraints of the computing performance and hardware resources of each resource-limited heterogeneous device, and establish an optimization problem of minimizing the joint inference time delay and accuracy loss of the resource-limited heterogeneous devices;

[0012] The number of resource-constrained heterogeneous devices is N, and a certain resource-constrained device is set as a center node. The original Transformer model is selected as the original neural network model that needs to be split, and a resource-constrained heterogeneous device joint inference delay and accuracy loss model is established.

[0013] Resource-constrained heterogeneous device joint inference is composed of three parts: forward propagation, data transmission, and result aggregation. Forward propagation is the forward propagation of each resource-constrained device executing the Transformer sub-model. The forward propagation delay of the nth Transformer sub-model on the nth resource-constrained device is .

[0014]

[0015] Data transmission is the transmission of intermediate data obtained by forward propagation by each resource-constrained device to the center node. The data transmission delay of the nth resource-constrained device is .

[0016]

[0017] where |X n | represents the size of the data features transmitted by the nth resource-constrained device, and r n represents the data transmission rate of the nth resource-constrained device.

[0018] Result aggregation is the aggregation of all intermediate data obtained by forward propagation by the center node i. The delay of result aggregation is .

[0019]

[0020] where d i represents the embedding layer dimension of the Transformer sub-model deployed on the center node, M represents the sequence length, g represents the computing performance of the resource-constrained device of the center node, and d agg is the sum of the embedding layer dimensions of all Transformer sub-models.

[0021] The resource-constrained heterogeneous device joint inference delay model is:

[0022]

[0023] where is the set of all resource-constrained heterogeneous devices.

[0024] The average loss value of all resource-constrained devices jointly reasoning on the validation dataset is taken as the resource-constrained heterogeneous device joint reasoning accuracy loss value, and the accuracy loss model of the resource-constrained heterogeneous device joint reasoning is represented as:

[0025]

[0026] wherein, represents the expected value on the validation dataset, represents the loss value of the nth resource-constrained device on the validation dataset, represents the set of the nth Transformer sub-model structure parameters, represents the set of all Transformer sub-models.

[0027] Based on the above established resource-constrained heterogeneous device joint reasoning delay and accuracy loss model, in order to achieve small resource-constrained heterogeneous device joint reasoning accuracy loss and reasoning delay, while adapting to the computing performance of each resource-constrained heterogeneous device n , and memory capacity The following optimization problem is established:

[0028]

[0029] wherein, δ>0 is a hyperparameter for adjusting the joint reasoning loss and reasoning delay. L0,d n ,h n and D0 respectively represent the number of layers, the embedding layer dimension, the number of self-attention heads and the dimension of the multi-layer perceptron hidden layer neurons of the original Transformer model, l n represents the number of layers of the nth Transformer sub-model, represents the number of self-attention heads of the kth layer of the nth Transformer sub-model, represents the number of embedding layer dimensions of the kth layer of the nth Transformer sub-model, ω(C n ) represents a mapping function of the floating-point operation number of the Transformer model, φ(C n ) represents a mapping function of the memory overhead of the Transformer model.

[0030] Step three: the Bayesian optimization splitting method is used to solve the optimization problem of minimum resource-constrained heterogeneous device joint reasoning accuracy and delay, and the original neural network is split into different sub-models that adapt to the computing performance and hardware resource constraints of the resource-constrained heterogeneous device;

[0031] For the optimization problem of minimizing the joint inference accuracy and latency of the above resource-constrained heterogeneous devices, the objective function is converted into a black-box function Solved by using Bayesian optimization splitting method.

[0032] Step 3.1: Use Gaussian process GP(·) as the prior probability model of the black-box function Initialization:

[0033]

[0034] Where μ(·) represents the mean, and K(·,·) represents the Matern kernel function.

[0035] Step 3.2: According to the Gaussian process, r groups of splitting strategy sets that meet the computational performance and hardware resource constraints of the corresponding resource-constrained heterogeneous devices are randomly sampled Get r groups of different Transformer sub-models. The prior probability model of the Gaussian process represents the black-box function obeys the Gaussian distribution:

[0036]

[0037] Where, r groups of splitting strategies The corresponding black-box function value, by evaluating the objective function value of the resource-constrained heterogeneous device joint inference accuracy and latency minimization problem under different groups of splitting strategies as the noise estimate of the black-box function Where is Gaussian white noise, then the posterior probability model of the Gaussian process can be updated as

[0038]

[0039] Where I represents the identity matrix, so for a new splitting strategy The posterior probability model of the Gaussian process can be used to predict the distribution of

[0040]

[0041] Where, and represent the expectation and standard deviation, respectively.

[0042] Step 3.3: According to the distribution of predicted by the posterior probability model of the Gaussian process, use the acquisition function ​The newly sampled splitting strategy is evaluated by finding the splitting strategy that minimizes the acquisition function as the next set of splitting strategies that meet the requirements:

[0043]

[0044] Step 3.4: The new splitting strategy obtained in step 3.3 is added to the splitting strategy set, and step 3.2 is repeated until the stopping condition is met or the optimal splitting strategy is obtained. The original Transformer model is split according to the final splitting strategy to obtain N Transformer sub-models.

[0045] Step four: using the progressive enhancement distillation algorithm to train all Transformer sub-models obtained in step three in turn using the original Transformer model as supervision, helping the Transformer sub-models learn the rich semantic features in the original Transformer model and improve the prediction accuracy of the Transformer sub-models;

[0046] The training data set is selected as where M represents the number of samples in the training data set, and the progressive enhancement distillation algorithm is used to train the N Transformer sub-models obtained in step three in turn.

[0047] Step 4.1: initialize the weight of each sample in the training data set:

[0048]

[0049] where, W n represents the weight set of each sample in the training data set when training the nth Transformer sub-model, represents the weight of the ith sample in the training data set when training the nth Transformer sub-model, and in the initialization stage, n = 1.

[0050] Step 4.2: use the training data set samples with weights W n to train the nth Transformer sub-model, and use the original Transformer model as the supervision of the Transformer sub-model, and the loss function of the training is:

[0051]

[0052] where, Λ(·) represents the prediction result of the Transformer sub-model, and y represents the true label value of the data, Y t ​(x) represents the prediction result of the original Transformer model, Y t represents the prediction probability value of the original Transformer model, represents the prediction probability value of the n-th Transformer sub-model, represents the cross-entropy loss function.

[0053] Step 4.3: After training the n-th Transformer sub-model, update the weight of each sample in the training data set Thus, the n+1-th Transformer sub-model is trained, and the weight value of each sample in the training data set is updated according to the following formula:

[0054]

[0055] Repeat step 4.2 to train the n+1-th Transformer sub-model until all Transformer sub-models are trained.

[0056] Step five: deploy all the trained Transformer sub-models in step four to the corresponding resource-limited heterogeneous devices, when receiving an inference task request, combine multiple idle resource-limited heterogeneous devices to perform the calculation of the Transformer sub-model in parallel, and the calculation result is transmitted to the center node through the local area network between the resource-limited devices to obtain the final neural network after aggregation, and the final neural network obtained after aggregation is used for collaborative inference, saving the calculation overhead and storage resources of multiple resource-limited heterogeneous devices, thereby reducing the inference time delay and energy consumption of multiple resource-limited heterogeneous devices.

[0057] Advantages:

[0058] 1. The neural network collaborative inference method for resource-limited heterogeneous devices disclosed in the present application can save the calculation overhead and storage resources of multiple resource-limited heterogeneous devices, thereby reducing the inference time delay and energy consumption of multiple resource-limited heterogeneous devices.

[0059] 2, The neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the application, a resource-constrained heterogeneous device joint inference delay and accuracy loss model is constructed, and according to the computing performance and hardware resources of each resource-constrained heterogeneous device, the available split points in the original neural network that need to be split are set, and an optimization problem of minimizing the resource-constrained heterogeneous device joint inference delay and accuracy loss is established, thereby maximizing the computing performance and hardware resources of each resource-constrained heterogeneous device, and facilitating the solution of the splitting strategy that minimizes the resource-constrained heterogeneous device joint inference delay and accuracy loss.

[0060] 3, The neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the application uses the Bayesian optimization splitting method to obtain the optimal splitting strategy, realizes the minimum joint inference accuracy and delay of resource-constrained heterogeneous devices, and adapts to the computing performance and hardware resources of the corresponding resource-constrained heterogeneous devices. The application can greatly reduce the performance loss caused by splitting, and maximize the computing performance and hardware resources of the resource-constrained heterogeneous devices.

[0061] 4, The neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the application uses the progressive enhancement distillation method to train the sub-models obtained by splitting the original neural network in sequence, helps the sub-models learn the rich semantic features in the original neural network, greatly improves the prediction accuracy of the neural network sub-models, and thereby improves the performance of resource-constrained heterogeneous device collaborative inference. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the application.

[0063] Figure 2 The framework diagram of the neural network collaborative inference of the plurality of resource-constrained heterogeneous devices in the embodiment of the application. DETAILED DESCRIPTION

[0064] The application will be described in detail below with reference to the accompanying drawings and embodiments, and the technical problems solved by the technical solutions of the application and the beneficial effects will be discussed. It should be noted that the described embodiments are intended to facilitate understanding of the application and do not limit the application in any way.

[0065] Embodiment 1

[0066] As Figure 1 shown, the neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the embodiment is applied in a plurality of resource-constrained heterogeneous devices. The specific implementation steps are as follows:

[0067] Step one, select the neural network model skeleton and resource-constrained heterogeneous device types, measure the latency of different neural network model structure parameters on different resource-constrained heterogeneous devices, and use the latency data to train the latency predictor of different resource-constrained heterogeneous devices;

[0068] The DeiT model is selected as the neural network skeleton model, NVIDIA Jetson Nano, NVIDIA Jetson TX2 and NVIDIA Jetson Orin Nano are used as three kinds of resource-constrained heterogeneous devices, and the DeiT-B model structure parameters are changed, including: number of layers l, embedding layer dimension d, number of self-attention heads per layer h, and dimension D of each layer of multilayer perceptron hidden layer neurons. The DeiT-B model structure parameter change range is shown in the following table:

[0069] Model structure parameters Variation range Number of layers l 2，3，4，5，6，7，8，9，10，11，12 Embedding layer dimension d 128，256，384，512，640，768，896，1024 Number of self-attention heads h 1，2，3，4，5，6，7，8，9，10，11，12 Layer perception machine hidden layer neuron dimension D 512，1024，1536，2048，2560，3072，3584，4096

[0070] The latencies of DeiT models with different structure parameters on NVIDIA Jetson Nano, NVIDIA Jetson TX2 and NVIDIA Jetson Orin Nano are measured respectively, and the latency datasets of the three resource-constrained heterogeneous devices are constructed.

[0071] A three-layer perceptron is used as the latency predictor, the hidden layer neuron dimension is set to 6000, ReLU is used as the activation function, the latency datasets of the three resource-constrained heterogeneous devices are used to train the latency predictor respectively, the training iteration step is 8000, the batch size is 32, the learning rate is 1e-5, and the mean square error RMSE is selected as the loss function. The performance of the predictor after training is shown in the following table:

[0072] Time delay predictor Mean square error RMSE NVIDIA Jetson Nano predictor 4.12 ms NVIDIA Jetson TX2 predictor 3.45 ms NVIDIA Jetson Orin Nano predictor 2.68 ms

[0073] Step two: collect the computing performance and hardware resource data of the resource-constrained heterogeneous devices for collaborative inference, select the neural network model that needs to be split, and establish an optimization problem of minimizing the joint precision loss and inference latency of the resource-constrained heterogeneous devices;

[0074] NVIDIA Jetson Nano, NVIDIA Jetson TX2 and NVIDIA Jetson Orin Nano are selected as the three resource-constrained heterogeneous devices for collaborative inference, NVIDIA Jetson TX2 is selected as the center node, and the computing performance and hardware resource data of the resource-constrained heterogeneous devices are collected as shown in the following table:

[0075] Resource-constrained heterogeneous device Floating-point operations per second Memory capacity NVIDIA Jetson Nano predictor 235.8 GFLOPS 4 GB NVIDIA Jetson TX2 predictor 665.6 GFLOPS 8 GB NVIDIA Jetson Orin Nano predictor 640.0 GFLOPS 4 GB

[0076] Among them, the number of floating-point operations per second and the memory capacity represent the computing performance and hardware resources of the device, respectively.

[0077] DeiT-B was chosen as the original neural network model to be split. The number of layers L0, the embedding layer dimension d0, the number of self-attention heads per layer h0, and the dimension D0 of neurons in the hidden layer of each multilayer perceptron were 12, 1024, 12, and 4096, respectively. The optimization problem of minimizing latency and accuracy loss in joint inference on resource-constrained heterogeneous devices is as follows:

[0078]

[0079] in, Ω1, Ω2 and Ω3 are 6, 16 and 15 respectively, and Φ1, Φ2 and Φ3 are 4, 8 and 4 respectively.

[0080] Step 3: Use the Bayesian optimization splitting method to solve the optimization problem of minimizing latency and accuracy loss in joint inference of resource-constrained heterogeneous devices. Use the optimal splitting strategy obtained from the solution to split the original neural network into sub-models.

[0081] Step 3.1: Use the Gaussian process GP(·) as the objective function for the optimization problem of minimizing the delay and accuracy loss of joint inference for resource-constrained heterogeneous devices obtained in Step 3. Prior probability model:

[0082]

[0083]

[0084] in, v = 1.5, Γ(·) and K v (·) represent the modified Bessel function and the gamma function, respectively.

[0085] Step 3.2: Randomly sample 100 groups of splitting strategies based on the prior probability model. We obtained 100 sub-models and defined the objective function under different group splitting strategies. After evaluation, the posterior probability model of the Gaussian process is updated.

[0086] Step 3.3: Through the updated Distribution, using the expected improvement function as the sampling function

[0087]

[0088] Where Z(·) and H(·) represent the objective function and cumulative function of probability, respectively, and Ψ * This indicates all previous splitting strategies. The corresponding objective function value The minimum splitting strategy is then used to obtain a new splitting strategy by minimizing the sampling function.

[0089]

[0090] Step 3.4: Obtain a new splitting strategy based on Step 3.3. Repeat step 3.2 and subsequent steps 100 times, then stop. Finally, select the corresponding objective function value from all splitting strategies. The smallest strategy is taken as the optimal splitting strategy, resulting in 3 sub-models.

[0091] The inference accuracy and inference latency of the three sub-models obtained from the split are as follows:

[0092] Model Inference accuracy Inference latency Sub-model 1 52.2％ 34.2 ms Sub-model 2 68.7％ 25.9 ms Sub-model 3 76.5％ 16.4 ms

[0093] Step 4: Train the split sub-models sequentially using the progressive enhancement distillation method;

[0094] CIFAR-100 was selected as the dataset, with 50,000 samples in the training dataset. The three sub-models obtained from the split in step three were trained sequentially using the progressive augmentation distillation method.

[0095] Step 4.1: Initialize the weights of each sample in the training dataset using an averaging method:

[0096]

[0097] Weights are assigned to each sample in the training dataset.

[0098] Step 4.2: Train the first sub-model using the training dataset samples with weights W1, and use DeiT-B as supervision to help the sub-model learn the features of DeiT-B. The training loss function is:

[0099]

[0100] Step 4.3: After training the first sub-model, update the weights of each sample in the training dataset:

[0101]

[0102] Repeat step 4.2 to continue training the next sub-model and subsequent steps until all sub-models have been trained.

[0103] Step 5: Deploy the trained sub-models to the corresponding resource-constrained heterogeneous devices, and all resource-constrained heterogeneous devices collaboratively execute inference tasks.

[0104] After training all the sub-models, they are respectively deployed to the corresponding resource-constrained heterogeneous devices, and the CIFAR-100 test dataset is used to test the collaborative inference task. When the resource-constrained heterogeneous device receives an inference task request, it starts parallel computing of the sub-models, and the calculation results are transmitted to the central node NVIDIA Jetson TX2 through the local area network between the resource-constrained devices for aggregation to obtain the final inference result. The final measurement results of the embodiment are as follows:

[0105]

[0106] The table compares the results of the original DeiT-B model, the lightweight model MobileViT V2 and the method of the present application under the same inference task. The floating point operation number represents the number of floating point operations required for a single sample inference, the video memory requirement represents the video memory required for deploying the model to the resource-constrained device, the parameter quantity represents the number of all parameters of the model, the inference accuracy represents the average inference accuracy of the test dataset, and the inference delay and energy consumption represent the average inference delay and energy consumption of each sample on the test dataset.

[0107] Compared with the original DeiT-B model, the method disclosed in the embodiment only loses 0.98% of the inference accuracy, can realize reduction of 59.14% of the floating point operation number, 39.31% of the video memory requirement, 56.35% of the parameter quantity, improvement of 2.35 times of the inference speed, and reduction of 45.18% of the inference energy consumption. At the same time, compared with the lightweight model MobileViT V2, the method disclosed in the embodiment can realize higher inference accuracy, lower inference delay and inference energy consumption.

[0108] Therefore, the neural network collaborative inference method for resource-constrained heterogeneous devices disclosed in the embodiment uses the Bayesian optimization splitting method to split the original neural network into multiple sub-models, reduces the calculation complexity of the original neural network, and uses the progressive enhancement distillation method to train each sub-model in turn, effectively improves the performance of the model inference, and finally each sub-model is deployed to the corresponding resource-constrained heterogeneous device for collaborative inference, which greatly reduces the inference delay and energy consumption.

[0109] The specific description described above further details the purpose, technical solutions and benefits of the application. It should be understood that the above description is only a specific embodiment of the application and does not limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A neural network collaborative inference method for resource-constrained heterogeneous devices, characterized in that: The method comprises the following steps, Step one: collecting inference delay data of the neural network on each resource-constrained heterogeneous device, and training multiple inference delay predictors using the inference delay data of each resource-constrained heterogeneous device; Step two: establishing a resource-constrained heterogeneous device joint inference delay and accuracy loss model, calculating the constraints of performance and hardware resources for each resource-constrained heterogeneous device, setting available split points in the original neural network that needs to be split, and establishing an optimization problem of minimizing the joint inference delay and accuracy loss of the resource-constrained heterogeneous device; Step three: solving the optimization problem of minimizing the joint inference accuracy and delay of the resource-constrained heterogeneous device by using a Bayesian optimization splitting method, and splitting the original neural network into different sub-models that adapt to the computing performance and hardware resource constraints of the resource-constrained heterogeneous device; Step four: using the original Transformer model as supervision, training all Transformer sub-models obtained in step three in turn using a progressive enhancement distillation algorithm, helping the Transformer sub-models learn rich semantic features in the original Transformer model, and improving the prediction accuracy of the Transformer sub-models; Step five: deploying all Transformer sub-models trained in step four to corresponding resource-constrained heterogeneous devices, jointly using multiple idle resource-constrained heterogeneous devices to perform parallel computation of the Transformer sub-models when receiving an inference task request, transmitting the computation results to a central node through a local area network between the resource-constrained devices for aggregation to obtain a final neural network, and performing collaborative inference using the final neural network obtained after aggregation, thereby saving the computing overhead and storage resources of the multiple resource-constrained heterogeneous devices and reducing the inference delay and energy consumption of the multiple resource-constrained heterogeneous devices.

2. The neural network collaborative inference method for resource-constrained heterogeneous devices of claim 1, wherein: The implementation method of step one is, When there are m resource-constrained heterogeneous devices, the Transformer model is used as the backbone model of the neural network, the number of layers l, the embedding layer dimension d, the number of self-attention heads h of each layer, and the dimension D of the hidden layer neurons of each layer of the multi-layer perceptron are changed, and the inference delay t on each resource-constrained heterogeneous device is tested to obtain m inference delay data sets {l, d, h, D, t} for different resource-constrained heterogeneous devices; Using a multi-layer perceptron as the inference latency predictor p(·), the inference latency predictor is trained using the inference latency dataset of different resource-constrained heterogeneous devices, the predictor is a multi-layer perceptron, a linear rectifier function ReLU is used as the activation function, the input of the inference latency predictor is l,d, and wherein, indicates the average value of the number of self-attention heads of each layer, indicates the average value of the multi-layer perceptron hidden layer neuron dimension of each layer, and the output of the inference latency predictor is the inference latency The inference latency of the neural network is efficiently and accurately predicted by the trained inference latency predictor.

3. The neural network collaborative inference method for resource-constrained heterogeneous devices of claim 2, wherein: The implementation method of step two is, The number of resource-constrained heterogeneous devices is N, a certain resource-constrained device is set as a central node, and the original Transformer model is selected as the original neural network model that needs to be split, and a resource-constrained heterogeneous device joint inference delay and accuracy loss model is established; The resource-constrained heterogeneous device joint inference is composed of three parts of forward propagation, data transmission and result aggregation; the forward propagation is forward propagation of each resource-constrained device executing a Transformer submodel, and the forward propagation delay of the nth Transformer submodel on the nth resource-constrained device is is: Data transmission is that each resource-limited device transmits the intermediate data obtained by forward propagation to the center node, and the data transmission delay of the nth resource-limited device is is: wherein |X n | represents the size of the data transmission characteristic of the nth resource-constrained device, r n represents the data transmission rate of the nth resource-constrained device; The result aggregation is the intermediate data obtained by the central node i aggregating all the intermediate data obtained by the forward propagation of all the resource-constrained devices, and the delay of the result aggregation is Tresult aggregation= Tforward propagation+ Tcentral node wherein d i denotes the embedding layer dimension of the Transformer sub-model on the deployment center node, M denotes the sequence length, g denotes the computing performance of the center node resource-constrained device, d agg is the sum of all Transformer sub-model embedding layer dimensions; The resource-constrained heterogeneous device joint inference delay model is: wherein, is a set of all resource-constrained heterogeneous devices; The average loss value of all resource-constrained devices on the validation data set is used as the joint inference accuracy loss value of the resource-constrained heterogeneous device, and the accuracy loss model of the resource-constrained heterogeneous device joint inference is expressed as: wherein, represents the expected value on the validation dataset, represents the loss value of the nth resource-constrained device on the validation dataset, represents the set of the nth Transformer sub-model structure parameters, represents the set of all Transformer sub-models; Based on the established resource-constrained heterogeneous device joint inference delay and precision loss model, in order to realize small resource-constrained heterogeneous device joint inference precision loss and inference delay, while adapting to the computing performance of each resource-constrained heterogeneous device and memory capacity The following optimization problem is established: where δ > 0 is a hyper-parameter to adjust the joint inference loss and inference latency; L0,d n ,h n and d0denote the number of layers, the embedding layer dimension, the number of self-attention heads and the dimension of multi-layer perceptron hidden neurons of the original Transformer model, respectively, l n denotes the number of layers of the nthTransformer sub-model, denotes the number of self-attention heads of the kthlayer of the nthTransformer sub-model, denotes the number of embedding layer dimensions of the kthlayer of the nthTransformer sub-model, ω(C n ) denotes a mapping function of the number of floating-point operations of the Transformer model, φ(C n ) denotes a mapping function of the memory overhead of the Transformer model.

4. The neural network collaborative inference method for resource-constrained heterogeneous devices of claim 3, wherein: The implementation method of step three is, For the optimization problem of joint inference precision and minimum delay of the above resource-constrained heterogeneous device, the objective function is converted into a black box function Solved by using Bayesian optimization splitting method; Step 3.1: Use the Gaussian Process GP(·) as the prior probability model for the black-box function Initialization: Where μ(·) represents the mean, and K(·,·) represents the Matern kernel function; Step 3.2: Randomly sample r sets of splitting strategies that satisfy the computational performance and hardware resource constraints of the corresponding resource-constrained heterogeneous devices according to the Gaussian process Obtain r different Transformer sub-models; the prior probability model of the Gaussian process represents the black box function Subject to Gaussian distribution: wherein, denotes the r-th group splitting strategy corresponding black-box function value, which is the noise estimation of the black-box function by evaluating the objective function value of the joint inference accuracy and latency minimization problem of the resource-constrained heterogeneous devices under different group splitting strategies wherein is the Gaussian white noise, the posterior probability model of the Gaussian process can be updated as where I denotes the identity matrix, so that for a new split strategy can be predicted using the posterior model of the Gaussian process distribution: wherein, and respectively represent the expectation and the standard deviation; Step 3.3: Prediction from the posterior probability model of the Gaussian process Distribution, using the acquisition function Evaluate the newly sampled split strategy by finding the split strategy that minimizes the acquisition function as the next set of split strategies that meet the requirements: Step 3.4: add the new splitting strategy obtained in step 3.3 into the splitting strategy set, repeat step 3.2 until the stopping condition is met or the optimal splitting strategy is obtained; split the original Transformer model according to the final splitting strategy to obtain N Transformer sub-models. add the new splitting strategy obtained in step 3.3 into the splitting strategy set, repeat step 3.2 until the stopping condition is met or the optimal splitting strategy is obtained; split the original Transformer model according to the final splitting strategy to obtain N Transformer sub-models.

5. The neural network collaborative inference method for resource-constrained heterogeneous devices of claim 4, wherein: The implementation method of step four is, The selected training data set is wherein M represents the number of samples in the training data set, and the N Transformer sub-models obtained in step three are sequentially trained using a progressive enhancement distillation algorithm. Step 4.1: initialize the weight of each sample in the training data set: wherein W n denotes the weight set of each sample in the training data set when training the nth Transformer sub-model, denotes the weight of the ith sample in the training data set when training the nth Transformer sub-model, initialization stage, n = 1; Step 4.2: Train the nthtransformer sub-model using the training dataset samples with weights W n and use the original transformer model as supervision of the transformer sub-models, the loss function of the training is: Wherein, Lambda (·) represents the prediction result of the Transformer sub-model, y represents the data true label value, Lambda represents the prediction result of the original Transformer model, Y t Lambda represents the prediction probability value of the original Transformer model, Lambda represents the prediction probability value of the n Transformer sub-models, Lambda represents the cross-entropy loss function; Step 4.3: After training the nth Transformer sub-model, update the weight of each sample in the training data set Thus, the (n+1)th Transformer sub-model is trained, and the weight value of each sample in the training data set is updated according to the following formula: Repeat step 4.2 to train the (n+1)th Transformer sub-model until all the Transformer sub-models are trained.

Images