A deep neural network classification method based on end-side cloud cooperation

Abstract

The application provides a deep neural network classification method based on end-edge-cloud cooperation, which acquires a trained multi-output classification model; the trained multi-output classification model is dynamically partitioned online according to the CPU, memory and network bandwidth available on the end device, the edge device and the cloud device to determine a cooperative inference strategy that minimizes the total processing delay expectation of end-edge-cloud cooperative inference; when the result of the early classification prediction of any output module of the multi-output classification model does not meet the requirement, the output module starts optimal output prediction to determine the optimal output that is most likely to make the result of the classification prediction meet the requirement from the downstream output module and the prediction layer, and instructs the next classification prediction to jump to the optimal output for execution.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, specifically to the field of deep neural network classification, and more specifically, to a deep neural network classification method based on edge-cloud collaboration. Background Technology

[0002] In recent years, the Internet of Things (IoT) has developed rapidly, and its related devices have experienced explosive growth. However, IoT devices typically have limited memory and computing power, making them unable to handle computationally intensive and complex tasks. With the development of the 5G era, the amount of data on the Internet is constantly increasing, and various big data processing needs are emerging. In this environment, the development of IoT faces numerous challenges in terms of energy consumption, transmission latency, bandwidth, application performance, and reliability.

[0003] Therefore, some existing technologies offload tasks from IoT devices to the cloud for cloud computing. However, when dealing with large amounts of data across various tasks, the centralized processing model of cloud computing faces challenges such as bandwidth limitations, high latency, and high energy consumption. To address this, edge computing, as a new computing paradigm, shifts cloud services to servers physically closer to the IoT devices—the edge of the network. Here, the edge refers to any computing and network resources along the path from the terminal device to the cloud computing end. Edge computing migrates data resources from traditional cloud data centers to the user side, shortening the physical distance between the user and information technology, achieving lower data interaction latency, saving network traffic, and thus providing users with a highly stable and low-latency solution.

[0004] Furthermore, with the rapid development of artificial intelligence, machine learning and deep learning technologies are being widely applied in people's lives, including in areas such as computer vision, natural language processing, speech recognition, and autonomous driving. Meanwhile, the vast amounts of multimodal data generated by IoT devices in the physical environment, such as images, audio, and video, need to be sensed. Against this backdrop, it is essential to apply artificial intelligence to the Internet of Things and edge computing. Through AI technology, multimodal data can be analyzed more quickly and high-quality decisions can be made, thus combining AI and edge computing into edge intelligence.

[0005] Edge intelligence runs artificial intelligence models through edge-cloud collaboration, thereby addressing issues such as latency, energy consumption, and privacy encountered in cloud computing. Edge intelligence fully leverages the available data and resources within the hierarchical structure of physical devices, edge devices, and cloud centers to optimize the overall performance of AI model training and inference.

[0006] For inference optimization of neural network models in edge intelligence, there are currently two main existing technologies:

[0007] The first approach is to improve the model structure, such as adding a multi-exit structure to the model and adding an early exit mechanism (see Branchynet). The exit structure adds several branches to the linear structure of the neural network model for inference exit. The early exit mechanism means that, based on the multi-exit structure, during the inference process, the inference operations of all layers of the neural network can be terminated in advance by selecting a branch.

[0008] The second method is model partitioning, which involves dividing the model into layers and having different layers collaborate on inference. Layer-by-layer partitioning means treating each layer of the neural network model as a unit (such as convolutional layers, pooling layers, fully connected layers, etc.), and then dividing and unloading the entire model according to a pre-designed partitioning strategy. For example, the model can be divided into two parts: the first part is deployed and run on edge devices such as mobile phones and cameras, and the second part is deployed and run on cloud devices. The first part of the model is run on the edge device first, and then the intermediate data is transmitted to the cloud server for the second part of the model to run, finally obtaining the model inference results.

[0009] For existing technologies (the first type) that add multiple exit structures to the model, since existing neural networks may have many network layers (such as deep neural networks AlexNet, VGG, ResNet, etc.), many exit structures will be added in the middle, resulting in redundancy of exit structures. The time delay caused by frequent exit calculations may also have a counterproductive effect. Summary of the Invention

[0010] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a deep neural network classification method based on edge-cloud collaboration.

[0011] The objective of this invention is achieved through the following technical solution:

[0012] According to a first aspect of the present invention, a multi-exit classification model based on a deep neural network is provided, comprising: a deep neural network including multiple network layers and a prediction layer, the prediction layer being used to perform classification prediction based on the output of the last network layer; and multiple exit modules, each exit module corresponding to a network layer other than the last network layer and performing advance classification prediction based on the output of that network layer; wherein, when the result of advance classification prediction performed by any exit module fails to meet the target, the exit module initiates optimal exit prediction to determine the optimal exit most likely to achieve the target classification prediction result from the downstream exit modules and the prediction layer, and instructs the next classification prediction to jump to the optimal exit for execution.

[0013] In some embodiments of the present invention, when there is one or more export modules between the optimal export and the export model whose current advance classification prediction results are not up to standard, the calculation process corresponding to the advance classification prediction of the one or more export modules will be cancelled.

[0014] In some embodiments of the present invention, each exit module includes: an intermediate feature extractor for extracting intermediate features based on the output of the network layer corresponding to the current exit module; an intermediate classifier for performing advance classification prediction based on the intermediate features to obtain the advance classification prediction result; an exit verifier for verifying whether the advance classification prediction result meets the standard and, if it does, indicating to exit the classification prediction from the current exit module and determining the final classification result based on the advance classification prediction result; and an exit predictor for determining the optimal exit based on the intermediate features extracted from the current exit model when the advance classification prediction result does not meet the standard, indicating that the next classification prediction will jump to the optimal exit for execution.

[0015] In some embodiments of the present invention, each exit module further includes: an aggregator for aggregating historical memory information obtained from a previous exit module that performed advance classification prediction with the advance classification prediction result output by the classifier of the current exit module to obtain an aggregated advance classification result, wherein the historical memory information is the aggregated advance classification result obtained from a previous exit module that performed advance classification prediction; wherein the exit verifier verifies whether the aggregated advance classification result meets the standard and, if it does, instructs the exit from classification prediction from the current exit module and determines the final classification result based on the aggregated advance classification prediction result.

[0016] In some embodiments of the present invention, the polymerization process includes:

[0017] p i =softmax(αp) i-1 +p′ i )

[0018] Where, p i This represents the result of the aggregated early classification prediction, where softmax(·) represents the softmax function, and p i-1 Represents historical memory information, α represents p i-1 The weights, p′ i This indicates the pre-classification prediction result output by the classifier of the exit module where the current exit verifier is located.

[0019] According to a second aspect of the present invention, a training method for a multi-exit classification model is provided, comprising: offline training of the multi-exit classification model described in the first aspect, wherein the offline training comprises: first performing classification prediction training on a deep neural network; after the deep neural network training is completed, locking the parameters of the deep neural network, and performing classification prediction training and optimal exit prediction training on the exit module respectively.

[0020] In some embodiments of the present invention, the multi-exit classification model includes: an intermediate feature extractor, an intermediate classifier, and an exit predictor, wherein the offline training includes: after the deep neural network is trained, locking the parameters of the deep neural network and performing classification prediction training on the intermediate feature extractor and the intermediate classifier; and after the intermediate feature extractor and the intermediate classifier are trained, locking the parameters of the intermediate feature extractor and performing optimal exit prediction training on the exit predictor.

[0021] In some embodiments of the present invention, the classification prediction training of the deep neural network includes: obtaining a first training set, which includes sample data and corresponding classification labels, wherein the classification labels are the true values ​​of the categories corresponding to the sample data; and using the first training set to train the deep neural network to output the categories corresponding to the sample data based on the sample data.

[0022] In some embodiments of the present invention, the classification prediction training of the intermediate feature extractor and the intermediate classifier includes: obtaining a second training set, which includes network layer feature data output by the network layer corresponding to the intermediate feature extractor to the sample data and the corresponding classification label; using the second training set to train the intermediate feature extractor to extract intermediate features based on the network layer feature data and the intermediate classifier to output the category corresponding to the sample data based on the intermediate features.

[0023] In some embodiments of the present invention, training the exit predictor to perform optimal exit prediction includes: obtaining a third training set, which includes intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located and the corresponding exit labels, wherein the exit labels indicate the optimal exit ground truth value corresponding to the sample data, and training the exit predictor to output the optimal exit based on the intermediate features using the third training set; or obtaining a third training set, which includes intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located, aggregated early classification prediction results and the corresponding exit labels, wherein the exit labels indicate the optimal exit ground truth value, and training the exit predictor to output the optimal exit based on the intermediate features and the aggregated early classification prediction results, wherein the aggregated early classification prediction results are obtained by aggregating the early classification prediction results obtained by the previous exit module that performed early classification prediction and the early classification prediction results obtained by the current exit module.

[0024] According to a third aspect of the present invention, an image classification method is provided, comprising: acquiring a trained multi-exit classification model obtained by training an image classification dataset according to the method described in the second aspect; and classifying an input image using the multi-exit classification model to obtain an image classification result.

[0025] According to a fourth aspect of the present invention, a text classification method is provided, comprising: acquiring a trained multi-exit classification model obtained by training a text classification dataset according to the method described in the second aspect; and classifying input text using the multi-exit classification model to obtain a text classification result.

[0026] According to a fifth aspect of the present invention, a deep neural network classification method based on edge-cloud collaboration is provided, comprising: acquiring a trained multi-exit classification model trained according to the method described in the second aspect; performing online dynamic partitioning of the trained multi-exit classification model based on the available CPU, memory, and network bandwidth on the end device, edge device, and cloud device to determine a collaborative inference strategy that minimizes the expected total processing latency of edge-cloud collaborative inference, the collaborative inference strategy including a first segmentation point and a second segmentation point of the deep neural network in the multi-exit classification model, wherein, for the deep neural network, the portion before and after the first segmentation point is deployed and executed on the end device, the portion after the first segmentation point to the second segmentation point is deployed and executed on the edge device, and the portion after the second segmentation point is deployed and executed on the cloud device, and each exit module and its corresponding network layer are deployed on the same device.

[0027] In some embodiments of the present invention, before online dynamic partitioning, the trained multi-exit classification model is subjected to exit screening to remove exit modules corresponding to some network layers.

[0028] In some embodiments of the present invention, when each exit module determines whether the result of the advance classification prediction meets the standard, if the confidence probability of any category in the corresponding advance classification prediction result is greater than or equal to the preset exit threshold, then the result of the corresponding advance classification prediction meets the standard.

[0029] According to a sixth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the methods of the first aspect, the second aspect, the third aspect, the fourth aspect, and the fifth aspect, or a combination thereof, by executing the executable instructions.

[0030] Compared with the prior art, the advantages of the present invention are as follows:

[0031] This invention provides a multi-exit classification model based on a deep neural network, comprising multiple exit modules. When the result of the preliminary classification prediction performed by any exit module fails to meet the target, that exit module initiates optimal exit prediction to determine the optimal exit most likely to achieve the target classification prediction result from downstream exit modules and the prediction layer, and instructs the next classification prediction to jump to the optimal exit for execution. This reduces some intermediate exit calculations, mitigates the negative impact of excessive exit calculations on inference in a multi-exit module structure, and improves overall inference efficiency. Attached Figure Description

[0032] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0033] Figure 1 This is a schematic diagram of a neural network-oriented quantization device according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram illustrating the structural principle of the high-order bit quantization module for neural networks according to an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of one embodiment of the multi-exit module according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of a partitioning module according to an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram illustrating the impact of the multi-exit structure on the model accuracy in the experimental results according to an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram illustrating the impact of the multi-exit structure on inference efficiency in the experimental results according to an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram illustrating the impact of network bandwidth between mid-range and edge devices on inference efficiency, based on experimental results according to an embodiment of the present invention.

[0040] Figure 8 This is a schematic diagram illustrating the impact of network bandwidth between edge devices and cloud devices on inference efficiency in experimental results according to an embodiment of the present invention;

[0041] Figure 9 This is a schematic diagram illustrating the impact of task arrival frequency on inference efficiency in experimental results according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0043] As mentioned in the background section, existing technologies for multi-exit structures in models suffer from redundancy due to the numerous network layers (such as deep neural networks like AlexNet, VGG, and ResNet) and the resulting redundant exit structures. The time delay caused by frequent exit calculations can also be counterproductive. To address this, this invention provides a multi-exit classification model based on deep neural networks. This model includes multiple exit modules. When the result of an early classification prediction by any exit module fails to meet the target, that module initiates optimal exit prediction to determine the most likely exit from downstream exit modules and prediction layers to achieve the target classification prediction result. The next classification prediction is then directed to jump to the optimal exit. This reduces intermediate exit calculations, mitigates the negative impact of excessive exit calculations on inference in a multi-exit module structure, and improves overall inference efficiency.

[0044] To illustrate the technical principles of this invention, we will first provide a detailed description using the scenario of edge-cloud collaborative reasoning.

[0045] According to one embodiment of the present invention, a deep neural network classification method based on edge-cloud collaboration is provided, see [link to relevant documentation]. Figure 1 This includes the process of offline model training and online dynamic partitioning of the model to adaptively partition the multi-exit classification model based on the resource status of edge and cloud devices. In summary, see [link to relevant documentation]. Figure 2 The deep neural network classification method based on edge-cloud collaboration includes steps A1 and A2. To better understand this invention, each step will be described in detail below with reference to specific embodiments.

[0046] Step A1: Obtain the trained multi-exit classification model obtained by training according to the preset multi-exit classification model training method.

[0047] The following will explain this from the aspects of model structure and model training.

[0048] I. Model Structure

[0049] According to an embodiment of the present invention, a multi-exit classification model based on a deep neural network is provided, comprising: a deep neural network and multiple exit modules (also referred to as exit structures). Preferably, the deep neural network includes multiple network layers and a prediction layer, wherein the prediction layer is used to perform classification prediction based on the output of the last network layer. The deep neural network can be a chain-like deep neural network, such as existing AlexNet, VGG (VGG11, VGG13, VGG16, VGG19, etc.), ResNet (Resnet18, Resnet50, Resnet101, Resnet152), etc.; or a custom structure defined by the implementer, such as a stack of convolutional layers, pooling layers, and fully connected layers (all of which can be called network layers), and finally a prediction layer to form a custom deep neural network; the present invention does not impose any limitations on this. Regarding adding exit modules to network layers, since the last network layer connects to the prediction layer, which is equivalent to the final exit, the last network layer does not need to have an exit module added. Preferably, each exit module corresponds to one network layer (excluding the last network layer) and performs advance classification prediction based on the output of that network layer. If the advance classification prediction result of any exit module fails to meet the target, that exit module initiates optimal exit prediction to determine the optimal exit from downstream exit modules and prediction layers that is most likely to achieve the target classification prediction result, and instructs the next classification prediction to jump to the optimal exit for execution. Preferably, if there are one or more exit modules between the optimal exit and the exit model where the current advance classification prediction result fails, the calculation process corresponding to the advance classification prediction of the one or more exit modules will be canceled. For example, a chain-like deep neural network, ResNet50, can be selected first, and then an exit module for exit can be added after each processing block (BottleBlock, corresponding to a network layer) in ResNet50, thus completing the initialization of the multi-exit classification model. Alternatively, a non-chain-like deep neural network is also feasible; the execution order of each network layer and its exit module can be customized and executed according to that order. This invention does not impose any restrictions on this.The technical solution of this embodiment can achieve at least the following beneficial technical effects: In the prior art, when the result predicted by an exit module is found to be unsatisfactory after running an exit module, the current exit module cannot exit the prediction process and will predict again in the next exit module in sequence. When there are many network layers and many exit modules fail to exit the prediction in the early stage, the overall prediction latency may be greater than that of directly running a deep neural network for prediction, resulting in the counterproductive effect of the multi-exit classification model. In this embodiment, when an exit module fails to exit the prediction, an optimal exit prediction is performed, and the next classification prediction is instructed to jump to the optimal exit for execution. The calculation of some intermediate exit modules is shielded, which can reduce some intermediate exit calculations, reduce the negative impact of too many exit calculations on inference in the multi-exit module structure, and improve the overall inference efficiency.

[0050] According to one embodiment of the present invention, see Figure 3 Each egress module includes: a tool for determining the output D of the network layer corresponding to the current egress module. i Extracting intermediate features F i Intermediate feature extractor 10; used to extract intermediate features (or feature data) F i Perform early classification prediction to obtain the early classification prediction result p′ i (or the probability vector p′ of the category) i The intermediate classifier 20 is used to process the historical memory information p obtained from the previous exit module that performs advance classification prediction. i-1 The result p′ of the early classification prediction output by the classifier of the current export module i Aggregator 30 performs aggregation processing to obtain the aggregated pre-classification results, wherein historical memory information p i-1 This refers to the aggregated pre-classification result obtained from the previous pre-classification prediction output module; p is used to verify the aggregated pre-classification prediction result. i Whether the criteria are met, and if so, indicating exit from the current exit module for classification prediction, and based on the results of the aggregated advance classification prediction p i Exit validator 40 determines the final classification result; used to extract intermediate features F based on the current exit model when the pre-classification prediction result fails to meet the target. i Determine the optimal exit to indicate the next classification prediction jump to the optimal exit EP. i The exit predictor 50 is executed. The intermediate feature extractor 10 can employ a single convolutional layer, or a structure consisting of one convolutional layer and one pooling layer, to avoid excessive additional computation time from the exit. The intermediate classifier extracts the intermediate features F. i Processed as a probability vector p′ of the class i It can employ a single-layer or double-layer sensor.

[0051] Preferably, the polymerization process includes:

[0052] p i =softmax(αp) i-1 +p′ i )

[0053] Where, p i This represents the result of the aggregated early classification prediction, where softmax(·) represents the softmax function, and p i-1 Represents historical memory information, α represents p i-1 The weights, p′ i This represents the pre-classification prediction result output by the classifier of the exit module where the current exit validator is located. α is a hyperparameter that can be manually set, ranging from [0, 1]. Since there is no exit module preceding the first exit module, p0 can be set to a zero vector (or a vector composed of values ​​close to 0). During the execution of the exit module, a preset class threshold vector τ is used to evaluate p0. i Perform a verification, if If the target is met, the classification prediction process will exit, and the result with the highest confidence level will be selected as the final classification prediction result, i.e., the output will be max(p). i As a result of classification prediction, otherwise, the model continues to run and starts the exit predictor to predict the optimal exit, setting p... i Historical memory information is transmitted to the exit module corresponding to the optimal exit. The exit predictor uses feature data F i Predict the next exit location with the highest probability of exiting (EP) i (Optimal exit) is used to reduce the latency of excessive exits. The exit predictor can also select a single-layer or dual-layer perceptron. The technical solution of this embodiment can achieve at least the following beneficial technical effects: This embodiment designs an exit module with historical memory information aggregation and exit prediction functions. Through the aggregated memory information, it will not only utilize the prediction results of the current exit module, but also the prediction results of the previous exit module. While ensuring the accuracy of the model, it improves the exit rate, thereby reducing the inference time latency.

[0054] It should be understood that, in addition to the above-mentioned pre-classification results using aggregation, implementers may also use other methods. For example, based on the aforementioned embodiment, aggregator 30 can be removed, and the output of intermediate classifier 20 can be passed to exit verifier 40. Exit verifier uses a preset category threshold vector τ to apply the pre-classification results output by intermediate classifier 20 to p′. i Perform a verification, if If the target is met, the classification prediction process will exit, and the output will be max(p′). iAs a result of the classification prediction, the model continues to run and starts the export predictor to predict the optimal export.

[0055] II. Model Training

[0056] The model training component includes offline training and model preset values. According to one embodiment of the present invention, the multi-exit classification model of the aforementioned embodiment is trained offline. The offline training includes: first, performing classification prediction training on the deep neural network; after the deep neural network training is completed, locking the parameters of the deep neural network, and then performing classification prediction training and optimal exit prediction training on the exit modules respectively. According to one embodiment of the present invention, offline training can be implemented in an offline cloud device environment (of course, it can also be implemented on a computer in the form of a server or workstation). For the deep neural network, offline training of the deep neural network is first performed, wherein the trainable parameters of the deep neural network (such as weight parameters and / or bias parameters) are trained and saved. After training, the layer structure information of each layer of the deep neural network is processed according to the model structure information of the deep neural network. The layer structure information includes the output data size and computational cost of the network layers. Next, while freezing the trainable parameters (model backbone parameters) of the deep neural network, the exit modules are trained. Preferably, in the case where the multi-exit classification model includes an intermediate feature extractor, an intermediate classifier, and an exit predictor, offline training includes: after the deep neural network is trained, locking the parameters of the deep neural network and training the intermediate feature extractor and intermediate classifier for classification prediction; and after the intermediate feature extractor and intermediate classifier are trained, locking the parameters of the intermediate feature extractor and training the exit predictor for optimal exit prediction. The classification prediction training can use a subset extracted from existing classification datasets (e.g., image classification, text classification, etc.) as the training set. The loss function used for classification prediction training can be the cross-entropy loss function.

[0057] According to one embodiment of the present invention, training a deep neural network for classification prediction includes: acquiring a first training set, which includes sample data and corresponding classification labels, wherein the classification labels are the true values ​​of the categories corresponding to the sample data; and training the deep neural network using the first training set to output the categories corresponding to the sample data. Preferably, during training, the deep neural network outputs the categories corresponding to the sample data, calculates the gradient of the classification loss based on the output categories and classification labels, and updates the trainable parameters of the deep neural network based on backpropagation of the gradient.

[0058] According to one embodiment of the present invention, training an intermediate feature extractor and an intermediate classifier for classification prediction includes: obtaining a second training set, which includes network layer feature data output by the network layer corresponding to the intermediate feature extractor to sample data and corresponding classification labels; and training the intermediate feature extractor to extract intermediate features based on the network layer feature data and the intermediate classifier to output the category corresponding to the sample data based on the intermediate features using the second training set. Preferably, during training, the intermediate classifier outputs the category corresponding to the sample data based on the intermediate features, calculates the gradient of the classification loss at the exit module based on the category corresponding to the sample data output by the intermediate classifier and the classification label, and updates the trainable parameters of the intermediate feature extractor and the intermediate classifier based on backpropagation of the gradient. It should be understood that the network layer feature data output by the network layer of the second training set to sample data can be obtained from the sample data in the first training set; of course, it can be obtained from other sample data, and the present invention does not impose any limitations on this.

[0059] According to one embodiment of the present invention, training an exit predictor for optimal exit prediction includes: acquiring a third training set, which includes intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located and corresponding exit labels, wherein the exit labels indicate the optimal exit ground truth value corresponding to the sample data; and training the exit predictor to output the optimal exit based on the intermediate features using the third training set. Alternatively, additional information can be combined to predict the optimal exit to improve the accuracy of the optimal exit prediction. According to one embodiment of the present invention, training an exit predictor for optimal exit prediction includes: acquiring a third training set, which includes intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located, aggregated early classification prediction results, and corresponding exit labels, wherein the exit labels indicate the optimal exit ground truth value; and training the exit predictor to output the optimal exit based on the intermediate features and aggregated early classification prediction results using the third training set, wherein the aggregated early classification prediction results are obtained by aggregating the early classification prediction results obtained by the previous exit module performing early classification prediction and the early classification prediction results obtained by the current exit module. Preferably, the optimal exit prediction training is also classification training, and the cross-entropy loss function can be used. During training, the loss is calculated and the gradient is obtained based on the optimal exit and exit label output by the exit predictor, and the trainable parameters of the exit predictor are updated based on the gradient backpropagation.

[0060] Model presetting is the process of presetting the number of exits and exit thresholds related to the model. According to one embodiment of the present invention, before online dynamic partitioning, the trained multi-exit classification model undergoes exit screening to remove some exit modules corresponding to certain network layers. Preferably, exit screening includes deleting exit modules whose prediction accuracy is lower than a preset accuracy threshold, deleting exit modules in the training sample data whose exit rate is lower than a preset exit rate, deleting exit modules whose accuracy drop caused by deleting a single exit module is less than a predetermined impact threshold, or a combination thereof. The exit screening function filters and deletes some exit modules to reduce the additional latency caused by too many exits. Of course, exit screening can also be omitted, or other methods can be used for exit screening, for example: first, delete exits with accuracy lower than a set value; second, delete exit modules whose usage frequency in the training samples is lower than a set value (i.e., exit modules whose exit rate is lower than a preset exit rate); finally, delete exit modules in ascending order of accuracy drop caused by deleting a single exit module, until the number of exit modules meets the set value.

[0061] According to one embodiment of the present invention, the preset value of the exit threshold (or category threshold vector) can be customized based on the implementer's experience. Alternatively, the exit threshold can also be determined using the test results of the offline-trained model on the test dataset and the following formula:

[0062]

[0063] Where τ is the exit threshold, f τ (x) represents the output result of the model when performing multi-exit inference on the input sample data x under the threshold of this category, and [·] represents the condition when f is true (i.e., f τ (x) = label x The formula returns 1 if the accuracy is 1 and 0 otherwise, and aims to find the category threshold vector that maximizes the accuracy on the test dataset. Therefore, according to one embodiment of the present invention, when each exit module determines whether the result of the pre-classification prediction meets the standard, the corresponding pre-classification prediction result meets the standard if the confidence probability of any category in the corresponding pre-classification prediction result is greater than or equal to the preset exit threshold.

[0064] Step A2: Based on the available CPU, memory, and network bandwidth on the end device, edge device, and cloud device, perform online dynamic partitioning of the trained multi-exit classification model to determine a collaborative inference strategy that minimizes the expected total processing latency of end-edge-cloud collaborative inference. This collaborative inference strategy includes a first segmentation point and a second segmentation point for the deep neural network in the multi-exit classification model. For the deep neural network, the portion before and after the first segmentation point is deployed and executed on the end device, the portion after the first segmentation point to the second segmentation point is deployed and executed on the edge device, and the portion after the second segmentation point is deployed and executed on the cloud device. Each exit module and its corresponding network layer are deployed on the same device.

[0065] According to one embodiment of the present invention, for online dynamic partitioning, this part can be implemented by a dynamic partitioning algorithm running in the online state. Preferably, the cloud device (such as a cloud server) first deploys the pre-set multi-exit classification model and related data transmission to edge devices (such as edge servers) and end devices (such as mobile phones, personal computers, smartwatches, VR glasses, etc.). The end devices can then act as the main controller to perform dynamic partitioning operations on the model. Each end device will dynamically partition the running model according to the pre-set multi-exit data and the device's available resource status information, including the device's available CPU, memory, and network bandwidth. Through the dynamic partitioning algorithm, the model's partitioning will adaptively change according to environmental changes, dividing the model into three parts, which are then collaboratively inferred by the edge, cloud, and end devices (i.e., end devices, edge devices, and cloud devices), thus improving the computational efficiency of model inference. The online model partitioning algorithm makes decentralized decisions across various devices. For a single device (end device or edge device), the dynamic partitioning function can be implemented by, for example, Figure 4 The partitioning module shown comprises a device monitoring unit, a parameter control unit, and a partitioning decision unit. As an illustration, according to an embodiment of the present invention, the functions of each unit in the partitioning module are as follows:

[0066] Preferably, the device monitoring unit is responsible for acquiring (e.g., periodically, timed) the available CPU, memory, and network bandwidth of end devices, edge devices, and cloud devices. The device monitoring unit may also acquire and store model-related data from the parameter control unit.

[0067] Preferably, the parameter control unit is responsible for managing various model-related parameters, including layer structure information, exit threshold, exit probability, and sub-network latency. Initialization is performed first, during the initial model deployment. This initialization process records the trained multi-exit classification model and pre-set hyperparameter values, such as layer structure information, exit threshold, and exit probability, locally on the device. Sub-network latency includes the predicted computation time latency of each network layer, exit computation time latency, exit predictor computation latency, and data transmission latency. Network layer computation time latency is predicted by building a regression model based on the computational load of each network layer to be computed, the available CPU, and memory on the device. Exit computation time latency is predicted by building a regression model based on the computational load of the intermediate feature extractor and classifier, the available CPU, and memory on the device. Exit predictor computation latency is predicted by building a regression model based on the computational load of the exit predictor, the available CPU, and memory on the device. Of course, the aforementioned network layer computation time latency, exit computation time latency, and exit predictor computation latency can also be predicted using other methods, such as by establishing a pre-defined fitting formula. Data transmission latency can be generated and predicted by fitting the formula shown below:

[0068]

[0069] in, D represents the output data transmission time of the i-th network layer. i Let B represent the output data size of the i-th layer network, and let B represent the network bandwidth. λ and γ are two variables, λ being a proportional variable and γ a correction variable, determined through function fitting. Of course, data transmission time delay can also be predicted using a regression model to obtain other implementation methods.

[0070] Preferably, the exit probability includes the exit conditional probability and the probability of entering the exit, used to predict the expected processing delay. The exit probability can be customized by the implementer based on experience, or calculated according to a predetermined exit probability calculation method. According to one embodiment of the present invention, the exit probability calculation can estimate the exit conditional probability of the model exiting the corresponding exit (including the exit module and the prediction layer) and the entry probability of entering the corresponding exit module (including the exit module and the prediction layer) based on the exit situation of sample data during inference on the test dataset. Preferably, the exit conditional probability (matrix) can be determined according to the following calculation method:

[0071]

[0072] Among them, P i,j This represents the probability that the user fails to exit at the i-th exit but successfully exits at or before the j-th exit. This represents the number of samples in the test dataset that failed to exit from the i-th exit. n represents the number of samples in the test dataset that failed to exit from the i-th exit but successfully exited from the j-th exit or earlier. e This represents the number of exports in the multi-export classification model (the total number of export modules and prediction layers). To prevent samples from having no correlation distribution during probability calculation, (n...) is added. e Smoothing is performed using -i+1).

[0073] Preferably, the probability of entering the exit can be determined using the following calculation method:

[0074]

[0075] Among them, Q i,j This represents the probability that the test data focuses on the failure to exit at the i-th exit and the entry at the j-th exit. This represents the number of samples in the test dataset that failed to exit from the i-th exit and entered the j-th exit, where n is the number of samples. e This represents the number of exports in the multi-export classification model. Similarly, adding (n...) e Smoothing is performed using -i+1).

[0076] It should be understood that the exit probability can also be calculated in other ways to obtain other implementation methods. For example, the test dataset of the previous embodiment can be changed to the training dataset or a combination of the training dataset and the test dataset to obtain other implementation methods.

[0077] Preferably, the partitioning decision unit is used to make partitioning decisions based on the model-related parameters transmitted by the parameter control unit. The core idea of ​​the partitioning decision unit is to minimize the expected total processing latency (or minimize the expected time delay). Preferably, online dynamic partitioning of the trained multi-exit classification model includes determining a collaborative inference strategy that minimizes the expected total processing latency based on the available CPU, memory, network bandwidth on the end device, edge device, and cloud device, as well as preset exit condition probabilities and entry / exit probabilities.

[0078] According to one embodiment of the present invention, the processing latency expectation includes the latency expectation of each network layer, the latency expectation of each egress module, and the latency expectation of data transmission. Illustratively, wherein:

[0079] The expected time delay of each network layer can be determined as follows:

[0080]

[0081] Among them, l cur Indicates the current network layer reached, l last σ represents the network layer number corresponding to the last exit module that did not exit. d σ represents the first dividing point. eP represents the second dividing point. last,i This represents the probability that the exit from the last exit module fails but the exit from the i-th exit module succeeds. This represents the predicted network layer i-th network layer computation time delay at the end device. This represents the computational delay of the predicted i-th network layer at the edge device's network layer. This represents the predicted computation time delay of the i-th network layer in the cloud device's network layer, where n is the number of layers. L Indicates the number of network layers. cur and l last This indicates the input variable, and the same applies below.

[0082] Preferably, the expected time delay for each export module can be determined as follows:

[0083]

[0084] Among them, l cur Indicates the current network layer reached, l last σ represents the network layer number of the last exit that has not yet exited. d σ represents the first dividing point. e P represents the second dividing point. last,i P represents the probability that the exit from the last exit module fails but the exit from the i-th exit module succeeds. last,i-1 Q represents the probability that the exit from the last exit module fails but the exit from the (i-1)th exit module succeeds. last,i μ represents the probability that the user fails to exit at the last exit but enters at the i-th exit. i This is a variable between 0 and 1. A value of 1 indicates that there is an exit point below the i-th network layer, while a value of 0 indicates that there is no exit point below the i-th network layer (the exit point was removed during the filtering process). This indicates the time delay of the egress calculation at the end device for the egress module corresponding to the i-th network layer. This indicates the time delay of the exit module corresponding to the i-th network layer at the exit point of the edge device. This indicates the computation time delay of the egress module corresponding to the i-th network layer at the cloud device's egress point. This represents the calculation delay of the exit predictor of the exit module corresponding to the i-th network layer at the end device. This represents the calculation delay of the exit predictor of the exit module corresponding to the i-th network layer at the exit predictor of the edge device. Let n represent the calculation latency of the egress predictor of the egress module corresponding to the i-th network layer in the cloud device's egress predictor, respectively. L Indicates the number of network layers.

[0085] Preferably, the expected data transmission delay can be determined as follows:

[0086]

[0087] Among them, l cur Indicates the current network layer reached, l last σ represents the network layer number of the last exit that has not yet exited. d σ represents the first dividing point. e Indicates the second dividing point. This indicates that the exit of the last exit module failed and at the σth... d The probability of a successful exit from an exit module or previously, [·] represents the probability when the condition is true (e.g., σ). d <n L , or σ e <n L Returns 1 if the condition is met, otherwise returns 0. This represents the transmission time delay of the output data of the i-th network layer from the end device to the edge device. γ represents the transmission time delay of the output data of the i-th network layer from the edge device to the cloud device. (d) γ represents the transmission time delay between the edge device and the end device when sending the result back. (e) Indicates the transmission time delay between the cloud device and the edge device when sending the result back, n L This indicates the network layer number. Of course, in some implementation scenarios, the transmission time delay of result return can be removed to obtain other implementation methods.

[0088] Preferably, the partition decision unit can calculate the total expected processing latency in the following manner:

[0089] E(T Total (l cur , l last , σ d , σ e ))=E(T Layer )+E(T Exit )+E(T Trans )

[0090] The final optimization goal is:

[0091] min E(T Total (l cur , l last , σ d , σ e ))

[0092]

[0093] Among them, σ can be solved by optimization algorithm.d , σ e Two split points, after which the multi-exit classification model's computation is divided into three parts, the 1-σ of the deep neural network. d The layer and its corresponding exit module are inferred and calculated by the end device, (σ d +1)-σ e The layer and its corresponding exit module are offloaded to the edge device for inference calculation, (σ e +1)-n L The layer and its corresponding exit module are offloaded to the cloud device for inference computation. After offloading, the multi-exit classification model will still follow the exit strategy during inference.

[0094] In addition to obtaining the expected time delays of each network layer, each exit module, and the data transmission delay as described above, historical data can also be stored in the parameter control unit for predicting collaborative inference strategies. According to one embodiment of the present invention, the historical data includes the historical expected time delays of each network layer, each exit module, and the data transmission delay, with a timestamp added to each data entry. When the difference between the timestamp of the model to be partitioned and the latest timestamp of the historical data meets a certain range, prediction can be made based on the historical data; the timestamp difference range is manually defined.

[0095] In addition to collaborative inference on edge and cloud, the multi-exit classification model of the present invention can also be inferred by a single device; or by a scenario of collaborative inference by the end device and the edge device (for example, by removing the latency calculation part related to the cloud device in the aforementioned embodiment), and the end device and the cloud device. The present invention does not impose any restrictions on this.

[0096] According to an embodiment of the present invention, an image classification method is provided, comprising: obtaining a trained multi-exit classification model trained using an image classification training set according to the aforementioned training method for a multi-exit classification model; and classifying an input image using the multi-exit classification model to obtain an image classification result. The image classification training set can be obtained from existing image classification datasets, such as CIFAR-10, CIFAR-100, MNIST, STL_10, ImageNet, etc.

[0097] According to an embodiment of the present invention, a text classification method is provided, comprising: obtaining a trained multi-exit classification model obtained by training a text classification training set according to the aforementioned training method for multi-exit classification models; and classifying input text using the multi-exit classification model to obtain a text classification result. The text classification training set can be obtained from existing text classification datasets, such as OpinRank, WordNet, IMDB, Yelp Reviews, etc.

[0098] To verify the effectiveness of the embodiments of the present invention, the inventors conducted an experiment, the process of which is described below.

[0099] (1) Experimental setup

[0100] The experiment was conducted through physical simulation. The mid-range device used was an NVIDIA Jetson Nano, the edge device was a PC with an Intel(R) Core(TM) i7-6700HQ CPU@2.60GHz and an NVIDIA GeForce GTX960M GPU, and the cloud server used an Intel(R) Xeon(R) Silver 4114 CPU@2.20GHz and an NVIDIA Tesla V100 GPU. To facilitate the construction and use of deep neural network models, the experiment used the PyTorch deep learning framework. ZeroMQ was used as the message queue for network communication, employing a publish-subscribe pattern. The task was image classification, tested using the CIFAR-10 and CIFAR-100 datasets. AlexNet, VGG16, ResNet34, and InceptionV3 were selected as baseline models for the experiments.

[0101] (2) Model accuracy

[0102] First, experiments were conducted to test the accuracy changes of the baseline models after adding the proposed multi-exit structure. The baseline models used were AlexNet, VGG16, ResNet34, and InceptionV3, with CIFAR-10 and CIFAR-100 datasets used for training and testing, respectively. During the baseline model training phase, each model used the same SGD optimizer with the same parameters on each dataset, and the parameters obtained from 50 training rounds with a batch size of 128 were used as the parameters for subsequent experiments. The simplified feature extraction in the exit structure used a 3x3 convolutional layer and an adaptive average pooling layer, while the classifier and exit prediction both used a fully connected layer. After adding the above exit structure to each unit module of the baseline model (a unit module refers to a linearly separable module of the baseline model, which can be composed of a single layer or multiple layers), the parameters of the backbone structure of the baseline network were frozen, and the parameters of the feature extraction and classifier modules in the multi-exit structure were trained. Finally, the exit predictor was trained. During the coarse screening of exits, exits with an accuracy lower than 75% of the original accuracy were removed. The subsequent process follows the methods described above, and finally, the accuracy of the model using the multi-exit structure described in this paper can be obtained as follows: Figure 5As shown in the figure, the test results of the four benchmark models on two datasets demonstrate that adding the multi-exit structure described in this paper does not significantly decrease the model's accuracy, and the model's precision can be guaranteed.

[0103] (3) Reasoning efficiency

[0104] First, this paper discusses the improvement in inference efficiency brought to the baseline model by the proposed multi-exit structure. The baseline model and the baseline model with the multi-exit structure are obtained through the above method. The inference time of a single NVIDIA Jetson Nano terminal device is used as the standard for judgment. Three inference methods are used: the first is the baseline model without the exit structure; the second is the baseline model with the early exit mechanism of the multi-exit structure but without using memory information aggregation and exit prediction; the third is the baseline model with the multi-exit structure using memory information and exit prediction. The average time obtained by running each of the three methods 1000 times in the same device environment is used as the final result. Figure 6 As shown in the figure, the special multi-exit structure refers to the multi-exit structure with memorized information aggregation and exit prediction described in this paper. It can be seen from the figure that in the relatively simple dataset CIFAR-10, the ordinary multi-exit structure performs well in deep neural networks like ResNet34 and InceptionV3, but performs poorly in shallower neural networks like AlexNet and VGG16, falling short of the benchmark. This is mainly because the model has difficulty exiting at shallower locations, and the additional exit operations increase inference time. The special multi-exit structure described in this paper performs better in both shallow and deep neural networks. On the other hand, in the more complex dataset CIFAR-100, the early exit rate of the model under the multi-exit structure decreases, and the inference efficiency improvement brought by the ordinary multi-exit structure is smaller, even worse than the benchmark model. However, the special multi-exit structure described in this paper still shows a significant acceleration in inference time.

[0105] Next, we will examine the impact of the dynamic partitioning algorithm on inference efficiency. We will continue to use four benchmark models and conduct experiments using different partitioning algorithms under the above multi-exit structure. Since the main purpose is to verify the effectiveness of the partitioning algorithm, this experiment will only be conducted on the CIFAR-10 dataset. The methods and mechanisms used are as follows: (1) OnlyDevice: The model runs only on the terminal device. (2) OnlyEdge: The model runs only on the edge device. (3) The model runs only on the cloud device. (3) Edge[1]: Edge-end collaborative inference, adaptively selects the split point based on time prediction and exits from the fixed exit. (4) Neurosurgeon[2]: Edge-cloud collaborative inference, adaptively selects the split point based on time prediction and does not exit from the branch exit. (5) SPINN[3]: Edge-cloud collaborative inference, selects the exit at the 15%, 30%, 45%, 60%, 75%, and 90% positions of the model and adaptively selects the split point based on time prediction. (6) MyMethod: The method described in this paper.

[0106] To verify the stability of the partitioning algorithm under dynamic conditions, the effects of changes in network bandwidth and device load on the partitioning algorithm were studied.

[0107] First, we investigated the impact of network bandwidth variations on the partitioning algorithm. We used the WonderShaper tool on Linux to limit network card bandwidth for experiments. Each experiment was conducted under the same environment, except for the network bandwidth, and the results were the average of 1000 inference iterations of the model on the test set. Initially, the network bandwidth between the edge and cloud was fixed at 200 Mbps. Then, the network bandwidth between the endpoints was varied, and the results were as follows: Figure 7 As shown, under the four benchmark models, the partitioning algorithm described in this paper outperforms or is no worse than other partitioning methods in most cases, and exhibits good stability. With increasing network bandwidth, this method can also adaptively select the partitioning point, allowing more parts of the model to be offloaded to the edge and cloud, thus achieving better inference efficiency. Similarly, with a fixed network bandwidth of 50Mbps between the edge and the cloud, changing the network bandwidth between the edge and the cloud yields the following results: Figure 8 As shown, in this experimental environment, the network bandwidth between edge clouds has a relatively small impact, but the partitioning algorithm described in this paper still has the best inference efficiency.

[0108] On the other hand, the impact of device load variations on the partitioning algorithm is also considered. Device load can be varied by changing the arrival frequency of model inference tasks. Therefore, the following experiment uses variations in task arrival frequency for testing. The model preparation is the same as the experimental procedure described above. Subsequently, the network bandwidth between the endpoint and the edge is fixed at 25Mbps, and the network bandwidth between the endpoint and the edge is fixed at 200Mbps. The task arrival frequency represents the average number of tasks arriving per second. With a time interval of 0.01 seconds, the arrival time of tasks within 1 second is randomly generated according to the set task arrival frequency and using a Poisson distribution. Four different models are used at different task arrival frequencies, and the experimental tasks are generated multiple times using the method described above for generating model inference task arrival times. The average waiting time of the model inference tasks is tested experimentally, and the results are as follows. Figure 9 As shown, it can be seen that as the frequency of task arrival increases, the device load increases and the overall inference waiting time increases. However, the method described in this paper can still maintain the optimal average inference waiting time relatively stably.

[0109] In summary, the edge-cloud collaborative deep neural network classification method proposed in this invention optimizes the inference time latency of the deep neural network model under an edge-cloud architecture through offline and online collaborative stages. It designs a multi-exit structure with aggregation of memorized information (corresponding to historical memory information) and exit prediction functions. The aggregated memory information improves the exit rate while ensuring model accuracy, thereby reducing inference time latency. Optimal exit prediction further reduces the time latency caused by frequent exit calculations. In the online stage, based on the runtime state of the inference calculation task, a dynamic partitioning algorithm is used to determine the model segmentation point to minimize the expected inference latency, thereby partitioning and unloading the deep neural network model. This provides a finer-grained advantage in addressing the inference latency stability problem caused by fluctuations in network bandwidth, device load, and other factors. This invention is used for deep neural network inference in edge-cloud collaborative systems, reducing inference latency and improving system real-time performance and stability while ensuring model accuracy.

[0110] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0111] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0112] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0113] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-exit classification system based on a deep neural network, used for classifying images or text, characterized in that, include: A deep neural network includes multiple network layers and a prediction layer, wherein the prediction layer is used to make classification predictions based on the output of the last network layer; as well as Multiple exit modules, each corresponding to one of the network layers except the last one, and performing pre-classification prediction based on the output of that network layer. Each exit module includes: An intermediate feature extractor used to extract intermediate features based on the output of the network layer corresponding to the current export module; An intermediate classifier used to perform early classification prediction based on the intermediate features to obtain the result of the early classification prediction; An exit verifier is used to verify whether the results of the advance classification prediction meet the standards, and when they do, to indicate that the classification prediction should be exited from the current exit module and to determine the final classification result based on the results of the advance classification prediction. An exit predictor used to determine the optimal exit based on intermediate features extracted from the current exit module when the results of the previous classification prediction do not meet the target, so as to indicate the next classification prediction to jump to the optimal exit for execution. An aggregator is used to aggregate the historical memory information obtained from the previous exit module that performed advance classification prediction with the advance classification prediction result output by the classifier of the current exit module to obtain an aggregated advance classification result. The historical memory information is the aggregated advance classification result obtained from the previous exit module that performed advance classification prediction. The exit verifier verifies whether the aggregated pre-classification result meets the standard and, if it does, instructs the exit from the current exit module for classification prediction and determines the final classification result based on the aggregated pre-classification prediction result. If the result of the advance classification prediction in any export module fails to meet the target, the export module initiates the optimal export prediction to determine the optimal export from the downstream export modules and prediction layers that is most likely to make the classification prediction result meet the target, and instructs the next classification prediction to jump to the optimal export for execution.

2. The multi-exit classification system according to claim 1, characterized in that, When there is one or more export modules between the optimal export and the export for which the current advance classification prediction result does not meet the standard, the calculation process corresponding to the advance classification prediction of the one or more export modules will be cancelled.

3. The multi-exit classification system according to claim 1, characterized in that, The processing methods for aggregation include: in, This represents the result of the aggregated early classification prediction. express function, Representing historical memory information, express The weight, This indicates the pre-classification prediction result output by the classifier of the exit module where the current exit verifier is located.

4. A training method for a multi-exit classification system, characterized in that, include: Offline training is performed on the multi-exit classification system according to any one of claims 1-3, wherein the offline training includes: first performing classification prediction training on the deep neural network; after the deep neural network training is completed, locking the parameters of the deep neural network, and performing classification prediction training and optimal exit prediction training on the exit module respectively.

5. The training method according to claim 4, characterized in that, The multi-exit classification system includes: an intermediate feature extractor, an intermediate classifier, and an exit predictor, wherein the offline training includes: After the deep neural network is trained, its parameters are locked, and the intermediate feature extractor and intermediate classifier are trained for classification prediction. After the intermediate feature extractor and intermediate classifier are trained, the parameters of the intermediate feature extractor are locked, and the exit predictor is trained to perform optimal exit prediction.

6. The training method according to claim 5, characterized in that, The training of the deep neural network for classification and prediction includes: Obtain the first training set, which includes sample data and corresponding classification labels, wherein the classification labels are the true values ​​of the categories corresponding to the sample data; The deep neural network is trained using the first training set and outputs the category corresponding to the sample data based on the sample data.

7. The training method according to claim 6, characterized in that, The training of the intermediate feature extractor and intermediate classifier for classification prediction includes: Obtain the second training set, which includes the network layer feature data and corresponding classification labels output by the network layer corresponding to the intermediate feature extractor to the sample data; The intermediate feature extractor is trained using the second training set to extract intermediate features from the network layer feature data, and the intermediate classifier outputs the category corresponding to the sample data based on the intermediate features.

8. The training method according to claim 7, characterized in that, The optimal export prediction training for the export predictor includes: Obtain a third training set, which includes the intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located, and the corresponding exit labels. The exit labels indicate the optimal exit ground truth value corresponding to the sample data. Use the third training set to train the exit predictor to output the optimal exit based on the intermediate features; or Obtain a third training set, which includes intermediate features of the corresponding sample data extracted by the intermediate feature extractor of the exit module where the exit predictor is located, aggregated advance classification prediction results, and corresponding exit labels. The exit labels indicate the optimal exit truth value. Use the third training set to train the exit predictor to output the optimal exit based on the intermediate features and aggregated advance classification prediction results. The aggregated advance classification prediction results are obtained by aggregating the advance classification prediction results obtained by the previous exit module that performed advance classification prediction and the advance classification prediction results obtained by the current exit module.

9. An image classification method, characterized in that, include: Obtain a trained multi-exit classification system obtained by training an image classification dataset according to any one of claims 4-8; The input image is classified using the multi-exit classification system to obtain the image classification result.

10. A text classification method, characterized in that, include: Obtain a trained multi-exit classification system obtained by training a text classification dataset according to the method described in any one of claims 4-8; The input text is classified using the multi-exit classification system to obtain the text classification result.

11. A deep neural network classification method based on edge-cloud collaboration, characterized in that, include: Obtain a trained multi-exit classification system obtained by the method according to any one of claims 4-8; Based on the available CPU, memory, and network bandwidth on end devices, edge devices, and cloud devices, the trained multi-exit classification system is dynamically partitioned online to determine a collaborative inference strategy that minimizes the expected total processing latency of end-edge-cloud collaborative inference. This collaborative inference strategy includes the first and second segmentation points of the deep neural network in the multi-exit classification system. For deep neural networks, the portion before and after the first segmentation point is deployed on the end device, the portion from the first segmentation point to the second segmentation point is deployed on the edge device, and the portion after the second segmentation point is deployed on the cloud device. Each exit module and its corresponding network layer are deployed on the same device.

12. The method according to claim 11, characterized in that, Before online dynamic partitioning, the trained multi-exit classification system is subjected to exit filtering to remove exit modules corresponding to some network layers.

13. The method according to claim 11, characterized in that, When each exit module determines whether the result of the advance classification prediction meets the standard, if the confidence probability of any category in the corresponding advance classification prediction result is greater than or equal to the preset exit threshold, then the result of the advance classification prediction meets the standard.

14. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 4 to 13.

15. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 4 to 13 by executing the executable instructions.

Images