Method and system for processing tasks

Abstract

The present application relates to a method for processing a task, comprising: a neural network (NN) model for processing the task receiving input data, the NN model comprising a plurality of layers in a sequence, one or more premature-exit points disposed between the plurality of layers, dividing the NN model into two or more sub-blocks, each sub-block comprising one or more layers, an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more pre-exit points; the output sub-NN at the early leaving point generates early leaving output data related to the task based on the feature map output by the sub-block corresponding to the early leaving point, and the confidence estimation sub-NN at the early leaving point predicts the confidence of the early leaving output data of the early leaving point based on the feature map output by the sub-block corresponding to the early leaving point; and on the basis of the predicted confidence coefficient, determining that the early-leaving output data is used as an output result of the task or providing the feature map output by the sub-block corresponding to the early-leaving point to the next sub-block for processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for processing tasks in an early retirement manner. Background Art

[0002] Generally, the more layers a deep neural network (DNN) has, the more accurately it can handle artificial intelligence tasks. Taking the convolutional neural network (CNN) as an example, it is generally believed that the more layers a CNN has, the better the performance of the network output becomes as the network depth increases. However, deeper neural networks mean larger scale, slower speed, and higher computational requirements.

[0003] In order to use computing resources more efficiently, an early exit mechanism is used in classification tasks using CNN. There are two main existing early exit methods: one is to determine whether to exit or continue reasoning based on whether the confidence of the classification result of the current layer exceeds the threshold. This method is called the confidence-based early exit method; the other is to determine whether to exit or continue reasoning by checking whether the classification result of the current layer is the same as the classification result of the previous layer. This method is called the patience-based early exit method.

[0004] For classification tasks in artificial intelligence tasks such as computer vision and natural language processing, the above-mentioned confidence-based early exit method and patience-based early exit method can work well using the confidence value in the classification result. However, for artificial intelligence tasks that are not classified tasks, it is difficult to determine the confidence and patience of the intermediate output results at the early exit point during the reasoning process of CNN. For example, for object detection tasks in computer vision, the intermediate output results of each layer of the CNN model are indications of objects in the image, such as bounding boxes, and the number and size of objects in each intermediate output result are not fixed, so it is difficult to evaluate the confidence of the intermediate output results, and it is also difficult to evaluate whether the two intermediate output results are the same. Therefore, for artificial intelligence tasks that are not classified tasks, how to use the early exit mechanism to achieve efficient utilization of computing resources is a problem that needs to be solved.

[0005] In addition, artificial intelligence tasks that use the above-mentioned confidence-based early exit method and patience-based early exit method are usually run in the cloud, by setting an early exit point at each layer of the CNN model and calculating the intermediate output result of the layer at the early exit point to determine whether to exit at the current early exit point. This early exit framework may have unknown computing overhead and resource waste for specific tasks, but because the computing resources in the cloud are sufficient and the amount of tasks processed is large, computing resources can be saved overall by adopting this early exit framework. However, when considering performing reasoning tasks on edge devices, such as using edge devices with limited computing resources to perform reasoning tasks in IoT application scenarios, the above-mentioned early exit framework running on the cloud may no longer be applicable. Therefore, how to use the early exit mechanism to perform reasoning tasks in IoT devices is a problem that needs to be solved. Summary of the invention

[0006] The following summary is provided to introduce selected concepts in a simplified form that are further described in the detailed description that follows. This summary is not intended to highlight key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0007] According to one aspect of the present application, a method for processing a task is provided, comprising: a neural network (NN) model for processing the task receives input data, wherein the NN model includes a plurality of layers in sequence, one or more early-departure points are arranged between the plurality of layers, wherein the one or more early-departure points divide the NN model into two or more sub-blocks, each sub-block includes one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-departure points; the output sub-NN at the early-departure point of the one or more early-departure points generates early-departure output data related to the task based on a feature map output by the sub-block corresponding to the early-departure point, and the confidence estimation sub-NN at the early-departure point predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point; based on the predicted confidence, determine whether to use the early-departure output data as the output result of the task and not to perform processing of the sub-block after the early-departure point, or to provide the feature map output by the sub-block corresponding to the early-departure point to the next sub-block for processing.

[0008] According to one aspect of the present application, a system for processing a task is provided, comprising: one or more devices, wherein a neural network (NN) model for processing the task is arranged on the one or more devices, wherein the NN model comprises a plurality of layers in sequence, and one or more early-departure points are arranged between the plurality of layers, wherein the one or more early-departure points divide the NN model into two or more sub-blocks, each sub-block comprises one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-departure points; wherein the N The N model receives input data, and the output sub-NN at an early-departure point among the one or more early-departure points generates early-departure output data related to the task based on a feature map output by a sub-block corresponding to the early-departure point, and the confidence estimation sub-NN at the early-departure point predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point; wherein, based on the predicted confidence, it is determined whether to use the early-departure output data as the output result of the task and not to perform processing of the sub-block after the early-departure point, or to provide the feature map output by the sub-block corresponding to the early-departure point to the next sub-block for processing.

[0009] According to one aspect of the present application, a method for training a neural network (NN) model for processing a task is provided, comprising: training a backbone NN based on training data, the backbone NN comprising a plurality of layers in sequence; setting one or more early-departure points between the plurality of layers of the trained backbone NN, wherein the one or more early-departure points divide the backbone NN into two or more sub-blocks, each sub-block comprising one or more layers of the plurality of layers of the backbone NN; setting an output sub-NN and a confidence estimation sub-NN at each of the one or more early-departure points, wherein the output sub-NN generates early-departure output data related to the task based on a feature map output by the sub-block corresponding to the early-departure point, and the confidence estimation sub-NN predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point; for the input data of the backbone NN, training the output sub-NN and the confidence estimation sub-NN at the early-departure point based on the output data of the backbone NN and the early-departure output data and confidence output by the output sub-NN and the confidence estimation sub-NN at each of the one or more early-departure points.

[0010] According to one aspect of the present application, a machine-readable storage medium is provided, which stores executable instructions. When the instructions are executed, one or more processors execute the methods described in various embodiments of the present disclosure.

[0011] According to one aspect of the present application, a computer program product is provided, which includes computer executable instructions. When the instructions are executed, one or more processors execute the methods described in various embodiments of the present disclosure.

[0012] By using the method for processing tasks based on the early exit mechanism according to the present application, the confidence of the corresponding intermediate output result is learned by setting the confidence estimation sub-NN at the early exit point of the neural network, thereby solving the confidence-based early exit problem of artificial intelligence tasks for non-classification tasks; and the method for processing tasks based on the early exit mechanism according to the present application can also be applied to classification tasks. In addition, by determining the exit point position in the backbone neural network based on the hardware capabilities and / or communication bandwidth of each device in the Internet of Things, the memory usage of the parameters of each layer in the backbone neural network, the size of the feature map output by each layer, and the amount of calculation of each layer, unnecessary computing overhead of edge devices in the Internet of Things is avoided and the hardware resource usage of edge devices is maximized. Other advantages of the technical solution of the present application will be described in the detailed description below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A further understanding of the nature and advantages of the present disclosure may be achieved by referring to the following drawings.In the accompanying drawings, similar components or features may have the same reference numeral.

[0014] Figure 1 A schematic diagram of a neural network model according to an embodiment is shown.

[0015] Figure 2 Shows Figure 1 Schematic diagram of a more granular convolutional layer of the neural network shown.

[0016] Figure 3 A schematic diagram of a feature graph according to an embodiment is shown.

[0017] Figure 4 A schematic diagram of a neural network model with early-falling branches according to one embodiment is shown.

[0018] Figure 5 A schematic diagram of an Internet of Things according to an embodiment is shown.

[0019] Figure 6 A schematic diagram showing parameters related to a neural network model according to one embodiment is shown.

[0020] Figure 7 A flow chart of a method for processing a task according to one embodiment is shown.

[0021] Figure 8 A flowchart of a method for training a neural network model according to one embodiment is shown.

[0022] Fig. 9 A schematic diagram of a system for processing tasks according to one embodiment is shown. DETAILED DESCRIPTION

[0023] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The function and arrangement of the elements discussed can be changed without departing from the scope of protection of the present disclosure. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0024] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0025] Figure 1 A schematic diagram of a neural network (NN) model according to one embodiment is shown.

[0026] exist Figure 1 In the example shown, the NN model 100 includes an input layer Li, multiple hidden layers L1 to LN, and a final output encoder Enf. The input layer Li receives input data I and outputs a feature map F0. The input data is, for example, an image, sound, text, sensor signal data detected by a sensor, etc. The input data can be data of any dimension, such as a one-dimensional array, a two-dimensional array, a three-dimensional array, a four-dimensional array, etc. For example, a one-dimensional array can be a time sampling or a spectrum sampling, and a two-dimensional array may contain a one-dimensional array of two channels, or a two-dimensional array of one channel, and so on. For example, the NN model 100 can be a one-dimensional convolutional neural network (CNN), whose input layer may receive a one-dimensional or two-dimensional array, the NN model 100 can be a two-dimensional CNN, whose input layer may receive a two-dimensional or three-dimensional array, and the NN model 100 can be a three-dimensional CNN, whose input layer may receive a three-dimensional or four-dimensional array. Taking the application of CNN in the field of computer vision as an example, the input data of a two-dimensional CNN can be a three-dimensional array, that is, a two-dimensional pixel point and an RGB channel on a plane. It can be understood that the above array can be called a vector or a matrix or a tensor.

[0027] In one embodiment, the input layer Li can perform normalization processing on the input data I. Taking CNN as an example, before inputting the data into the convolution layer, the input layer Li normalizes the input data in the channel, time or frequency dimension. For example, if the input data is a pixel, the pixel values ​​distributed in the [0,255] interval can be normalized to the [0,1] interval. In another embodiment, the input layer Li can perform embedding processing on the input data I. For example, when the input data I is text, the input layer Li can convert the input text into a feature map F0 of the hidden space. It can be understood that in a specific application, the input layer Li is not necessarily required, and the input data I can also be directly input into the hidden layer L1, or the function of the input layer Li can be merged in the hidden layer L1.

[0028] Hidden layer L1 receives and processes feature map F0 and outputs feature map F1, hidden layer L2 receives and processes feature map F1 and outputs feature map F2, and so on, hidden layer LN receives and processes feature map FN-1 and outputs feature map FN, and the final encoder Enf receives and processes feature map FN and outputs the final output data Of. Still taking the CNN network as an example, hidden layer L1 can be a convolution layer, which generally includes a convolution kernel and an excitation function. The parameters of the convolution layer include the convolution kernel size, step size and padding, which together determine the size of the feature map output by the convolution layer. Hidden layer L2 can also be a convolution layer, which can have the same or different parameters as the convolution layer L1, and accordingly, the size of the feature map F2 output by the convolution layer L2 can be the same or different from the size of the feature map F1 output by the convolution layer L1. Hidden layer L3 can be a pooling layer. After the convolution layers L1 and L2 perform feature extraction, the output feature map F2 is passed to the pooling layer L3 for feature selection and information filtering. The pooling layer L3 contains a preset pooling function, whose function is to replace the result of a single point in the feature map with the feature map statistics of its adjacent area. The parameters of the pooling layer include the pooling area size, step size and padding, which together determine the size of the feature map output by the pooling layer. Hidden layers L4, L5, and L6 can be convolutional layers, L7 can be a pooling layer, and so on. The last four hidden layers LN-3 to LN can be three convolutional layers and one pooling layer, respectively, and the pooling layer LN outputs the feature map FN. It can be understood that in a specific implementation, the convolutional layers and pooling layers can be arranged in other ways than the above examples, for example, one or more convolutional layers are included between every two pooling layers.

[0029] Although the concept of layers in CNN100 is described at the granularity of a single convolutional layer and a single pooling layer in the above example, it can be understood that in the CNN network, the function of the convolutional layer is to extract features from the input data, and the function of the pooling layer is to replace the result of a single point in the feature map with the feature map statistics of its adjacent area. Therefore, the concept of layers in CNN can also be described at a larger granularity, for example, the combination of one or more convolutional layers and a corresponding pooling layer is considered to be a convolution block. Figure 1 In the above example, the convolutional layers L1, L2 and the pooling layer L3 can be used together as a convolutional block C1, the convolutional layers L4, L5, L6 and the pooling layer L7 can be used as a convolutional block C2, and so on. Therefore, in CNN, the convolutional blocks C1 to CM described here can also be used as convolutional layers of larger granularity.

[0030] Figure 2 Shows Figure 1 A schematic diagram of a more granular convolutional layer of neural network 100 is shown. Figure 1 and 2 The same reference numerals in the figures represent the same contents. Figure 2 As shown, the convolutional layer C1 includes Figure 1 The finer-grained convolutional layers L1, L2 and pooling layer L3 shown, whose output feature map F1' corresponds to Figure 1 As shown in the feature map F3, the convolution layer C2 includes Figure 1 The finer-grained convolutional layers L4, L5, L6 and pooling layer L7 shown, whose output feature map F2' corresponds to Figure 1 The feature map F7 shown in the figure, and so on, the convolution layer CM includes Figure 1 The finer-grained convolutional layers LN-3 to LN-1 and pooling layer LN are shown, whose output feature map FM' corresponds to Figure 1 It can be understood that the specific implementation method and granularity of the layers in the neural network model are not limited to Figure 1 and Figure 2 In the described examples, the technical solution of the present application can be applied to a neural network model with a sequential or progressive layer structure.

[0031] The output encoder Enf can be implemented by a fully connected network (FCN), for example, the output encoder Enf includes one or more fully connected layers and an output layer. For example, for a classification task, the output encoder outputs a classification label, and for an object recognition task, the output encoder outputs the center coordinates and size of the object. The output encoder Enf can be implemented in other ways, for example, the output encoder Enf can be implemented by a transformer model. It can be understood that various appropriate output encoders can be used in the technical solution of the present application.

[0032] The neural network model 100 can be used to process a specific task. For example, the task can be a computer vision task, for example, the input data I can be an image, the computer vision task can be an image classification task, and the output data Of can be a classification of the image; the computer vision task can be an object recognition task, and the output data Of can be indicative information of the recognized object, such as the area corresponding to the recognized object in the image. For another example, the task can be a natural language processing task, for example, the input data I can be a text sentence, and the output data Of can be the semantic classification of the sentence or the translation of the sentence into another language. For another example, the task can be any other task, such as a task for estimating the service life of a device, a task for determining whether a device is normal, and so on.

[0033] Figure 3 A schematic diagram of a feature graph according to an embodiment is shown.

[0034] exist Figure 3 In the example shown, the input data is a 224×224 RGB image, which includes a two-dimensional matrix of three channels. Therefore, the input data is three-dimensional data with a dimension of (224, 224, 3). Figure 1 and Figure 2 , the input data can be either Figure 1 and 2 The input data I shown can be Figure 1 and 2 Feature map F0 shown.

[0035] by Figure 2 Taking the representation of the neural network structure as an example, the input data passes through the convolution layer C1 to output a feature map F1' with a dimension of (112, 112, 64), and the feature map F1' passes through the convolution layer C2 to output a feature map F2' with a dimension of (56, 56, 64). Similarly, the feature map F4' with a dimension of (14, 14, 256) passes through the convolution layer C5 to output a feature map F5' with a dimension of (7, 7, 512). The output encoder Enf can further process the feature map F5' to obtain the output data. For example, for an image classification task, the output data can be the category of the image predicted by the neural network. For example, for an object recognition task, the output data can be the recognized object in the input image.

[0036] Figure 4 A schematic diagram of a neural network model with early-falling branches according to one embodiment is shown.

[0037] Figure 4 The example neural network structure 400 shown includes Figure 2 The example neural network structure shown can be Figure 2 The CNN shown can be called a backbone network, which includes multiple convolutional layers C1 to CM and an encoder Enf, which generates task-related output data based on the feature maps output by the convolutional layer CM.

[0038] Early exit points can be set at appropriate locations in the backbone network, such as Figure 4 As shown, an early exit point Ex1 is set at the output of the convolution layer C1, and an early exit point Ex2 is set at the output of the convolution layer C3. It can be understood that the number and position of the early exit points are not limited to Figure 4 In the example shown, more or fewer early exit points may be set. For example, an early exit point may be set at the output of each convolutional layer except the last convolutional layer CM.

[0039] An early exit encoder En1 and an early exit evaluator Ev1 are provided at the early exit point Ex1, and an early exit encoder En2 and an early exit evaluator Ev2 are provided at the early exit point Ex2. The structure including the early exit encoder En1 and the early exit evaluator Ev1 may be referred to as an early exit branch 1, and the structure including the early exit encoder En2 and the early exit evaluator Ev2 may be referred to as an early exit branch 2. It can be understood that the early exit point and the early exit branch are in one-to-one correspondence, and therefore the two terms may be used interchangeably in this article.

[0040] Similar to the final encoder Enf in the backbone network, the early-out encoder En1 receives the feature map F1' output by the convolution layer C1, and outputs the task-related early-out output data O1, and the early-out encoder En2 receives the feature map F3' output by the convolution layer C3, and outputs the task-related early-out output data O2. The early-out encoders En1 and En2 can adopt the same or similar NN structure as the final encoder Enf, and the early-out encoders En1 and En2 can be called output sub-NNs.

[0041] The early-departure evaluator Ev1 receives the feature map F1' output by the convolution layer C1, and outputs the confidence E1 of the early-departure output data about the early-departure point Ex1. The early-departure evaluator Ev2 receives the feature map F3' output by the convolution layer C3, and outputs the confidence E2 of the early-departure output data about the early-departure point Ex2. The early-departure evaluators Ev1 and Ev2 can be implemented using any appropriate NN structure, for example, they can be implemented using FCN, transformer, etc. The early-departure evaluators Ev1 and Ev2 can be called confidence estimation sub-NNs.

[0042] In one embodiment, the early-falling evaluator can output a binary classification value based on the input feature map. Figure 4, the early-leaving evaluator Ev1 can output a binary classification value based on the input feature map F1'. For example, when the output classification value is 1, it means that the early-leaving output result O1 of the corresponding early-leaving encoder En1 meets the confidence requirement, so that the early-leaving output data O1 is used as the task-related output result and the processing after the early-leaving point is no longer performed, for example, the feature map F1' is no longer transmitted to the subsequent convolutional layer C2; when the output classification value is 0, it means that the early-leaving output of the corresponding early-leaving encoder En1 does not meet the confidence requirement, so that the feature map F1' at the early-leaving point is provided to the subsequent convolutional layer C2 for further processing. Correspondingly, the convolutional layer C2 receives and processes the feature map F1' and outputs the feature map F2', and the convolutional layer C3 receives and processes the feature map F2' and outputs the feature map F3'. The early-leaving evaluator Ev2 can output a binary classification value based on the input feature map F3'. For example, when the output classification value is 1, it indicates that the early-departure output result O2 of the corresponding early-departure encoder En2 meets the confidence requirement, so that the early-departure output data O2 is used as the task-related output result and the processing after the early-departure point is no longer performed, for example, the feature map F3' is no longer transmitted to the subsequent convolution layer C4; when the output classification value is 0, it indicates that the early-departure output result O2 of the corresponding early-departure encoder En2 does not meet the confidence requirement, so that the feature map F3' at the early-departure point is provided to the subsequent convolution layer C4 for further processing.

[0043] In one embodiment, the early-departure evaluator may output a probability value based on the input feature map. The probability value may be a value in the [0,1] interval representing the confidence of the corresponding early-departure output result. When the probability value output by the early-departure evaluator is greater than a threshold value, it indicates that the early-departure output result of the corresponding early-departure encoder meets the confidence requirement, so that the early-departure output data is used as a task-related output result and the processing after the early-departure point is no longer performed; when the probability value output by the early-departure evaluator is less than a threshold value, it indicates that the early-departure output of the corresponding early-departure encoder does not meet the confidence requirement, so that the feature map at the early-departure point is provided to the subsequent convolutional layer for further processing.

[0044] In one embodiment, the early-departure evaluator can output a distance value based on the input feature map. For example, the distance value can be a predicted value of the Euclidean distance, cosine distance, dot product distance, etc. between the corresponding early-departure output data and the reference output data, and the distance value reflects the deviation of the corresponding early-departure output data relative to the reference output data. During the training process of the early-departure evaluator, the output data of the final encoder Enf can be used as the reference output data. When the distance value output by the early-departure evaluator is less than the threshold, it means that the early-departure output result of the corresponding early-departure encoder meets the confidence requirement, so that the early-departure output data is used as the task-related output result and the processing after the early-departure point is no longer performed; when the distance value output by the early-departure evaluator is greater than the threshold, it means that the early-departure output of the corresponding early-departure encoder does not meet the confidence requirement, so that the feature map at the early-departure point is provided to the subsequent convolution layer for further processing.

[0045] like Figure 4 As shown, when a task is processed using a neural network model with an early-outgoing branch, an early-outgoing evaluator is used sequentially at each early-outgoing point to evaluate whether the early-outgoing output data of the early-outgoing point meets the confidence requirement. If so, it is considered that the early-outgoing output data of the current early-outgoing point can be used as the output result of the task, so it is no longer necessary to use computing resources to perform calculations after the early-outgoing point, thereby saving computing resources while providing output results that meet the requirements. By using an additional early-outgoing evaluator at the early-outgoing point to evaluate whether the early-outgoing output data of the corresponding early-outgoing encoder meets the requirements, the judgment of whether to exit early at the current early-outgoing point no longer depends on, for example, the confidence value in the output result of the early-outgoing encoder of each layer in the classification task, so Figure 4 The neural network model shown can be effectively applied to tasks other than classification tasks as well as classification tasks. Tasks other than classification tasks can be called regression tasks, for example, Figure 4 Regression tasks to which the illustrated neural network model can be applied may include object detection tasks in images, among others.

[0046] According to one embodiment, during the training process of the NN model 400, the backbone network 100 may be trained first, and after the training of the backbone network 100 is completed, the parameters of the backbone network 100 may be fixed or frozen, and on this basis, the early-departure encoder and the early-departure evaluator in the early-departure branch may be further trained using a self-distillation method.

[0047] In the first stage of training the backbone network 100, a data set related to a specific task can be used to train the backbone network parameters. Any suitable method can be used to train the neural network 100 as the backbone network. The training of the neural network 100 is known in the art and will not be described again.

[0048] In the second stage, the early-departure encoder and the early-departure evaluator at the early-departure point are trained in a self-distillation manner using the trained backbone network 100. Taking the early-departure encoder En1 as an example, for an input data sample, the early-departure encoder En1 at the early-departure point and the final encoder Enf of the backbone network respectively obtain the early-departure output data O1 and the final output data Of, and the final output data Of can be used as the true value or label data to update the parameters of the early-departure encoder En1. For example, the current task is an object recognition task, the input data is a three-dimensional matrix representing an RGB image, and the output data is a three-dimensional matrix representing the object area recognized in the image. For an input image sample, the early-departure encoder En1 and the final encoder Enf respectively output the early-departure output data O1 and the final output data Of in the form of a three-dimensional matrix. In one embodiment, the parameters of the early-departure encoder En1 can be updated based on the distance between the early-departure output data O1 and the final output data Of, and the distance is, for example, the Euclidean distance, cosine distance, dot product distance, etc. between the two three-dimensional matrices. It can be understood that the technical solution of the present disclosure can adopt any distance that can represent the distance between the early-retirement output data O1 and the final output data Of, and is not limited to the distance exemplified above.

[0049] Taking the early-departure evaluator Ev1 as an example, for an input data sample, the early-departure encoder En1 at the early-departure point and the final encoder Enf of the backbone network NN obtain the early-departure output data O1 and the final output data Of respectively. The data representing the similarity between the two can be used as the true value or label data to update the parameters of the early-departure evaluator Ev1.

[0050] In one embodiment, for an input data sample, the early-out evaluator Ev1 predicts a distance value based on the input feature map F1', and the early-out encoder En1 at the early-out point and the final encoder Enf of the backbone network NN obtain the early-out output data O1 and the final output data Of, respectively. The distance value between the early-out output data O1 and the final output data Of can be calculated, and the calculated distance value is used as the true value or label data to update the parameters of the early-out evaluator Ev1.

[0051] In one embodiment, for an input data sample, the early exit evaluator Ev1 predicts a binary classification value based on the input feature map F1', and the early exit encoder En1 at the early exit point and the final encoder Enf of the backbone network NN obtain the early exit output data O1 and the final output data Of, respectively. The similarity between the early exit output data O1 and the final output data Of can be classified as 0 or 1 based on a statistical method or a clustering method, and the classified value is used as a true value or label data to update the parameters of the early exit evaluator Ev1. In one embodiment, for multiple input data samples, such as 100 data samples, the convolution layer C1 can generate 100 feature maps at the exit point Ex1, the early exit encoder En1 and the final encoder Enf can generate 100 groups of early exit output data O1 and final output data Of, and 100 distance values ​​can be generated based on the 100 groups of early exit output data O1 and the final output data Of. For example, the 100 distance values ​​can be classified into two categories using a clustering method, corresponding to 0 and 1, respectively. For another example, the probability distribution of the 100 distance values ​​is statistically calculated, the probability of each distance value is calculated accordingly, and each distance value is classified as 0 or 1 based on the comparison between the probability of each distance value and the threshold. It can be understood that the implementation of the present application can use any suitable clustering method and statistical method to complete the above classification process. After obtaining the above 100 classified label values, the training data for training the early-leaving evaluator Ev1 is obtained, so the early-leaving evaluator Ev1 can be trained accordingly.

[0052] In one embodiment, for an input data sample, the early-departure evaluator Ev1 predicts a probability value based on the input feature map F1', and the early-departure encoder En1 at the early-departure point and the final encoder Enf of the backbone network NN obtain the early-departure output data O1 and the final output data Of, respectively. The probability representing the confidence between the early-departure output data O1 and the final output data Of can be calculated based on a statistical method, and the calculated probability is used as the true value or label data to update the parameters of the early-departure evaluator Ev1. For example, the above-mentioned statistical method can be used to calculate the probability distribution of the above-mentioned 100 distance values, and the probability of each distance value is calculated accordingly as a representation of the confidence.

[0053] The training of the early-leaving evaluator is described above by taking the early-leaving evaluator Ev1 as an example. The early-leaving evaluator Ev2 and possible other early-leaving evaluators can be trained through a similar process. In one embodiment, the early-leaving encoder is first trained, and after the training of the early-leaving encoder is completed, the parameters of the early-leaving encoder are fixed or frozen, and the early-leaving evaluator is further trained on this basis.

[0054] Figure 5 A schematic diagram of an Internet of Things according to an embodiment is shown.

[0055] The Internet of Things 500 may include devices at multiple different levels, such as Figure 5 The first level device 510, the second level device 520, and the third level device 530 are shown. In one embodiment, the first level device 510 may be a sensor level device, the second level device 520 may be a gateway level device connected to one or more sensor level devices, and the third level device 530 may be a server level device connected to one or more gateway level devices. The sensor level device 510 usually has limited hardware capabilities, which are usually reflected in the computing power and memory size of the processor. The gateway level device 520 usually has a larger but also limited hardware capability than the sensor level device 510, and the server 530 usually has sufficient hardware capabilities. The first level device 510 and the second level device 520 can be connected through a first communication method with a first bandwidth, such as Bluetooth, ZigBee and other short-range wireless communication technologies, and the second level device 520 and the third level device 530 can be connected through a second communication method with a second bandwidth, such as through the Internet, Ethernet and other long-range communication technologies.

[0056] In an example of an IoT application, the sensor-level device 510 is a surveillance camera device, the gateway-level device 520 is a router, which can also be provided by a tablet or a mobile phone, and the server-level device 530 is a cloud server. Figure 4 The CNN model 400 with the early-falling branch shown in the figure is used to perform an object recognition task based on the image collected by the monitoring camera device 510. In one embodiment, the CNN model 400 can be arranged on the server device 530, and the computing resources of the server device 530 are used to run the CNN model 400 to perform the object recognition task. In another embodiment, the CNN model 400 is divided into a plurality of sub-blocks based on the early-falling point, each sub-block includes a corresponding convolution layer and an early-falling branch, and the divided sub-blocks are arranged on the devices 510 to 530 respectively. For example, see Figure 4 The early exit points Ex1 and Ex2 divide the CNN model 400 into three sub-blocks. The layers before the early exit point Ex1 in the backbone network and the corresponding early exit branches can be arranged on the monitoring camera device 510, the layers between the early exit points Ex1 and Ex2 in the backbone network and the corresponding early exit branches can be arranged on the router device 520, and the layers after the early exit point Ex2 in the backbone network can be arranged on the server 530.

[0057] Considering the differences in hardware capabilities and communication bandwidths of the various devices 510 to 530 in the IoT application, in one embodiment, a suitable early exit point is selected in the NN model based on the hardware capabilities and communication bandwidths of the devices 510 to 530. This method of selecting the early exit point will help to maximize the use of the capabilities of the sensor device 510 and the router device 520. Under the premise of ensuring the execution performance of the task, the sensor device 510 and the router device 520 will output the results of the object detection task as much as possible, which not only improves the speed of outputting the task results, but also saves as much processing resources of the server 530 as possible.

[0058] Figure 6 A schematic diagram showing parameters related to a neural network model according to one embodiment is shown.

[0059] Combination Figure 3 and 5 Description of the embodiment Figure 6 Parameters shown. Figure 6 The reference numeral 601 in the figure indicates the serial number of the convolutional layer. The reference numeral 602 indicates the dimension of the feature map output by each convolutional layer. The reference numeral 603 indicates the size or amount of data of the feature map output when the data type is a 32-bit floating point number. For example, the size of the feature map output by the convolutional layer C1 is 3.2 megabytes (mb), and the size of the feature map output by the convolutional layer C2 is 803 kilobytes (kb). The reference numeral 604 indicates the number of floating point operations (FLOP), and its unit is 10 9 Reference numeral 605 indicates the number of parameters of the model, the unit of which is kilo (k), and reference numeral 606 indicates the memory size required to store the parameters of the model.

[0060] exist Figure 3 and Figure 6 In the example shown, as the number of layers in the CNN increases, the number of corresponding model parameters also increases, which means that the device used to run the CNN model requires a larger memory to store these parameters. For example, the memory size of the sensor-level device 510 is different in different applications, and similarly, the memory size of the gateway-level device 520 is different in different applications. Different numbers of layers can be arranged on the devices 510 and 520 based on the memory size of the corresponding devices.

[0061] On the other hand, in the multiple layers C1 to C5 in sequence, as the number of layers increases, the size of the feature map output by the later layers decreases significantly. For example, the data size of the feature map output by layer C1 is 3.2 mb, and the data size of the feature map output by the subsequent layer C2 is 803 kb. This means that when two layers, such as C2 and C3, are arranged on different devices, the amount of communication data between the devices is smaller than when two layers, such as C1 and C2, are arranged on different devices, so that a lower communication bandwidth can be used. For example, when a sensor-level device 510 and a gateway-level device 520 communicate using a short-range, low-rate ZigBee method, based on the smaller communication bandwidth, arranging layers C1 and C2 on the sensor-level device 510 is a better choice than arranging only layer C1 on the sensor-level device 510.

[0062] On the other hand, the amount of floating-point operations may increase significantly on some intermediate layers. For example, the amount of floating-point operations of layer C4 increases significantly, and a more powerful processor unit is required on the device to meet this computing demand. For example, the processor capabilities of the gateway-level device 520 are also different in different applications. If the processor capabilities of the gateway device are strong, and / or the computing resources that can be allocated to the corresponding sensor device can meet the computing capability requirements of layer C4, for example, layer C4 can be arranged on the gateway-level device 520. Otherwise, layer C4 and subsequent layers can be arranged on the server-level device 530.

[0063] Considering the above model parameters related to the computing power, storage capacity and communication bandwidth of the device and the actual hardware capacity and communication bandwidth of the device, the early exit point can be set at an appropriate position. Figure 3 and 6 In the example shown, the first early exit point E1 can be set after the convolution layer C2, and the second early exit point can be set after the convolution layer C4, so that the multiple layers C1 to C5 of the CNN are divided into three sub-blocks, and the sub-block including the convolution layers C1 and C2 and the early exit point Ex1 are arranged on the first level of equipment, the sub-block including the convolution layers C3 and C4 and the early exit point Ex2 are arranged on the second level of equipment, and the sub-block including the convolution layer C5 is set on the third level of equipment.

[0064] exist Figure 5 In the IoT application shown, the devices 510 and 520 before the server-level device 530 can be collectively referred to as edge devices. When deploying the various sub-blocks of the neural network to the edge devices at various levels, only one exit point is set at the output of the last layer of the corresponding sub-block on each device. In this way, the hardware resources of each edge device can be fully utilized, avoiding the additional computing overhead caused by setting more than one exit point in a sub-block on a certain edge device.

[0065] In an Internet of Things application according to one embodiment, see Figure 5 , the sensor-level device 510 may be an electric tool or an electronic device deployed on the electric tool, and the electronic device includes a sensor, a processor, a memory, a communication unit, etc. integrated on the electric tool. The sensor is, for example, an inertial measurement unit (IMU), and the IMU includes, for example, an inclinometer and an accelerometer. The processor is, for example, a microcontroller unit (MCU), which executes programs stored in the memory, processes data cached in the memory, and controls the communication unit to communicate with other devices. The communication device is, for example, a short-range communication device such as a Bluetooth communication device and a ZigBee communication device. The gateway-level device 520 may be a small router device, or the function of the gateway-level device 520 may be provided by a mobile phone, a portable computer, etc. The gateway-level device 520 may include a processor, a memory, and a communication unit. The processor is, for example, a central processing unit (CPU), a graphics processor (GPU), or other processors with higher capabilities than an MCU, and the memory of the device 520 is usually larger than the content of the device 510. The server-level device 530 may be an industrial computer, or a cloud server or a server cluster.

[0066] In one example of the IoT application, the power tool 510 may be a tightening gun. For example, multiple power tools 510 at a construction site may be connected to a gateway-level device 520 arranged at the site via short-range wireless communication, and the device 520 is then connected to a server-level device 530 via long-range wireless communication. In one task, a neural network model may be used to predict the remaining service life of the power tool based on a vibration signal monitored by a sensor of the power tool 510. For example, the shortening of the service life caused by wear of the power tool due to long-term use or improper use may be predicted. In another task, a neural network model may be used to predict whether the power tool is normal or abnormal based on a vibration signal monitored by a sensor of the power tool 510. According to the method of an embodiment of the present disclosure, the neural network model used to perform the task is divided into three sub-blocks based on the hardware capabilities and / or communication bandwidth of devices 510 and 520, and the three sub-blocks are arranged on devices 510 to 530, respectively.

[0067] In an Internet of Things application according to one embodiment, see Figure 5, the sensor-level device 510 may be a household appliance or may be an electronic device deployed on a household appliance, and the electronic device includes a sensor, a processor, a memory, a communication unit, etc. integrated on the household appliance. The sensor is, for example, an inertial measurement unit, a camera, a microphone, etc. The processor is, for example, an MCU, which executes a program stored in the memory, processes data cached in the memory, and controls the communication unit to communicate with other devices. The communication device is, for example, a short-range communication device such as a Bluetooth communication device and a ZigBee communication device. The gateway-level device 520 may be a small router device, or the function of the gateway-level device 520 may be provided by a mobile phone, a portable computer, etc. The gateway-level device 520 may include a processor, a memory, and a communication unit. The processor is, for example, a CPU, a GPU, etc., which has higher capabilities than an MCU, and the memory of the device 520 is usually larger than the content of the device 510. The server-level device 530 may be a cloud server or a server cluster.

[0068] In one example of the IoT application, the household appliance 510 may be a washing machine, which is connected to a gateway-level device 520 via short-range wireless communication, and the device 520 is connected to a server-level device 530 via long-range wireless communication. In one task, a neural network model may be used to predict the remaining service life of the washing machine based on a vibration signal detected by a sensor of the household appliance 510. In another task, a neural network model may be used to predict whether the household appliance is normal or abnormal based on a vibration signal detected by a sensor of the household appliance 510. According to the method of an embodiment of the present disclosure, the neural network model used to perform the task is divided into three sub-blocks based on the hardware capabilities and / or communication bandwidth of the devices 510 and 520, and the three sub-blocks are arranged on devices 510 to 530, respectively.

[0069] In an Internet of Things application according to one embodiment, see Figure 5, the sensor-level device 510 may be a camera module of a video surveillance device, and the camera module includes a camera, a processor, a memory, a communication unit, etc. The communication device is, for example, a short-distance communication device such as a Bluetooth communication device and a ZigBee communication device. The gateway-level device 520 may be a small router device, or the function of the gateway-level device 520 may be provided by a mobile phone, a portable computer, etc. The gateway-level device 520 may include a processor, a memory, and a communication unit. The server-level device 530 may be a cloud server or a server cluster. In one task, a neural network model may be used to perform object recognition based on the image captured by the sensor of the camera module 510. In another task, a neural network model may be used to perform graphic classification based on the image captured by the sensor of the camera module 510. According to the method of the embodiment of the present disclosure, the neural network model used to perform the task is divided into three sub-blocks based on the hardware capabilities and / or communication bandwidth of the devices 510 and 520, and the three sub-blocks are arranged on the devices 510 to 530 respectively.

[0070] The above examples illustrate the implementation of the embodiments of the present application in the Internet of Things application. It can be understood that the NN model structure with early-exit branches of the present application is not limited to applications in the Internet of Things. For example, the NN model structure with early-exit branches of the present application can be applied in an embodiment that only uses cloud or server resources to run the neural network model, so that the early-exit problem of the regression task can be better handled. For example, although three levels of devices are usually included in Internet of Things applications, fewer or more levels of devices may be included in actual applications, and the neural network model can be divided into fewer or more sub-blocks accordingly, not limited to three sub-blocks.

[0071] Figure 7 A flow chart of a method for processing a task according to one embodiment is shown.

[0072] In step 710, a neural network (NN) model for processing the task receives input data, wherein the NN model includes a plurality of sequential layers, one or more early-backward points are arranged between the plurality of layers, wherein the one or more early-backward points divide the NN model into two or more sub-blocks, each sub-block includes one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-backward points.

[0073] In step 720, the output sub-NN at the early-departure point among the one or more early-departure points generates early-departure output data related to the task based on the feature map output by the sub-block corresponding to the early-departure point, and the confidence estimation sub-NN at the early-departure point predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point. According to one embodiment, the confidence includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimation sub-NN based on the feature map.

[0074] In step 730, based on the confidence of the prediction, it is determined whether to use the early-leaving output data as the output result of the task and not to perform processing of the sub-block after the early-leaving point, or to provide the feature map output of the sub-block corresponding to the early-leaving point to the next sub-block for processing. According to one embodiment, when the confidence of the prediction meets the confidence requirement, the early-leaving output data is used as the output result of the task and the processing of the sub-block after the early-leaving point is not performed, and when the confidence of the prediction does not meet the confidence requirement, the feature map output of the sub-block corresponding to the early-leaving point is provided to the next sub-block for processing.

[0075] According to one embodiment, the task includes one of a classification task and a regression task. According to one embodiment, the task includes a classification task. According to one embodiment, the input data includes an image, and the output result includes the category of the image, such as the category of an object in the image. According to one embodiment, the input data includes a sensing signal detected by a sensing device of the device, and the output result includes the abnormality or normality of at least a part of the device. Those skilled in the art will understand that the image itself is also a sensing signal, and the sensing signal may include various signals detected by the sensing device, such as images, videos, sounds, vibrations, speeds, angles, positions, etc.

[0076] According to one embodiment, the task includes a regression task. According to one embodiment, the input data includes an image, the regression task includes an object detection task, and the output result includes an indication of the detected object in the image, and specifically may include an indication of the location of the boundary of the detected object in the image. According to one embodiment, the input data includes a sensing signal detected by a sensing device of the device, the regression task includes a remaining service life estimation task, and the output result includes the remaining service life of at least a part of the device. As described above, the sensing signal may include various signals detected by the sensing device, such as images, videos, sounds, vibrations, speeds, angles, positions, etc.

[0077] According to one embodiment, the one or more early retirement points include a first early retirement point and a second early retirement point, the first early retirement point and the second early retirement point divide the multiple layers of the NN model into a first sub-block, a second sub-block and a third sub-block, the first sub-block and the first early retirement point, the second sub-block and the second early retirement point, and the third sub-block are respectively arranged on a first device, a second device, and a third device.

[0078] According to one embodiment, the number of layers included in the first sub-block is determined based on the hardware capability of the first device and / or the communication bandwidth between the first device and the second device, and the number of layers included in the second sub-block is determined based on the hardware capability of the second device and / or the communication bandwidth between the second device and the third device. According to one embodiment, the hardware capability includes the computing power and memory size of the processor. According to one embodiment, the first device communicates with the second device via a first communication protocol, and the second device communicates with the third device via a second communication protocol.

[0079] According to one embodiment, the first device includes a sensor level device, the second device includes a gateway level device connected to one or more sensor level devices, and the third device includes a server level device connected to one or more gateway level devices. According to one embodiment, three levels of devices constitute the Internet of Things.

[0080] Figure 8 A flowchart of a method for training a neural network (NN) model for processing a task according to one embodiment is shown.

[0081] At step 810, a backbone NN is trained based on training data, the backbone NN comprising a plurality of sequential layers;

[0082] In step 820, one or more early-falling points are set between the multiple layers of the trained backbone NN, wherein the one or more early-falling points divide the backbone NN into two or more sub-blocks, each sub-block including one or more layers of the multiple layers of the backbone NN.

[0083] In step 830, an output sub-NN and a confidence estimation sub-NN are set at each of the one or more early-exit points, wherein the output sub-NN generates early-exit output data related to the task based on the feature map output by the sub-block corresponding to the early-exit point, and the confidence estimation sub-NN predicts the confidence of the early-exit output data of the early-exit point based on the feature map output by the sub-block corresponding to the early-exit point. According to one embodiment, the confidence value includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimation sub-NN based on the feature map.

[0084] In step 840, with respect to the input data of the backbone NN, the output sub-NN and the confidence estimation sub-NN at the early-departure point are trained based on the output data of the backbone NN and the early-departure output data and confidence output by the output sub-NN and the confidence estimation sub-NN at each of the one or more early-departure points.

[0085] According to one embodiment, in step 840, a distance value between the output data of the backbone NN and the output data of the output sub-NN at the early-departure point is determined based on the output data of the backbone NN and the output data of the output sub-NN at the early-departure point, and the parameters of the output sub-NN at the early-departure point are updated based on the determined distance value between the two; a confidence value of the output data of the output sub-NN at the early-departure point is determined based on the determined distance value between the two, and the parameters of the confidence estimation sub-NN at the early-departure point are updated based on the difference between the determined confidence value and the confidence value output by the confidence estimation sub-NN at the early-departure point.

[0086] According to one embodiment, the task includes a classification task. According to one embodiment, the input data includes an image, and the output data includes a category of the image. According to one embodiment, the input data includes a sensing signal detected by a sensing device of the device, and the output result includes an abnormality or normality of at least a part of the device. The abnormality includes a malfunction or abnormal use of a device component.

[0087] According to one embodiment, the task comprises a regression task. According to one embodiment, the input data comprises an image, the regression task comprises an object detection task, and the output data comprises the detected object in the image. According to one embodiment, the input data comprises a sensing signal detected by a sensing device of the device, the regression task comprises a remaining useful life estimation task, and the output data comprises the remaining useful life of at least a part of the device.

[0088] According to one embodiment, the one or more early withdrawal points include a first early withdrawal point and a second early withdrawal point, the first early withdrawal point and the second early withdrawal point divide the multiple layers of the backbone NN into a first sub-block, a second sub-block and a third sub-block, the first sub-block and the first early withdrawal point, the second sub-block and the second early withdrawal point, and the third sub-block are respectively arranged on a first device, a second device, and a third device.

[0089] According to one embodiment, in step 820, the number of layers included in the first sub-block is determined based on the hardware capability of the first device and / or the communication bandwidth between the first device and the second device, and the first early exit point is set at the output of the last layer of the first sub-block; the number of layers included in the second sub-block is determined based on the hardware capability of the second device and / or the communication bandwidth between the second device and the third device, and the second early exit point is set at the output of the last layer of the second sub-block.

[0090] According to one embodiment, the first device comprises a sensor level device, the second device comprises a gateway level device connected to one or more sensor level devices, and the third device comprises a server level device connected to one or more gateway level devices.

[0091] Fig. 9 A schematic diagram of a system for processing tasks according to one embodiment is shown.

[0092] The system 900 may include one or more devices. For example, the system 900 may include one device 910 or may include multiple devices 910 to 930. It is understood that the system 900 may include any suitable number of devices, and is not limited to Fig. 9 One device or three devices as shown. A neural network (NN) model for processing the task is arranged on the one or more devices, wherein the NN model includes a plurality of layers in sequence, one or more early-departure points are arranged between the plurality of layers, wherein the one or more early-departure points divide the NN model into two or more sub-blocks, each sub-block includes one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-departure points. The NN model receives input data, and the output sub-NN at the early-departure point of the one or more early-departure points generates early-departure output data related to the task based on the feature map output by the sub-block corresponding to the early-departure point, and the confidence estimation sub-NN at the early-departure point predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point, wherein, based on the predicted confidence, it is determined whether to use the early-departure output data as the output result of the task and not to perform the processing of the sub-block after the early-departure point, or to provide the feature map output by the sub-block corresponding to the early-departure point to the next sub-block for processing.

[0093] According to one embodiment, the confidence includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimator NN based on the feature map.

[0094] According to one embodiment, the task comprises a classification task. According to one embodiment, the input data comprises an image, and the output result comprises a category of the image. According to one embodiment, the input data comprises a sensing signal detected by a sensing device of a device, and the output result comprises whether at least a part of the device is abnormal or normal.

[0095] According to one embodiment, the task comprises a regression task. According to one embodiment, the input data comprises an image, the regression task comprises an object detection task, and the output comprises the detected object in the image. According to one embodiment, the input data comprises a sensing signal detected by a sensing device of the device, the regression task comprises a remaining useful life estimation task, and the output comprises the remaining useful life of at least a portion of the device.

[0096] According to one embodiment, the one or more devices include a first device, a second device, and a third device, wherein the one or more early regression points include a first early regression point and a second early regression point, the first early regression point and the second early regression point divide the multiple layers of the NN model into a first sub-block, a second sub-block, and a third sub-block, and the first sub-block and the first early regression point, the second sub-block and the second early regression point, and the third sub-block are respectively arranged on the first device, the second device, and the third device.

[0097] According to one embodiment, the number of layers included in the first sub-block is determined based on the hardware capabilities of the first device and / or the communication bandwidth between the first device and the second device, and the number of layers included in the second sub-block is determined based on the hardware capabilities of the second device and / or the communication bandwidth between the second device and the third device.

[0098] According to one embodiment, the first device comprises a sensor level device, the second device comprises a gateway level device connected to one or more sensor level devices, and the third device comprises a server level device connected to one or more gateway level devices.

[0099] In one embodiment, Fig. 9 The processing system shown may include a device 910, for example, the device 910 is a server. The device 910 may include one or more processing units and one or more memories, and the processing unit executes one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., the memory). When the instructions are executed by the processing unit, the above combined embodiments of the present application may be executed. Figure 1-8 Describes the various operations and functions.

[0100] In one embodiment, Fig. 9 The processing system shown may include a plurality of devices 910 to 930, for example, the devices 910 to 930 are Figure 5 The IoT device described. The devices 910 to 930 respectively include one or more processing units and one or more memories, and the processing units execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., the memory). When the instructions are executed by the processing units, the above combined embodiments of the present application can be executed. Figure 1-8Describes the various operations and functions.

[0101] According to one embodiment, a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may include a computer program or instruction stored thereon, which, when executed by a processing unit, may execute the above-mentioned combination of various embodiments of the present application. Figure 1-9 Describes the various operations and functions.

[0102] According to one embodiment, a computer program product is provided. The computer program product may include a computer program, which, when executed by a processing unit, may perform the above combined Figure 1-9 Describes the various operations and functions.

[0103] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "example" used throughout this specification may mean "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0104] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for processing a task, comprising: A neural network (NN) model for processing the task receives input data, wherein the NN model includes a plurality of layers in sequence, one or more early-outgoing points are arranged between the plurality of layers, wherein the one or more early-outgoing points divide the NN model into two or more sub-blocks, each sub-block includes one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-outgoing points; The output sub-NN at an early-leaving point among the one or more early-leaving points generates early-leaving output data related to the task based on a feature map output by a sub-block corresponding to the early-leaving point, and the confidence estimation sub-NN at the early-leaving point predicts the confidence of the early-leaving output data of the early-leaving point based on the feature map output by the sub-block corresponding to the early-leaving point; Based on the confidence of the prediction, it is determined whether to use the early-leaving output data as the output result of the task and not to perform processing of the sub-block after the early-leaving point, or to provide the feature map output by the sub-block corresponding to the early-leaving point to the next sub-block for processing.

2. The method of claim 1, wherein: The confidence includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimator NN based on the feature map.

3. The method of claim 1, wherein: The task includes one of a classification task and a regression task.

4. The method of claim 3, wherein: The tasks include classification tasks, where The input data includes an image, and the output result includes a category of the image; or The input data includes a sensing signal detected by a sensing device of the equipment, and the output result includes abnormality or normality of at least a part of the equipment.

5. The method of claim 3, wherein: The task includes a regression task, where The input data comprises an image, the regression task comprises an object detection task, and the output comprises an indication of a detected object in the image; or The input data includes a sensing signal detected by a sensing device of the equipment, the regression task includes a remaining service life estimation task, and the output result includes the remaining service life of at least a part of the equipment.

6. The method of claim 1, wherein: The one or more early withdrawal points include a first early withdrawal point and a second early withdrawal point, wherein the first early withdrawal point and the second early withdrawal point divide the multiple layers of the NN model into a first sub-block, a second sub-block and a third sub-block, and the first sub-block and the first early withdrawal point, the second sub-block and the second early withdrawal point, and the third sub-block are respectively arranged on a first device, a second device, and a third device.

7. The method of claim 6, wherein: The number of layers contained in the first sub-block is determined based on the hardware capability of the first device and / or the communication bandwidth between the first device and the second device, and the number of layers contained in the second sub-block is determined based on the hardware capability of the second device and / or the communication bandwidth between the second device and a third device.

8. The method according to claim 6 or 7, wherein: The first device comprises a sensor-level device, the second device comprises a gateway-level device connected to one or more sensor-level devices, and the third device comprises a server-level device connected to one or more gateway-level devices.

9. A system for processing a task, comprising: One or more devices, wherein a neural network (NN) model for processing the task is arranged on the one or more devices, wherein the NN model includes a plurality of sequential layers, and one or more early-leaving points are arranged between the plurality of layers, wherein the one or more early-leaving points divide the NN model into two or more sub-blocks, each sub-block includes one or more layers of the plurality of layers, wherein an output sub-NN and a confidence estimation sub-NN are arranged at each of the one or more early-leaving points; The NN model receives input data, the output sub-NN at an early-departure point among the one or more early-departure points generates early-departure output data related to the task based on a feature map output by a sub-block corresponding to the early-departure point, and the confidence estimation sub-NN at the early-departure point predicts the confidence of the early-departure output data of the early-departure point based on the feature map output by the sub-block corresponding to the early-departure point; Among them, based on the confidence of the prediction, it is determined to use the early-departure output data as the output result of the task and not to perform processing of the sub-block after the early-departure point, or to provide the feature map output by the sub-block corresponding to the early-departure point to the next sub-block for processing.

10. The system of claim 9, wherein: The confidence includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimator NN based on the feature map.

11. The system of claim 10, wherein: The task includes one of a classification task and a regression task.

12. The system of claim 11, wherein: The tasks include classification tasks, where The input data includes an image, and the output result includes a category of the image; or The input data includes a sensing signal detected by a sensing device of the equipment, and the output result includes abnormality or normality of at least a part of the equipment.

13. The system of claim 11, wherein: The task includes a regression task, where The input data includes an image, the regression task includes an object detection task, and the output result includes a detected object in the image; or The input data includes a sensing signal detected by a sensing device of the equipment, the regression task includes a remaining service life estimation task, and the output result includes the remaining service life of at least a part of the equipment.

14. The system of claim 9, wherein: The one or more devices include a first device, a second device, and a third device, wherein the one or more early regression points include a first early regression point and a second early regression point, the first early regression point and the second early regression point divide the multiple layers of the NN model into a first sub-block, a second sub-block, and a third sub-block, and the first sub-block and the first early regression point, the second sub-block and the second early regression point, and the third sub-block are respectively arranged on the first device, the second device, and the third device.

15. The system of claim 14, wherein: The number of layers contained in the first sub-block is determined based on the hardware capability of the first device and / or the communication bandwidth between the first device and the second device, and the number of layers contained in the second sub-block is determined based on the hardware capability of the second device and / or the communication bandwidth between the second device and a third device.

16. The system of claim 14 or 15, wherein: The first device comprises a sensor-level device, the second device comprises a gateway-level device connected to one or more sensor-level devices, and the third device comprises a server-level device connected to one or more gateway-level devices.

17. A method for training a neural network (NN) model for processing a task, comprising: Training a backbone NN based on the training data, wherein the backbone NN includes a plurality of sequential layers; Setting one or more early-falling points between the multiple layers of the trained backbone NN, wherein the one or more early-falling points divide the backbone NN into two or more sub-blocks, each sub-block including one or more layers of the multiple layers of the backbone NN; An output sub-NN and a confidence estimation sub-NN are set at each of the one or more early-exit points, wherein the output sub-NN generates early-exit output data related to the task based on a feature map output by a sub-block corresponding to the early-exit point, and the confidence estimation sub-NN predicts the confidence of the early-exit output data of the early-exit point based on the feature map output by the sub-block corresponding to the early-exit point; For the input data of the backbone NN, the output sub-NN and the confidence estimation sub-NN at each early-departure point are trained based on the output data of the backbone NN and the early-departure output data and confidence respectively output by the output sub-NN and the confidence estimation sub-NN at each early-departure point among the one or more early-departure points.

18. The method of claim 17, wherein: The confidence includes one of a distance value, a probability value, and a binary classification value predicted by the confidence estimator NN based on the feature map.

19. The method of claim 18, wherein: The step of training the output sub-NN and the confidence estimation sub-NN at each of the one or more early-departure points based on the output data of the backbone NN and the early-departure output data and confidence output by the output sub-NN and the confidence estimation sub-NN at the early-departure point comprises: Determine a distance value between the output data of the backbone NN and the output data of the output sub-NN at the early-degradation point based on the output data of the backbone NN and the output data of the output sub-NN at the early-degradation point, and train the output sub-NN at the early-degradation point based on the determined distance value between the two; The confidence of the output data of the output sub-NN at the early-degradation point is determined based on the determined distance value between the two, and the confidence estimation sub-NN at the early-degradation point is trained based on the difference between the determined confidence and the confidence output of the confidence estimation sub-NN at the early-degradation point.

20. The method of claim 17, wherein: The one or more early withdrawal points include a first early withdrawal point and a second early withdrawal point, wherein the first early withdrawal point and the second early withdrawal point divide the multiple layers of the backbone NN into a first sub-block, a second sub-block and a third sub-block, and the first sub-block and the first early withdrawal point, the second sub-block and the second early withdrawal point, and the third sub-block are respectively arranged on a first device, a second device, and a third device.

21. The method of claim 20, wherein: The step of setting one or more early regression points between the multiple layers of the trained backbone NN comprises: Determine the number of layers included in the first sub-block based on the hardware capability of the first device and / or the communication bandwidth between the first device and the second device, and set the first early exit point at the output of the last layer of the first sub-block; The number of layers included in the second sub-block is determined based on the hardware capability of the second device and / or the communication bandwidth between the second device and a third device, and the second early exit point is set at the output of the last layer of the second sub-block.

22. The method of claim 21, wherein: The first device comprises a sensor-level device, the second device comprises a gateway-level device connected to one or more sensor-level devices, and the third device comprises a server-level device connected to one or more gateway-level devices.

23. A processing system comprising: One or more devices, the one or more devices comprising one or more processors and one or more memories, the memories storing computer executable instructions, which when executed cause the one or more processors to perform the method as claimed in any one of claims 1 to 8 and 17 to 22.

24. A machine-readable storage medium storing executable instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 8 and 17 to 22.

Images