Apparatus and method for local and remote joint reasoning

By designing a learning control function in DDNN, using the semantic features of data samples to determine the inference position, it solves the problem of training multi-export DNN and decision-making, and achieves more efficient and accurate distributed inference, which is suitable for a variety of ML tasks.

CN118922839APending Publication Date: 2024-11-08HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093408.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-24
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Prior Art In distributed deep neural networks (DDNNs), it is difficult to effectively train DNNs with multiple exits and make offload decisions, especially in non-classified tasks. The limitations of the confidence method lead to accuracy and capability. Consumption problem.

Method used

Design a learning control function, use the inherent semantic features of data samples, decide whether to perform inference tasks locally or remotely at the device, and form prediction outputs through the first model or remote second model stored in the device, avoiding dependence on local prediction confidence , optimize the accuracy and efficiency of reasoning.

Benefits of technology

It reduces energy consumption, improves the accuracy and speed of inference, and is suitable for various ML tasks, whether it is image classification or regression type prediction, achieving more flexible and efficient distributed inference decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922839A_ABST
    Figure CN118922839A_ABST
Patent Text Reader

Abstract

An apparatus (400, 703) for processing data samples (401, 701) to form a predicted output (409, 410), the apparatus for: receiving (801) the data samples (401, 701); inputting (802) the data samples (401, 701) and / or one or more arbitrary intermediate outputs of the arbitrary intermediate outputs (406) derived from the data samples to a learnable control function (402); from an output (403) of the function (402), one of the following is performed (803): (i) processing the data samples (401, 701) by a locally stored first model (404, 405, 704) to form a predicted output (409); (ii) sending one or more arbitrary intermediate outputs of the data samples (401, 701) and / or the arbitrary intermediate outputs (406) to a remote location (407, 702) for input to a remotely stored second model (408, 705) to form a predicted output (410). The controller may advantageously utilize intrinsic semantic features of the data to determine where to perform the process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to data processing for forming a prediction output, such as data processing during inference, which can be performed locally on a device or jointly on the device and at a remote location (such as a cloud server). Background Art

[0002] Recently, the field of Machine Learning (ML) has achieved several breakthroughs in familiar tasks and applications such as image classification, analysis, face / object recognition, natural language processing, and regression-type prediction through the Deep Learning (DL) paradigm. As a result, many technology companies have decided to deploy DL models and provide related services to their users to perform various ML tasks. However, considering that executing these models is very energy-consuming and time-consuming, the standard approach for these services is cloud-based. Specifically, when a data point (or sample) arrives and a device (such as a laptop) has to perform an inference task, the sample is sent to the cloud, where it is processed by a large DL model. Then, the output is returned to the device. In this context, ML inference is defined as the process of inputting data into an ML algorithm or model to calculate an output, such as a numerical score, a numerical value for regression tasks, or a class when dealing with classification problems.

[0003] Therefore, the current standard practice in the industry is to send data samples to the cloud, where powerful servers pass the samples forward. This approach has the obvious benefit that arbitrarily large DL models can be used for the forward pass operation, which is not feasible for energy-constrained devices using batteries. On the other hand, inference takes longer, and according to recent research, using too many layers can potentially harm prediction accuracy.

[0004] As users use their mobile devices more and more frequently, there is a new demand to perform ML tasks on devices that reside at the edge of the network and are wirelessly connected to the cloud. Therefore, hardware manufacturers have begun to integrate DL models on devices to speed up ML inference and / or avoid using the wireless medium for each inference request, making the inference latency extremely low. Obviously, it is impractical to deploy large models (such as AlexNet or ResNet) on mobile devices due to energy consumption. Therefore, this approach is limited to smaller models, but smaller models may have lower prediction accuracy.

[0005] In mobile computing, this raises an important question: “Where should the model be executed? In the cloud, on the device, or a combination of both?” This question is particularly tricky because it presents a clear trade-off: local inference (i.e., on the device) has low millisecond latency, while remote execution (in the cloud) enables the use of significantly larger DL models, but at the expense of much higher response latency.

[0006] To this end, a flexible distributed deep neural network (DDNN) for inference has recently become a promising approach. An example of this approach is described in S. Teerapittayanon, B. McDanel, HT Kung, "Distributed Deep Neural Networks over the Cloud, the Edge and End Devices" (IEEE ICDCS, 2017). Figure 1 A representative view is shown. A deep neural network (DNN) comprising N layers is divided into layers. edge The layers are arranged at the edge device 101, and the remaining N cloud =NN edge The layers are located at the cloud 102. The system is equipped with one or more so-called "local exports". The local exports are the output predictions of certain layers on the edge side of the DNN, such as numerical values ​​for regression or classes for classification.

[0007] During the inference phase, as each data sample x 103 arrives, the DDNN mechanism should ideally be able to optimally decide whether the prediction should be offloaded to the cloud using the full DNN, or whether the local hardware is sufficient. If the local egress is chosen, only the local model is executed and the final inference result is the local prediction. In this case, no bits are sent to the cloud server over the wireless channel, so it is impossible to know what might have happened if all DNN layers were used. In other cases, the sample is first forward-passed through the local layers, and then the output of one of the layers is sent to the cloud server over the wireless channel and used as the input to the remaining layers at the cloud. The inference of the full DNN is returned to the device over the wireless channel.

[0008] Despite the flexibility and practicality of the DDNN setup, there are still two significant challenges to be addressed. The first challenge is how to train a DNN with multiple outlets, and more importantly, how to train the DNN to output useful and relevant predictions for all outlets of the DNN. Since edge and cloud entities are usually geographically far apart, centralized or distributed training can be used for this purpose depending on the needs of the problem.

[0009] The second key question for DDNNs is how to make offloading decisions. In principle, a device receives a sample and can process it using local layers (and all intermediate outputs) and implemented logic that decides which outlet to use. Based on this information, which part of the data and which type of algorithm should output the decision? A common approach is to use the confidence of the local outlet prediction. However, this approach is mainly applicable to classification tasks, where the entropy of the outlet (the probability mass function of the class) can be used as a quantitative indicator of confidence.

[0010] There are three major weaknesses in making decisions based on confidence. First, it is limited to classification problems and cannot be generalized to other tasks, such as regression (e.g., time series forecasting, analysis, etc.). These tasks are equally important when it comes to distributed reasoning with DNNs. In addition, local export confidence is not necessarily a relevant proxy for export quality; local exports may be correct or uncertain. In addition, useful data is not utilized, such as the input sample x, which can reveal relevant structural properties for making decisions.

[0011] Another approach described in “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge” by Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang (ASPLOS, 2017) demonstrates the benefits of partitioning a DNN by layers. In this approach, the DNN is partitioned between the device and the cloud, and samples are forward-passed through all available layers at a time. By performing an exhaustive search over (a) multiple “breakpoint” layers (how many layers are at the device and how many are in the cloud) and (b) various wireless protocols, the goal of this research is to provide performance measurements of energy consumption and latency. Architectures such as Figure 2 As shown, each block F0, F1, F2 shown as 201, 202 and 203, respectively, is a DL model (or function) composed of one or more DNN layers. Blocks 201 and 202 are located at edge devices 204, and block 203 is located at the cloud 205. Samples 206 are forward-passed through all available layers each time.

[0012] An obvious drawback of this is the lack of flexibility, since by default all samples are sent to the cloud, which can result in high latency and is not necessarily beneficial, since for some samples the accuracy may be higher with fewer layers.

[0013] In recent research (see S. Teerapittayanon, B. McDanel, HT Kung, "Distributed Deep Neural Networks over the Cloud, the Edge and End Devices" (IEEE ICDCS, 2017)), such as Figure 3 As shown, a certain percentage of samples can exit at the device 301 without being sent to the cloud 302. For this purpose, the concept of a decision mechanism and a local exit is introduced. The decision mechanism schematically shown in box 303 determines whether the sample will be parsed locally (on the device) or whether additional processing should be performed. It should be noted that this operation is only for image classification problems. The algorithm for indicating where inference is performed is based on comparing the entropy of the local prediction with a predetermined threshold.

[0014] Sample x 304 is forward passed through F0 305, producing an edge layer output z. The edge layer output z is fed to the output local prediction y 0 Function F1 306. Calculate y 0 If the entropy is high, z is sent to the cloud 302 where it passes through the remaining layers F2 307. Otherwise, the local prediction y is returned. 0 , thus avoiding sending it to the cloud.

[0015] A major limitation of this approach is that the controller is based on heuristic selection, since in principle the entropy / uncertainty of the output is not necessarily a good proxy for the actual accuracy of the output. Furthermore, the entropy of the output is a natural way to quantify the uncertainty of a probability mass function for a discrete support set (e.g., classifying between M different classes) and does not generalize to other tasks such as regression (e.g., time series forecasting).

[0016] A method needs to be developed to address these issues. Summary of the invention

[0017] According to one aspect, a device for processing a data sample to form a predicted output is provided, the device being used to: receive the data sample; input the data sample and / or one or more of any intermediate outputs derived from the data sample into a learnable control function; and perform one of the following based on the output of the learnable control function: (i) process the data sample through a first model stored locally on the device to form the predicted output; (ii) send the data sample and / or one or more of any intermediate outputs derived from the data sample to a remote location for input into a second model stored at the remote location to form the predicted output.

[0018] The controller is designed as a learnable function that can utilize the inherent semantic features of the data sample. The device can determine whether to execute locally on the device or jointly locally and remotely (for example, at a cloud server) based on one or more of the following, rather than the final output of the first model: the input sample itself and any intermediate output of the first model. Since the method does not rely on local prediction confidence, this can reduce energy consumption. A local prediction output is formed only when the controller indicates a local prediction output, and a local prediction output is not formed when the output of the controller indicates that the prediction output should be formed at a remote location. Since the controller can determine whether the prediction output will have a higher accuracy when formed at a remote location, this can also make reasoning faster and more accurate.

[0019] If the output of the learnable control function exceeds a threshold, the device may be configured to process the data sample through a first model stored locally on the device to form a predicted output. The threshold may be a predetermined threshold. This may be a convenient way to determine whether to process the data sample locally to form a predicted output.

[0020] If the output of the learnable control function does not exceed the threshold, the device may be configured to send the data sample and / or one or more of the intermediate outputs derived from the data sample to the remote location for input to the second model stored at the remote location to form the predicted output. The threshold may be a predetermined threshold. This may be a convenient way to determine whether a predicted output is formed by a remotely stored second model.

[0021] The device can be used to store and implement the first model. Any intermediate output derived from the input sample can be one or more intermediate outputs of the first model (i.e., not the final output). There can be multiple intermediate outputs of the first model. The one or more intermediate outputs input to the learnable control function can be one or more different intermediate outputs of the one or more intermediate outputs sent to a remote location. The first model can include multiple parts. An intermediate output can include the output of the first part of the multiple parts. Other intermediate outputs can include corresponding outputs of other parts of the first model.

[0022] The first part of the first model can be used to encode the data samples. In this way, the data samples can be locally encoded while being simultaneously input into the learnable control function. The encoded data samples can then be sent to the cloud. This can improve the efficiency of the process.

[0023] In some embodiments, the device can be used to input data samples to a learnable control function and, based on the output of the learnable control function, send the data samples to a remote location for input to a second model to form a predicted output.

[0024] As described above, one or more intermediate outputs input to the learnable control function may be one or more different intermediate outputs from the one or more intermediate outputs sent to a remote location. For example, where the device is used to input an intermediate output of a first portion of a first model to a learnable control function, the device may be used to send the output of another intermediate portion (i.e., not the final portion) of the first model to a remote location based on the output of the learnable control function. Alternatively, where the first intermediate output includes the output of another intermediate portion of the first model, the device may be used to send the output of the first portion of the first model to a remote location based on the output of the learnable control function. In other examples, one or more intermediate outputs input to the learnable control function may be one or more intermediate outputs that are the same as the one or more intermediate outputs sent to a remote location.

[0025] The computational requirements and / or storage size requirements of the first model may be lower than the computational requirements and / or storage size requirements of the second model. For example, the first model may include fewer convolutional layers than the second model. This allows predictions to be formed on the device quickly and efficiently.

[0026] The learnable control function can be used to form an output based on features extracted from the data sample. For example, if the data sample is an image, the learnable control function can form an output based on features extracted from the image.

[0027] The learnable control function can be used to optimize based on a series of data samples and their corresponding true outputs. For example, the control function can be trained to learn which features of an input image should be used to reliably determine whether inference should be performed at the device or at a remote location. The control function can also be trained in a principled manner based on gradient descent steps rather than heuristic threshold-based methods.

[0028] The first model and the second model may be learnable models. Each of the first model and the second model may be used to optimize based on a series of data samples and their corresponding true outputs. The first model and the second model may be trained jointly or sequentially using a control function. This allows the system to be optimized as a whole, thereby producing more accurate predictions.

[0029] The remote location may be a cloud server. This allows a large, powerful second model to be stored at the remote location, which can be used to form a prediction output when necessary. This allows for a more accurate prediction.

[0030] The learnable control function can be a neural network including one or more convolutional layers. Depending on the application, different DL building blocks such as DNN, CNN, and FC layers can be used. This allows a flexible approach tailored to the application.

[0031] The input samples can be images or time series data, which enables the device to be used for various image classification, object / face recognition applications and multivariate time series prediction tasks.

[0032] The device may be a network node or an edge device in a communication network. This allows the method to be used in various network configurations.

[0033] According to a second aspect, a method for processing a data sample to form a predicted output is provided, the method comprising: receiving the data sample; inputting the data sample and / or one or more of any intermediate outputs derived from the data sample into a learnable control function; and performing one of the following according to the output of the learnable control function: (i) processing the data sample through a first model stored locally on the device to form the predicted output; (ii) sending the data sample and / or one or more of any intermediate outputs derived from the data sample to a remote location for input into a second model stored at the remote location to form the predicted output.

[0034] In the method, the controller is designed as a learnable function that can exploit the inherent semantic features of the data sample. The method determines whether to execute locally on the device or jointly locally and remotely (e.g., at a cloud server) based on one or more of the following, rather than the final output of the first model: the input sample itself and any intermediate output of the first model. This can reduce energy consumption because the method does not rely on local prediction confidence.

[0035] According to yet another aspect, there is provided a computer readable storage medium storing computer readable instructions, which, when executed at a computer system, cause the computer system to perform the above method. The computer system may include one or more processors. The computer readable storage medium may be a non-transitory computer readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention is described below by way of examples with reference to the accompanying drawings.

[0037] In the attached picture:

[0038] Figure 1 Schematic illustration of a basic prior art setup for distributed inference, including edge devices with DNN layers and local egress and potential signaling of intermediate layer outputs to a cloud server.

[0039] Figure 2 Schematically illustrates a prior art DNN partitioned on edge devices and cloud servers.

[0040] Figure 3 A prior art DDNN including early (local) exit and entropy-based controller mechanisms is schematically shown.

[0041] Figure 4 An exemplary architecture is schematically shown, where the controller is implemented as a learnable function.

[0042] Figure 5 A flow chart schematically illustrates the decisions, actions, and signal flows.

[0043] Figure 6 An example of joint training of all involved sub-models is schematically shown.

[0044] Figure 7 An exemplary embodiment arrangement is schematically shown.

[0045] Figure 8 An example of a method for forming a prediction output provided by an embodiment of the present invention is shown.

[0046] Fig. 9 (a) to Fig. 9(c) in Fig. 2 shows the histogram of p in the full training set at different training cycles. Fig. 9 (a) shows the value of period 1, Fig. 9 (b) in the figure shows the value of period 3, Fig. 9 (c) in FIG. 7 shows the value of the period 70 .

[0047] Fig.10 An example of results is shown, comparing a time series signal with a DDNN forecast. DETAILED DESCRIPTION

[0048] In an embodiment of the present invention, the decision / controller mechanism is a learnable function that can utilize machine learning techniques and use semantic properties of input samples to determine whether the input samples are processed locally on the device or at least partially processed at a remote location (such as a cloud server).

[0049] For example, the device may be a network node or edge device in a communication network. In one implementation, the device may be a base station. In other implementations, the device may be a mobile phone, a tablet computer, a laptop computer, or a PC. Preferably, the device and the remote location are connected via a network. The remote location may be a cloud server or some other remote server or device.

[0050] The device may include a processor and a memory. The memory stores code in a non-transient manner, which can be executed by the processor to implement the device in the manner described herein. The device may also include a transceiver for sending and receiving data. The network may be a wireless network or a wired network.

[0051] like Figure 4 In the example of FIG. 4 , the device 400 is used to receive an input sample x t 401.

[0052] In this example, edge device 400 is a mobile phone. A predicted output needs to be formed based on input sample 401. This can be an output related to applications such as image classification, analysis, face / object recognition, natural language processing, and regression type prediction. For example, the input sample can be an image, time series data, or some other data sample. For example, in a specific classification application, the predicted output can be a prediction indicating whether the input image represents an image of a dog.

[0053] The device 400 implements a decision controller module that controls which samples are parsed locally and which samples are parsed remotely. In a preferred implementation, samples parsed remotely to form a predicted output will be processed by a full DNN forward pass of a larger model than the model used for local processing.

[0054] like Figure 4 As shown, input sample x t 401 is passed forward through a controller function C(x) 402 to form an output 403. The controller function 402 is preferably stored locally on the device 400, which is used to implement the controller function. The output 403 of the controller function can be regarded as a decision signal. In this example, the output of the learnable control function is the edge controller output p.

[0055] C(x) is a learnable control function. The learnable control function 402 can be used to form an output 403 based on features extracted from the input sample 401, for example, based on features in or extracted from the input image. The learnable control function can be used to optimize during training based on a series of data samples and their corresponding true outputs, which will be described in more detail below. The learnable control function can be a neural network including one or more convolutional layers.

[0056] The output 403 of the learnable control function indicates whether inference will be performed locally or remotely, which will be described in more detail below.

[0057] The first model is stored locally on the device. The device is used to implement the first model. In this example, the first model includes a plurality of functions F0(x) and F1(z), as shown in 404 and 405, respectively. In this example, the input of the first part 404 of the model is the same as the input of the separate learnable control function 402 (i.e., the input is the data sample x). t The output of the first part 404 of the first model is z t , as shown in 406. The first part of the first model can be used to encode the data sample. Therefore, z t Can be an input data sample x t The encoded version of .

[0058] In other embodiments, the input of the learnable control function may be an intermediate output derived from the data sample, such as z t The input of the learnable control function may be the output of the first part 404 of the first model (the output of the first part 404 of the first model), or may be the output of another intermediate part of the first model (i.e., not the final output) if the first model includes more than two parts. In some embodiments, the input of the learnable control function may be multiple intermediate outputs of the first model. The input of the learnable control function does not include the final output of the first model. In some embodiments, the data sample and any intermediate output of the first model (or multiple intermediate outputs of the first model, where the first model includes more than two parts) may be input to the learnable control function.

[0059] The second model is stored at a remote location of the device, for example, at Figure 4At the cloud server shown in 407 in FIG. In this example, the second model includes function F2(z) 408. Cloud server 407 is used to implement the second model.

[0060] The computational requirements and / or storage size requirements of the first model 404, 405 may be lower than the computational requirements and / or storage size requirements of the second model 408. For example, the first model may include fewer convolutional layers than the second model.

[0061] Optionally, x t can be compared with x passed forward through the controller function C(x) 402 t Simultaneously input to the first part 404 of the first model.

[0062] like Figure 4 As shown in the example, the input sample x t Through the forward pass of F0(x), the edge layer output z is generated t .z t is the intermediate output of the first model. t Can be stored at the device for potential transmission to the cloud or for input into the second part F1(z) of the first model.

[0063] exist Figure 4 In the example shown, the intermediate output z is only used when the output p of the learnable control function indicates that inference should be performed locally on the device. t is fed to the second part F1(z) of the first model. In this case, the local prediction is formed 409 and use it as the prediction output.

[0064] Otherwise, the output z of the first part of the first model is t It can be forwarded to the cloud for additional processing by the second model F2(z) 408. In this case, the predicted output is the output from the cloud, i.e. 410.

[0065] Alternatively, in response to the learnable control function 402 indicating that the input sample should be processed remotely, x t directly to the cloud server 406. However, if encoding the input sample 401 by inputting the input sample 401 to the first part 404 of the first model is performed at the device 400 simultaneously with inputting the input sample 401 to the learnable control function 402, the processing time can be reduced.

[0066] In reference Figure 4 In the example described, the device is used to convert the data sample x tInput to a learnable control function. In other embodiments where the device is used to input an intermediate output derived from a data sample into a learnable control function, if the output of the learnable control function indicates that reasoning should be performed at a remote location via a second model, the device can be used to send the intermediate output and / or different intermediate outputs to a remote location for input to the second model. Optionally, data samples can also be sent. For example, where the first model includes more than two parts and where the intermediate output input to the learnable control function includes the output of a first part of the first model, the device can be used to send the output of another intermediate part of the first model to a remote location based on the output of the learnable control function. Alternatively, where the intermediate output input to the learnable control function includes the output of another intermediate part of the first model, the device can be used to send the output of the first part of the first model to a remote location based on the output of the learnable control function.

[0067] Thus, the learnable control function may receive as input one or more of the following: (i) an input sample (ii) at least one arbitrary intermediate output of the first model, such as z t (but will not receive the final output of the first model) and output a decision signal.

[0068] Based on the decision signal, the device is configured to: (i) process the input sample by inputting the input sample into a first model stored locally in the device to form a predicted output, or (ii) process the input sample and / or one or more of any intermediate outputs (e.g., z t ) is sent to the cloud (but not the final output of the first model) and processed by the second model to form a predicted output. The output of the learnable control function (i.e., the decision signal) is not based on the entropy or uncertainty of the final output of the first model.

[0069] In one example, if the output 403 of the learnable control function 402 ( Figure 4 p) in the example shown exceeds a threshold, the device is used to process the data sample through a first model stored locally on the device to form a predicted output. If the output of the learnable control function does not exceed the threshold, the device is used to send one or more of the following to a remote location for input to a second model stored at the remote location to form a predicted output: the data sample and any intermediate output derived from the data sample.

[0070] Figure 5 A flow chart outlining one example of system operation is shown.

[0071] At step 501, an input sample arrives at the device. At step 502, the device performs a forward pass on the input sample through a learnable control function. The learnable control function outputs a control / decision signal. In this example, at step 503, the device simultaneously passes the first layer ( Figure 4 F o (x), 404) performs a forward pass on the input sample.

[0072] At 504 , the device determines whether inference (ie, formation of a predicted output) should be performed at the device or at a remote location based on the output of the learnable control function (but not the output of the first portion of the first model).

[0073] If the device determines that the inference should be performed locally on the device based on the output of the control function, the device performs z through the last device layer (i.e., one or more remaining parts of the first model 405). t In the forward pass, as shown in 505, the inference result is formed at the device.

[0074] If the device determines that inference should be performed remotely on the cloud server based on the output of the control function, the device will t The second model is sent to the cloud server, as shown in 506, and the cloud server implements the second model to t The forward pass is performed, as shown in 507. The inference result is formed in the cloud. Then, the inference result can be returned to the device, as shown in 508.

[0075] Thus, based on the output of the learnable control function, the device is configured to process the data sample through a first model stored locally on the device to form a predicted output, or to send one or more of the following to a remote location for input to a second model stored at the remote location to form a predicted output: the data sample and any intermediate output derived from the data sample (e.g., z t ). The device does not send the final output of the first model to a remote location for further processing.

[0076] A learnable control function C(x) is a function that can be optimized using training data. For example, the control function can learn which features of an image should be used to reliably determine whether inference should be performed at the device or at a cloud server. In addition, the first model and the second model can also be trainable.

[0077] Each of the first model and the second model can be used to optimize based on a series of training data samples and their corresponding true outputs.

[0078] An example of a training process for a learnable function is now described.

[0079] The goal is to design a training (optimization) process based on the expected loss function and the variables that affect the expected loss function. Figure 6 The signal flow graph shown provides an abstract level description of DNN training, schematically illustrating the joint training of all sub-models involved.

[0080] In these examples, the first model includes two parts F0, F1, as shown in 601 and 602, respectively, and the second model includes a part F2, as shown in 603. In other embodiments, the first model may include at least two parts and the second model may include multiple parts. F0, F1, F2, as shown in 601, 602, 603, respectively, are DNN models of appropriate size. In this example, the controller function C 604 is also a DNN. The output of the controller function is p, as shown in 605, and the output z of the first part of the first model is shown in 606.

[0081] Each training data set includes a data sample x as shown in 607 and a corresponding true output y of the data sample as shown in 608 true .

[0082] Local prediction output y 0 As shown in 609, the remote prediction output y 1 As shown in 610.

[0083] Below are two examples of how to perform DDNN training: first, a joint approach (all models together), and second, a sequential approach (first the F0, F1, F2 models, then the learnable controller).

[0084] One way to tie all four sub-models in this example is to consider a technique called "joint training", where multiple outlets are included in a single loss function. Without a controller, to indicate the importance of each outlet, one can consider using fixed scalar weights w0 and w1 to represent the importance of local and remote outlets, respectively, during backpropagation via the following loss function:

[0085] L(y true ,y 0 ,y 1 )=w0L(y true ,y 0 )+w1L(y true ,y 1 )

[0086] Among them, L(y true,yi) is the loss function depending on the ML task, such as Mean Squared Error (MSE) for regression or Categorical Cross Entropy (CCE) for classification tasks. In this way, the controller can probabilistically choose which exit is more important by relaxing the decision signal p and treating it as a probability value ∈ [0,1].

[0087] L(y true ,y 0 ,y 1 ,p)=(1-p)L(y true ,y 0 )+pL(y true ,y 1 )

[0088] With the above loss function, the DNN parameters of all models (i.e., F0, F1, F2, and C) are jointly updated in each back-propagation step.

[0089] The above process trains all DNN models F0, F1, F2 and C, which together generate the three output signals p, y of the system. 0 ,y 1 However, this process can also be done in steps during sequential training. Specifically, the training cycle can be divided into the following two steps:

[0090] Consider the importance of the outputs of the two outlets, for example, the importance is equal, that is, w0 = w1 = 1, so that L(y true ,y 0 ,y 1 )=L(y true ,y 0 )+L(y true ,y 1 ). This step trains all sub-models required to perform inference.

[0091] Furthermore, once F0, F1, and F2 are all trained, they are considered known / fixed models; they can be trained by forward passing all available samples x and measuring the loss L0(x)≡L(y true ,y 0 ) and L1(x)≡L(y true ,y 1 ) to make use of it.

[0092] So in this example, there are two scalar numbers per sample representing the loss incurred in the two exits. You can use these two values ​​to view the training of the controller:

[0093] L(p;x)=(1-p)L0+pL1

[0094] And only perform gradient steps on the DNN model parameters of C through the above loss function.

[0095] However, the problem of selecting the exit with the lowest cost can also be explicitly regarded as a classification problem. Assuming that the transmission costs (from the device to the cloud and from the cloud to the device) are known as T, the following operations can be performed: for each x, if L1(x)+T < L0(x), the label of x is "1", otherwise it is "0". The training loop can run as a classification (supervised learning) problem with known labels.

[0096] It should be noted here that the inequality L1(x)+T < L0(x) means that for a sample x, if the cost of remote inference plus the transmission cost T is less than the cost of local inference, the system must consider remotely parsing the sample.

[0097] Figure 7 Another exemplary embodiment of the system in the inference phase is schematically shown. In this example, the input sample 701 includes a traffic signal stream arriving at a certain cloud server 702 from M base stations (BS). In this example, the task of the cloud server 702 is to perform a certain analysis task on this measurement set. For example, the cloud server can perform traffic prediction for the next (or multiple) time instances of all BSs. In this setting, the server can be mapped to a cloud server node. However, considering the new requirements of future wireless networks for more accurate and faster inference, an auxiliary node 703 can be used as a local device.

[0098] Therefore, in this embodiment, local inference is performed at the auxiliary node 703, and remote inference of the additional layer is performed at the cloud server 702.

[0099] In summary, the system operates as follows. A time series of N past samples (t-N... t-1) of M BSs is fed into the auxiliary node 703, as shown in 701. Given these measurements, the auxiliary node 703 can predict the traffic of each involved BS at time t, for example, for resource provisioning.

[0100] The auxiliary node 703 is equipped with a controller that receives the input data 701 and determines at each time t whether it is better to perform the prediction locally (only through the device layer) or forward the output z of the intermediate layer to the cloud server. The controller implements a learnable control function, as described above.

[0101] At Figure 7In one implementation of the layout shown, real data from the Milan traffic dataset is used to train the controller function and the first and second models for a realistic and challenging environment. M = 49 BSs are considered. For prediction, N = 6 past time instances are used. Therefore, the input sample is an array of size M × N, and the expected output is an array of size M (one traffic prediction for each BS).

[0102] The mobile networking community has shown that it is beneficial to use Convolutional Neural Networks (CNNs) for traffic prediction related tasks, especially when multiple (spatially or temporally correlated) BSs are involved.

[0103] To this end, the auxiliary nodes can advantageously implement a CNN as F0 to encode the input sample (to produce z) and one or more fully connected (FC) layers as F1 to decode z and return the local output y 0 The first model includes F0 and F1, in Figure 7 The 704 locations are uniformly shown.

[0104] At cloud server node 702, CNN and / or one or more FC layers may be arranged as F2, as shown in 705. The function F2 receives z and outputs y 1 (Long-range prediction output).

[0105] In this example, the learnable function at the controller is represented by one or more FC layers that receive data samples x as input and output a decision signal. According to the decision signal, the auxiliary node processes the data sample through a first model stored locally on the device to form a predicted output, or sends an intermediate output z derived from the data sample to the cloud server 702 to be input to a second model stored at the cloud server to form a predicted output.

[0106] The above architecture is suitable for multivariate time series forecasting tasks. Depending on the application, different DL building blocks can be used (instead of CNN and FC layers).

[0107] Figure 8A flowchart of an example of a method for processing a data sample to form a predicted output provided by an embodiment of the present invention is shown. In step 801, the method includes receiving a data sample. In step 802, the method includes inputting the data sample and / or one or more of any intermediate outputs derived from the data sample into a learnable control function. In step 803, the method includes performing one of the following according to the output of the learnable control function: (i) processing the data sample through a first model stored locally on the device to form a predicted output; (ii) sending the data sample and / or one or more of any intermediate outputs derived from the data sample to a remote location to be input into a second model stored at the remote location to form a predicted output.

[0108] The current method is different from the confidence-based method. Alternatively, the controller is designed as a learnable function that can utilize the inherent semantic features of the input sample. The device can determine whether to execute locally on the device or remotely (e.g., at a server) based on the input sample itself and / or one or more arbitrary intermediate outputs of the first model, rather than based on the final output of the first model implemented by the device.

[0109] The proposed architecture and training have several advantages over earlier approaches, as described below.

[0110] The methods described herein can reduce energy consumption. Function F1 is not executed for each input sample as in previous DDNN methods. Known DDNN architectures utilize local prediction confidence. Therefore, these methods must generate prediction outputs for all input samples. However, in an embodiment of the present invention, local prediction outputs are generated only when the controller indicates local prediction outputs, and when the output of the controller indicates that the prediction output should be generated at a remote location, no local prediction output is generated.

[0111] The approach is application-independent. The controller C(·) is designed as a learnable function that can generate decisions independently of the task; there is no need to find a way to quantify the confidence of the exit. The approach described in this paper provides a more general way of making decisions. There are no application (task)-independent DDNN architectures in the literature or industry. The controller described in this paper is not based on local prediction uncertainty / confidence (special attention is required for different ML tasks), but is built as a learnable function at a higher level of abstraction.

[0112] As mentioned above, the controller can also use other intermediate outputs (e.g., intermediate outputs of F0) of the local DNN (i.e., the first model stored locally on the device) as input without much modification. Earlier technical levels only considered local export data and discarded the rich information in the input data samples (and possible intermediate outputs).

[0113] The performance of the controller C(·) is based on a principled gradient descent step rather than a heuristic threshold protocol.

[0114] Previous studies considering the “cloud offloading” problem used simplified information (only local prediction y 0 ), we focus on the confidence (entropy) of the output (a proxy for output quality), and the confidence level is usually arbitrary.

[0115] To this end, the controller described in this paper can exploit the semantic structure of the input samples and use more information from the local layer. By formulating the question of "whether to offload to the cloud" as an ML problem itself, rather than using predefined heuristic rules, the loss function and signal flow can be formulated so that the controller model minimizes the true loss as an explicit goal.

[0116] To obtain the following results, the model (inference model plus controller) was trained using the “sequential training” method.

[0117] After describing the system architecture, a reasonable benchmark is to consider a fully centralized DNN deployed at the cloud server with two CNN and three FC layers (thus, having the same representation power as the case where all samples are parsed at the cloud).

[0118] Fig. 9 (a) to Fig. 9 (c) in FIG. 5 shows a plotted histogram of the values ​​of the decision probability p (ie, the probability of performing more processing at the remote cloud exit). Fig. 9 (a) shows the value of period 1, Fig. 9 (b) in the figure shows the value of period 3, Fig. 9 (c) in FIG. 7 shows the values ​​of period 70. These values ​​can be used to verify whether the learnable controller performs the non-trivial task of distinguishing samples.

[0119] In each histogram, a full forward pass of the training data points is performed and the p-value produced by the controller is reported. It is clear from these plots that the controller is deterministically splitting the samples as more training steps are executed (the histogram is "polarized" to 0 and 1 in epoch 70).

[0120] Specifically, if the controller outputs a value close to 1.0, the full DNN (local plus remote) is chosen. If the controller outputs a value close to 0.5, its decision is less certain. Finally, if the controller outputs a value close to 0.0, the local export is deterministically chosen.

[0121] In the following, the performance gains of the architecture described in this paper will be quantified and compared with the baseline at the inference stage. The following discussion and evaluation rely on two metrics: the prediction cost from the perspective of mean squared error (MSE), and the percentage of samples that are locally resolved. Table 1 below describes these two metrics in the two schemes under consideration.

[0122] Table 1: Main performance comparison results

[0123] MSE cost (normalized) Percentage of local predictions (%) Methods Described in This Article 0.69 20.1 Benchmarking Centralized DNN 1.0 0.0

[0124] Table 1 shows the improvement over the baseline in two directions. First, the approach described in this paper is able to resolve 1 / 5 of the input traffic samples locally (so very fast, no latency), and more importantly, using an intelligent controller on the edge side reduces the MSE by more than 30% compared to a fully centralized DNN. The latter is evidence of the presence of a structure in the input samples. The controller described in this paper is able to detect this structure and decide whether the prediction will have higher accuracy using a local DNN local to the device or a full DNN in the cloud.

[0125] Fig.10 The time series signal (continuous line) and DDNN prediction (edge / local: “x”, cloud / remote: “o”) are shown, i.e., the true BS traffic signal and the predicted signal. At each time step, the controller decides whether the predicted signal is the signal output by the device or the signal output by the cloud. The time series plot shows that the decision made by the controller is reasonable because in the snapshot shown, the predicted value selected from the edge or the cloud is close to the true signal and can therefore be considered a good prediction.

[0126] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, without considering whether such features or combinations of features can solve any problems disclosed herein, and without limiting the scope of the claims. Applicants point out that various aspects of the present invention may include any such individual features or combinations of features. In view of the above description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. An apparatus (400, 703) for processing a data sample (401, 701) to form a predicted output (409, 410), the apparatus (400, 703) being configured to: Receiving (801) the data sample (401, 701); Inputting (802) the data sample (401, 701) and / or one or more of the intermediate outputs (406) derived from the data sample (401, 701) into a learnable control function (402); Based on the output (403) of the learnable control function (402), perform (803) one of the following: (i) processing the data sample (401, 701) by a first model (404, 405, 704) stored locally on the device (400, 703) to form the predicted output (409); (ii) converting the data sample (401, 701) and / or any intermediate output (406) derived from the data sample (401) One or more of any intermediate outputs in the prediction output (410) are sent to a remote location (407, 702) to be input into a second model (408, 705) stored at the remote location to form the prediction output (410).

2. The device (400) according to claim 1, characterized in that If the output (403) of the learnable control function (402) exceeds a threshold, the device is configured to process the data sample (401) through the first model (404, 405) stored locally on the device (400) to form the predicted output (409).

3. The device (400) according to claim 1 or 2, characterized in that If the output (403) of the learnable control function (402) does not exceed a threshold, the device (400) is configured to send the data sample (401) and / or one or more of the arbitrary intermediate outputs (406) derived from the data sample to the remote location (407) for input into the second model (408) stored at the remote location to form the predicted output (410).

4. The device (400) according to any one of the preceding claims, characterized in that The any intermediate output (406) derived from the data sample (401) is one or more intermediate outputs (406) of the first model (404, 405) stored on the device (400).

5. The device (400) according to claim 4, characterized in that The first model includes a plurality of parts (404, 405), and an intermediate output (406) includes an output of a first part (404) of the plurality of parts.

6. The device (400) according to claim 5, characterized in that The first part (404) is used to encode the data sample (401).

7. The device (400) according to any one of the preceding claims, characterized in that The computational requirements and / or storage size requirements of the first model (404, 405) are lower than the computational requirements and / or storage size requirements of the second model (408).

8. The device (400) according to any one of the preceding claims, characterized in that The first model (404, 405) includes fewer convolutional layers than the second model (408).

9. The device (400) according to any one of the preceding claims, characterized in that The learnable control function (402) is used to form the output (403) based on features extracted from the data sample (401).

10. The device (400) according to any one of the preceding claims, characterized in that The learnable control function (402, 604) is used to perform optimization based on a series of data samples (607) and their corresponding true outputs (608).

11. The device (400) according to claim 10, characterized in that The first model (404, 405, 601, 602) and the second model (408, 603) are learnable models, and each of the first model and the second model is used to optimize according to the series of data samples (607) and their corresponding true outputs (608).

12. The device (400, 703) according to any one of the preceding claims, characterized in that The remote location is a cloud server (407, 702).

13. The device (400) according to any one of the preceding claims, characterized in that The learnable control function (402) is a neural network comprising one or more convolutional layers.

14. The device (400, 703) according to any one of the preceding claims, characterized in that The input samples (401, 701) are images or time series data.

15. The device (400, 703) according to any one of the preceding claims, characterized in that The device is a network node or an edge device in a communication network.

16. A method (800) for processing a data sample (401, 701) to form a predicted output (409, 410), the method comprising: Receiving (801) the data sample (401, 701); Inputting (802) the data sample (401, 701) and / or one or more of the intermediate outputs (406) derived from the data sample (401) into a learnable control function (402); Based on the output (403) of the learnable control function (402), perform (803) one of the following: (i) processing the data sample (401, 701) by a first model (404, 405, 704) stored locally on the device (400, 703) to form the predicted output (409); (ii) sending the data sample (401, 701) and / or one or more of the arbitrary intermediate outputs (406) derived from the data sample (401) to a remote location (407, 702) for input into a second model (408, 705) stored at the remote location to form the predicted output (410).

17. A computer-readable storage medium storing computer-readable instructions which, when executed at a computer system, cause the computer system to perform the method of claim 16.