Method and system for partitioning and bit width allocation of deep learning models for distributed system inference

Abstract

A system and method are provided for partitioning a trained neural network into a first neural network for execution on a first device and a second neural network for execution on a second device. The partitioning is performed to optimize the overall latency of the following operations within accuracy constraints: executing the first neural network on the first device to generate a feature map output based on input data, sending the feature map output from the first device to the second device, and executing the second neural network on the second device to generate an inference output based on the feature map output from the first device.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 985,540, filed March 5, 2020, entitled "Secure End-to-End Mixed-Precision Separable Neural Networks for Distributed Inference," the contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to artificial intelligence and distributed computing, and more specifically to a method and system for partitioning and bit-width allocation of deep learning models for inference in distributed systems. Background Technology

[0004] The proliferation of edge devices and advancements in communication and processing systems are driving the generation of massive amounts of data and the demand for large-scale deep learning models to process such data. Large deep learning models are typically hosted on powerful computing platforms (e.g., servers, server clusters, and relational databases) accessible via the internet. In this invention, "cloud" can refer to one or more computing platforms accessible via the internet, as well as the software and databases running on those platforms. The cloud can possess powerful computing capabilities, achieved through multiple powerful processing units and large amounts of memory and data storage. Simultaneously, data collection is typically distributed at the edge of the cloud, i.e., edge devices connected to the cloud via the internet, such as smart home cameras, authorized access devices (e.g., license plate recognition cameras), smartphones and smartwatches, surveillance cameras, medical devices (e.g., hearing aids, personal health and fitness trackers), and Internet of Things (IoT) devices. The combination of powerful deep learning models and abundant data is driving the advancement of AI applications.

[0005] However, the gap between massive amounts of data and large-scale deep learning models persists, posing an increasingly significant challenge to broader AI applications. Exchanging data from deep learning models and the resulting inference results between edge devices and the cloud is far from simple. Due to the extremely limited computing power of edge devices (e.g., edge devices often have limited processing power, limited memory and storage capacity, and limited power supplies), large-scale deep learning models cannot be loaded onto them. In fact, deep learning models are becoming increasingly powerful and large, making them increasingly impractical for edge devices. Recently rolled out large-scale deep learning models cannot even be supported by a single cloud server—these models require cloud clusters.

[0006] Uploading data from edge devices to the cloud is not always ideal, or even feasible. Sending high-resolution, high-volume input data to the cloud can result in high transmission latency and potentially high end-to-end latency for AI applications. Furthermore, sending high-resolution, high-volume input data to the cloud can introduce additional privacy risks.

[0007] Generally, edge-cloud data collection and processing schemes fall into three categories: (1) Edge-Only; (2) Cloud-Only; and (3) Edge-Cloud Collaboration. In the Edge-Only scheme, all data collection and processing functions are executed on the edge device. Model compression techniques are applied to force a fit to the entire AI application, which includes one or more deep learning models on the edge device. In many AI applications, the Edge-Only scheme can suffer from significant accuracy loss. The Cloud-Only scheme is a distributed scheme where data is collected and preprocessed on the edge device, but then sent to the cloud for inference processing by one or more deep learning models in the AI ​​application. The Cloud-Only scheme can result in high data transmission latency, especially when high-resolution data is used for high-precision AI applications. Furthermore, the Cloud-Only scheme may raise data privacy concerns.

[0008] In an edge-cloud collaborative solution, the software program implementing a deep learning model to perform a specific inference task can be decomposed into multiple programs that implement smaller deep learning models to perform the specific inference task. Some of these smaller software programs can run on edge devices, while the rest can run in the cloud. The output generated by the smaller deep learning models running on the edge devices is sent to the cloud for further processing by other smaller deep learning models running in the cloud.

[0009] An example of an edge-cloud collaboration approach is the cascaded edge-cloud inference method, which divides a task into multiple subtasks, deploys some of these subtasks on edge devices, and sends the outputs of these subtasks to the cloud running the other tasks. Another example is the multi-exit approach, which deploys a lightweight model (e.g., a compressed deep learning model) on the edge device to handle simpler cases and sends more difficult cases to a larger deep learning model implemented in the cloud. Both the cascaded edge-cloud inference method and the multi-exit approach are application-specific and therefore not flexible in many use cases. The multi-exit approach may also suffer from low accuracy and nondeterministic latency.

[0010] A flexible solution is needed to enable edge-cloud collaboration, including a mechanism for partitioning deep learning models between asymmetric computing systems (e.g., between edge devices and the cloud) to minimize end-to-end latency in AI applications, allowing deep learning models to be implemented asymmetrically on both computing systems. Furthermore, the solution should be general and flexible so that it can be applied to many different tasks and deep learning models. Summary of the Invention

[0011] According to a first aspect, a system and method are disclosed for partitioning a trained neural network into a first neural network for execution on a first device and a second neural network for execution on a second device. The method includes: identifying from the trained neural network a first set of neural network layers for inclusion in one or more neural network layers in the first neural network, and identifying from the trained neural network a second set of neural network layers for inclusion in one or more neural network layers in the second neural network; assigning weight bit widths to weights of the first set of one or more neural network layers for configuration, and assigning feature map bit widths to feature maps generated by the first set of one or more neural network layers. The identification and assignment are performed to optimize the overall latency of the following operations within accuracy constraints: executing the first neural network on the first device to generate a feature map output based on input data, sending the feature map output from the first device to the second device, and executing the second neural network on the second device to generate an inference output based on the feature map output from the first device.

[0012] This approach allows the inference task of neural networks to be distributed efficiently across multiple computing platforms, including computer platforms with different computing capabilities.

[0013] In some aspects of the method, the identification and the allocation may include: selecting a set of one or more feasible schemes within the accuracy constraint from a plurality of potential partitioning schemes for partitioning the trained neural network into a first set of one or more neural network layers and a second set of one or more neural network layers, wherein each feasible scheme identifies: (i) a partition point indicating a layer in the trained neural network included in the first set of one or more layers; (ii) a set of weight bit widths configuring the weights of the first set of one or more neural network layers; and (iii) a set of feature map bit widths of the feature map generated by the first set of one or more neural network layers.

[0014] In one or more of the foregoing aspects, the method may include: selecting an implementation scheme from the set of one or more feasible schemes; generating first neural network configuration information defining the first neural network and second neural network configuration information defining the second neural network according to the implementation scheme; providing the first neural network configuration information to the first device and providing the first and second neural network configuration information to the second device.

[0015] In one or more of the above aspects, the selection may also be based on the memory constraints of the first device.

[0016] In one or more of the foregoing aspects, the method may include: determining the plurality of potential partitioning schemes based on identifying the transmission cost associated with different possible partitioning points before selecting the set of one or more feasible schemes, the transmission cost being lower than the transmission cost associated with including all layers of the trained neural network in the second neural network.

[0017] In one or more of the foregoing aspects, the selection may include: calculating the quantization error of the combined performance of the first neural network and the second neural network for different weight bit widths and feature map bit widths for each of the plurality of potential schemes, wherein the selection of the set of one or more feasible schemes is based on selecting weight bit widths and feature map bit widths that would cause the calculated quantization error to be within the accuracy constraints.

[0018] In one or more of the above aspects, the different weight bit widths and feature map bit widths of each of the plurality of potential solutions may be uniformly selected from the set of possible weight bit widths and feature map bit widths, respectively.

[0019] In one or more of the foregoing aspects, the accuracy constraint may include a defined accuracy degradation tolerance threshold for the combined performance of the first neural network and the second neural network relative to the performance of the trained neural network.

[0020] In one or more of the foregoing aspects, the first device may have a lower memory capacity than the second device.

[0021] In one or more of the foregoing aspects, the first device is an edge device, and the second device is a cloud-based computing platform.

[0022] In one or more of the foregoing aspects, the trained neural network is an optimized trained neural network represented as a directed acyclic graph.

[0023] In one or more of the foregoing aspects, the first neural network is a mixed-precision network comprising at least some layers having weight bit widths and feature map bit widths different from other layers.

[0024] According to another exemplary aspect, a computer system is disclosed, the computer system including one or more processing devices and one or more non-transient memories storing computer-implementable instructions for execution by the one or more processing devices, wherein execution of the computer-implementable instructions causes the computer system to perform the method according to any one of the preceding claims.

[0025] According to another exemplary aspect, a non-transient computer-readable medium is disclosed, the non-transient computer-readable medium storing computer-implementable instructions for causing a computer system to perform the method according to any one of the preceding claims. Attached Figure Description

[0026] The accompanying drawings, which illustrate exemplary embodiments of this application, will now be shown by way of example, in which:

[0027] Figure 1 It is a block diagram of a distributed environment in which the systems and methods described in this paper can be applied;

[0028] Figure 2 This is a block diagram of the artificial intelligence model partitioning module provided in the example of the present invention;

[0029] Figure 3 This is a flowchart illustrating the actions performed to generate a list of potential partitioning schemes. Figure 2 It is part of the artificial intelligence model partitioning module;

[0030] Figure 4 yes Figure 3 The pseudocode representation of the action, followed by Figure 2 The further actions performed by the optimization scheme selection operation of the artificial intelligence model partitioning module;

[0031] Figure 5 This is a block diagram of an exemplary processing system that can be used to implement the examples described herein;

[0032] Figure 6 This is a block diagram of an exemplary hardware structure of an NN processor provided in an exemplary embodiment;

[0033] Figure 7 This is a block diagram of yet another example of the neural network segmentation system provided by the present invention;

[0034] Figure 8 It shows that according to Figure 7 An example of system segmentation;

[0035] Figure 9 It is based on Figure 7 The pseudocode representation of the method executed by the system;

[0036] Figure 10 An example of a practical application of the method of the present invention is shown.

[0037] Similar reference numerals were used in different accompanying drawings to indicate similar components. Detailed Implementation

[0038] An exemplary scheme for collaborative data processing using distributed deep learning models is disclosed. The collaborative scheme disclosed herein can be applied to different types of multi-platform computing environments, including environments where deep learning models for performing inference tasks are partitioned between asymmetric computing platforms (including, for example, between a first computing platform and a second computing platform with higher computing power and capabilities than the first computing platform).

[0039] refer to Figure 1 The method and system are illustrated in the context of a first computing platform, 88, as an edge device, and a second computing platform, 86, as a cloud computing platform (part of a cloud 82). Specifically, the cloud 82 includes multiple cloud computing platforms 86 accessible to the edge device 88 via a network 84, including the internet. The cloud computing platforms 86 may include powerful computer systems (e.g., cloud servers, cloud server clusters (cloud clusters), and associated databases) accessible via the internet. The cloud computing platforms 86 possess powerful computing capabilities, achieved through multiple powerful and / or specialized processing units, as well as substantial memory and data storage. The edge device 88 is distributed at the edge of the cloud 82 and may include smartphones, personal computers, smart home cameras and appliances, authorized access devices (e.g., license plate recognition cameras), smartwatches, surveillance cameras, medical devices (e.g., hearing aids and personal health and fitness trackers), various smart sensors and monitoring devices, and Internet of Things (IoT) nodes, etc.

[0040] An edge-cloud collaboration scheme is disclosed that leverages the fact that the amount of data processed in an intermediate layer of a deep learning model (also known as a deep neural network model (DNN)) is significantly less than the amount of raw input data to the DNN. This data reduction allows the DNN to be partitioned (i.e., divided) into an edge DNN and a cloud DNN, thereby reducing transmission latency, lowering end-to-end latency for AI applications involving DNNs, and adding privacy elements to data uploaded to the cloud. In at least some examples, the disclosed edge-cloud collaboration scheme is general and can be applied to a wide range of AI applications.

[0041] in this regard, Figure 2 This is a block diagram representation of a system applicable to implementing an edge-cloud collaboration scheme, provided by an example of the present invention. A deep learning model partitioning module 10 (hereinafter referred to as partitioning module 10) receives a trained deep learning model for an inference task as input and automatically processes the trained deep learning model to partition it into a first deep learning model and a second deep learning model, which can be implemented on a first computing platform (e.g., edge device 88) and a second computing platform (e.g., a cloud server cluster or cloud cluster, or other cloud computing platform 86, hereinafter referred to as "cloud device" 86), respectively. As used herein, "module" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry. The hardware processing circuitry may include any or some combinations of a microprocessor, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuitry. In some examples, partitioning module 10 may be hosted on cloud computing platform 86, which provides the edge-cloud collaboration scheme as a service. In some examples, partitioning module 10 may be hosted on a computing platform that is part of a proprietary enterprise network.

[0042] exist Figure 2In the example, the deep learning model provided as input to the partitioning module 10 is the trained DNN 11, and the first and second deep learning models generated by the partitioning module 10 are an edge DNN 30 for deployment on the target edge device 88 and a cloud DNN 40 for deployment on the target cloud device 86. As will be explained in more detail below, the partitioning module 10 is used to partition the trained DNN 11 into edge DNN 30 and cloud DNN 40 based on a set of constraints 20 received by the partitioning module 10 as input. These constraints may include, for example: (i) edge device constraints 22: defining one or more parameters defining the computing power (e.g., memory size, CPU bit processing size) of the target edge device 88 that will be used to implement the edge DNN 30. These parameters may include explicit parameters, such as memory size, bit width supported by the processor, etc.; (ii) Cloud device constraint 24: defines one or more parameters that define the computing power of the target cloud device 86 to implement the cloud DNN 40; (iii) Error constraint 26: specifies one or more parameters that specify the inference error tolerance threshold; (iv) Network constraint 28: specifies one or more parameters that specify information about the communication network link existing between the cloud device 86 and the edge device 88, including, for example: one or more network types (e.g., Bluetooth, 3G-5G cellular link, wireless local area network (WLAN) link attributes); network latency, power and / or noise ratio measurements; and / or link transmission metering costs.

[0043] DNN 11 is a DNN model trained for a specific inference task. DNN 11 comprises multiple network layers, each performing a specific computational operation to achieve a corresponding function. For example, layers can conform to a known NN layer structure, including: (i) fully connected layers, where a set of multiplication and summation functions are applied to all input values ​​included in the input feature map to generate an output feature map of the output value; (ii) convolutional layers, where multiplication and summation functions are applied via convolution to a subset of input values ​​included in the input feature map to generate an output feature map of the output value; (iii) batch normalization layers, applying a normalization function to a batch of multiple input feature maps to generate a corresponding normalized output feature map; and (iv) activation layers, applying a non-linear transformation function (e.g., the ReLU function or the sigmoid function) to each value included in the input feature map to generate activation values. The layers are: (v) an output feature map (also called an activation map or activation); (vi) a multiplication layer that multiplies two input feature maps to produce a single output feature map; (vii) a summation layer that adds two input feature maps to produce a single output feature map; (vii) a linear layer that applies a defined linear function to the input feature maps to produce the output feature map; (viii) a pooling layer that performs an aggregation function to combine the values ​​in the input feature maps into a smaller number of values ​​in the output feature map; (ix) an input layer of the DNN that organizes the input feature maps into the DNN to feed into the intermediate set of the hidden layers; and (x) an output layer that organizes the feature maps output from the intermediate set of the hidden layers into the output feature map of the DNN. In some examples, layers can be organized into computational blocks; for example, convolutional layers, batch normalization layers, and activation layers can collectively provide convolutional blocks.

[0044] The operations of at least some layers of a trained DNN 11 can be configured using a set of learned weight parameters (hereinafter referred to as weights). For example, multiplication operations in the multiplication and summation functions of fully connected and convolutional layers can be used to apply matrix multiplication to determine the dot product of the input feature map (or a subset of the input feature map) with a set of weights. In this invention, a feature map refers to an ordered data structure of values, where the position of a value within the data structure is meaningful. Tensors such as vectors and matrices are examples of possible feature map formats.

[0045] As is known in the art, a DNN can be represented as a complex directed acyclic graph (DAG) consisting of a set of 14 nodes connected by directed edges 16. An example of DAG 62 is shown in... Figure 3This is illustrated in more detail below. Each node 14 represents a corresponding layer in the DNN and has a corresponding node type corresponding to the type of layer it represents. For example, layer types can be represented as: C layer, representing a convolutional network layer; P layer, representing a point convolutional network layer; D layer, representing a depthwise convolutional network layer; L layer, representing other linear network layers; G layer, representing a global pooling network layer; BN layer, representing a batch normalization network layer; A layer, representing an activation layer (which may include activation types, e.g., R layer representing a ReLU activation layer, σ node representing a sigmoid activation layer); + layer, representing a summation layer; X layer, representing a multiplication layer; Input layer, representing the input layer; Output layer, representing the output layer. Directed edges 16 represent the directed flow of feature maps through the DNN.

[0046] refer to Figure 2 As will be explained in more detail below, the partitioning module 10 performs multiple operations to generate edge DNN 30 and cloud DNN 40, including preprocessing operation 44 to generate a list of potential partitioning schemes, selection operation 46 to generate a final, optimized partitioning scheme, and packaging and deployment operation 48 to package and deploy the resulting edge DNN 30 and cloud DNN 40.

[0047] In an exemplary embodiment, dividing the trained DNN 11 into edge DNN 30 and cloud DNN 40 is regarded as a nonlinear integer optimization problem. The goal is to minimize the overall latency given edge device constraints 22 and user-given error constraints 26 by jointly optimizing the partition points used to partition DNN 11 and the bit widths of the weight parameters for the layers included in edge DNN 30, as well as the input and output tensors.

[0048] The following variable names will be used to explain the operation of partitioning module 10.

[0049] N represents the total number of layers in the optimized trained DNN 12 (the optimized DNN 12 is an optimized version of the trained DNN 11, which will be described in more detail below), n represents the number of layers included in the edge DNN 30, and (N–n) represents the number of layers included in the cloud DNN 40.

[0050] s w The vector s represents the magnitude of the weights used to configure the layers of the trained DNN 12. w Each value s in w i This represents the number of weights in the i-th layer of the trained DNN 12. a The vector s represents the size of the output feature map generated by the layers of DNN 12. a Each value s in a iThis represents the number of feature values ​​included in the feature map generated by the i-th layer of the trained DNN 12. In an exemplary embodiment, the weights and the number of feature values ​​of each layer remain constant throughout the partitioning process; that is, for the corresponding layer i that is ultimately implemented in the edge DNN 30 or cloud DNN 40, the number s of the weights of a specific layer i in the trained DNN 12 is... w i and the number of activations s a i It will remain the same.

[0051] b w Vector b represents the bit width used to configure the weights of the DNN layer. w Each value b in w i This represents the bit width (e.g., number of bits) of the weights in the i-th layer of the DNN. a The vector b represents the bit width of the output feature values ​​that are the output of a layer in a DNN. a Each value b in a i This represents the bit width (i.e., the number of bits) of the feature values ​​used in the i-th layer of the DNN. For example, the bit width can be 128, 64, 32, 16, 8, 4, 2, and 1 bits, with each decrease in bit width corresponding to a decrease in precision. In an exemplary embodiment, the bit width of the layer weights and the output feature map is set based on the capabilities of the device hosting the specific DNN layer.

[0052] L edge (.) and L cloud (.) represent the latency functions for edge device 88 and cloud device 86, respectively. In s w and s a When L is fixed, edge and L cloud It is a function of the weight bit width and the feature map value bit width.

[0053] The latency of executing the i-th layer of the DNN on edge device 88 and cloud device 86 can be expressed as follows: and

[0054] L tr (.) represents a function that measures the latency of sending data from edge device 88 to cloud device 86. This represents the transmission delay of the i-th layer.

[0055] w i (.) and a i(.) represent the weight tensor and output feature map for a given weight bit width and eigenvalue bit width, respectively, at layer i. Using the mean squared error function MSE(.,.), the quantization error of the weights at layer i can be expressed as: in, This indicates the bit width used in the trained DNN 12. The quantization error of the output feature map at layer i, representing the bit width of the target DNN, can be expressed as: in, This indicates the bit width used in the trained DNN 12. This represents the bit width of the target DNN. MSE is a known quantization error metric; however, other distance metrics, such as cross-entropy or KL divergence, can also be used to quantize quantization error.

[0056] The objective function for partitioning module 10 can be expressed as follows based on the aforementioned delay function: If the trained DNN 12 is partitioned at layer n (i.e., the first n layers are assigned to edge DNN 30, and the remaining N–n layers are assigned to cloud DNN 40), the objective function can be defined by adding all the delays of the corresponding layers of edge DNN 30 and cloud DNN 40 and the intervention transmission delay between DNN 30 and DNN 40, expressed as follows:

[0057]

[0058] In Equation 1, tuple (b) w ,b a (b, n) represents the DNN partitioning scheme, where n is the number of layers allocated to the edge NNNs, and b w It is the bit-width vector of the weights of all layers, b a It is the bit-width vector of the output feature maps of all layers.

[0059] When n = 0, all layers of the trained DNN 12 are assigned to the cloud DNN 40 for execution by the cloud device 86. Typically, the training device used to train DNN 11 and the cloud device 86 will have comparable computational resources. Therefore, in the exemplary embodiment, the raw bit width trained from DNN 12 is also used in the cloud DNN 40, thereby avoiding any quantization errors in the layers included in the cloud DNN 40. Therefore, latency... It is a constant, where i = 1, ..., ... . Furthermore, due to transmission delay... This represents the time cost of sending the raw input to the cloud device 86, therefore it is reasonable to assume that under given network conditions... It is a constant. Therefore, it is only applicable to cloud solutions. The objective function is also a constant.

[0060] Therefore, the objective function can be expressed as:

[0061]

[0062] Removing constants Then, the objective function for partitioning module 10 can be expressed as:

[0063]

[0064] In an exemplary embodiment, constraint 20, specifically edge device constraint 22 (e.g., memory constraint) and user-specified error constraint 26, are also factors defining the formula for the nonlinear integer optimization problem of partitioning module 10. Regarding memory constraints, in a typical device hardware configuration, "read-only" memory stores parameters (weights), and "read-write" memory stores feature maps. The weighted memory cost on edge device 88 can be expressed as… Unlike weights, input and output feature maps only need to be stored in memory for a given time. Therefore, the read / write memory required for feature map storage is equal to the maximum working set size of the activation layer for a given time. In the case of a simple DNN chain (i.e., layers stacked one on top of another), the maximum working set of activation layer feature maps can be computed as follows: However, for complex DNN DAGs, the working set needs to be determined based on the DNN DAG. As an example, Figure 3 An example of an illustrative DAG 64 generated for the original trained DNN 12 is shown. When processing layer L4 (depthwise convolutional layer D), the output feature maps of layers L2 (convolutional layer C) and L3 (pointwise convolutional layer P) need to be stored in memory. Although processing layer L4 does not require the output feature map of layer L2, it is necessary to store the output feature map of layer L2 for use by future layers (e.g., layer 11, i.e., the summation+ layer). Assuming that the available memory size of the edge device 88 used to execute the edge DNN 30 is M, the memory constraint can be expressed as:

[0065]

[0066] Regarding error constraints, to maintain the accuracy of the combined edge DNN 30 and cloud DNN 40, the total quantization error is constrained by a user-given error tolerance threshold E. When the original bit width of DNN 12 is also used in layers of cloud DNN 40, the quantization error determination can be based solely on the summation of errors occurring in edge DNN 30, expressed as:

[0067]

[0068] Therefore, in an exemplary embodiment, the partitioning module 10 is used to select a DNN partitioning scheme based on the objective function (2) and memory constraints (3) and error constraints (4), which can be summarized as a problem (5) having a delay minimization component (5a), a memory constraint component (5b), and an error constraint component (5c):

[0069] DNN partitioning problem (5):

[0070]

[0071]

[0072]

[0073] in, This is the set of candidate bitwidths for weights and feature maps. In an exemplary embodiment, the edge device 88 has a fixed set of candidate bitwidths. For example, the candidate bit width set of edge device 88 It can be set to ={2,4,6,8}.

[0074] In the example, the delay function (e.g., L) edge (.), L cloud (.)) is not an explicitly defined function. Instead, simulator functions (as known in the art) can be used by partitioning module 10 to obtain the delay value. Since the delay function is not explicitly defined, and the error function (e.g., Since n is nonlinear, problem (5) is a nonlinear integer optimization function and a nondeterministic polynomial-time hard (NP-hard) problem. However, problem (5) does have a known feasible solution, i.e., n=0, which means executing all layers of DNN 12 on a cloud device 86.

[0075] As stated above, problem (5) is constrained by the error tolerance threshold E given by the user. In practice, it might be easier to handle if the user provides an accuracy degradation tolerance threshold A instead of the error tolerance threshold E. Furthermore, calculating the corresponding error tolerance threshold E for a given degradation tolerance threshold A is still difficult to handle. As will be explained in more detail below, in an exemplary embodiment, the partitioning module 10 can be used to enable the user to provide an accuracy degradation tolerance threshold A and also solve the difficult problem.

[0076] Furthermore, since problem (5) is an NP-hard problem, in the exemplary embodiment, the partitioning module 10 is used to apply a multi-step search method to find a list of potential solutions that satisfy the memory constraint component (5b), and then selects from the list of potential solutions the solution that minimizes the delay component (5a) and satisfies the error constraint component (5c).

[0077] In the example shown, partitioning module 10 includes operation 44 for generating a list of potential solutions by determining the size (e.g., quantity) of data that needs to be sent from that layer to one or more subsequent layers for each layer. Next, for each partition point (i.e., for each possible value of n), solve two sets of optimization problems to generate a list of feasible solutions that satisfy the memory constraint component (5b).

[0078] In this regard, we will refer to Figure 3 The example illustrates a method for generating a list of potential solutions. The three-step operation 44. Figure 3 The input is an unoptimized trained DNN 11, denoted as DAG 62, where layers are shown as nodes 14, and relationships between layers are indicated by directed edges 16. A set of initial graph optimization actions 50 is performed to optimize the unoptimized trained DNN 11. Specifically, as is known in the art, actions such as batch norm folding and activation fusion can be performed on the trained DNN to incorporate the functionality of batch norm layers and activation layers into the preceding layers, resulting in an optimized DAG 63 for inference purposes. Figure 3 As shown, the optimized DAG 63 (representing the optimized trained DNN 12 for inference purposes) does not include discrete batch normalization and ReLU activation layers.

[0079] Then, a set of weight allocation actions 52 are performed to generate a weighted DAG 64, which includes weights assigned to each edge 16. Specifically, if the partition point n is located on the edge, the weight assigned to each edge represents the lowest possible transmission cost t of that edge. i It should be noted that some nodes (e.g., D-layer nodes representing layer L4) will have multiple associated edges, each edge being assigned a transmission cost t. i The lowest transmission cost is selected as the edge weight. The potential partition point n should satisfy the memory constraint of having the lowest bit width allocation. Among them, b min It is the minimum bit width constrained by the edge device 88. The minimum transmission cost t at the edge. i For b min s a The minimum transmission cost T of partition point n nis the sum of all individual edge transmission costs t for the unique edge cut at the partitioning point n. For example, as shown in the weighted DAG 64, at the partitioning point n = 4, the transmission cost T4 is t2 + t4 (it should be noted that although the two edges of layer L4 are cut, the data of the two edges is the same, so it only needs to be transmitted once); at the partitioning point n = 9, the transmission cost T9 is t2 + t9; at the partitioning point n = 11, the transmission cost T i is t 11 is t 11 .

[0080] Then, for the weighted DAG 64, the sorting and selection action 54 is performed. Specifically, the weighted DAG 64 is sorted in topological order based on the transmission cost, a list of possible partitioning points is identified, and an output 65 including a list of potential partitioning point schemes is generated. In an exemplary embodiment, in order to identify possible partitioning points, it is assumed that the original data transmission cost T0 is a constant, such that the potential partitioning point n should have a transmission cost T n < T0 (i.e., ). This assumption effectively assumes that there is a better scheme than sending all the original data to the cloud device 86 and performing the entire trained DNN 12 on the cloud device 86. Accordingly, the list of potential partitioning points can be determined as:

[0081]

[0082] In summary, the list of potential partitioning points will include all potential partitioning points whose transmission cost is less than the original transmission cost T0, where the transmission cost of each edge is constrained by the minimum bit-width allocation of the edge device 88. In this regard, the list of potential partitioning points provides a set of filtered partitioning points that can satisfy the memory constraint component (5b) of problem (5). Referring again to Figure 3 , then the list of potential partitioning points is provided to operation 46, and operation 46 performs a set of actions to solve a set of optimization problems, thereby determining a list of feasible solutions S. Operation 46 is used to identify all feasible solutions that satisfy the constraints of problem (5) for each potential partitioning point . In an exemplary embodiment, the list of feasible solutions S is presented as a list of tuples (b w , b a , n).

[0083] As described above, explicitly setting the error tolerance threshold E is intractable. Therefore, in order to obtain a feasible solution to problem (5), operation 46 is used to determine the partitioning point Which split point in the graph will cause an error in quantizing the weights and feature maps, and this error is within a user-specified precision degradation threshold A? In this respect, the optimization problem (7) can be expressed as:

[0084]

[0085]

[0086] A partition scheme for the optimization problem (7) that keeps the quantization error within the accuracy degradation threshold A can be selected and included in the list of feasible solutions S. For a given partition point p, the search space within the optimization problem (7) is exponential, i.e. To reduce the search space, problem (7) was split into two problems (8) and (9):

[0087]

[0088]

[0089] Among them, M wgt and M act These are the memory budgets for the weights and feature maps, respectively, and M... wgt +M act ≤M. Different methods can be applied to solve problems (8) and (9), including, for example, the Lagrange method proposed below: [Y. Shoham and A. Gersho. 1988. Efficient bit allocation for an arbitrary set of quantizers. IEEE Trans. Acoustics, Speech, and Signal Processing 36 (1988)].

[0090] To find the corresponding memory budget M wgt and M act The feasible candidate bit-width pairs can be used for memory budget M wgt and M act Perform a two-dimensional grid search. M wgt and M act Candidates are selected by using the candidate bit-width set. The bit-width vector b is uniformly allocated in the middle. w and b a The result is such that, for a given n, the maximum number of feasible bit-width pairs is By splitting problem (7) into two problems (8) and (9), problem (7) represents The search space was significantly reduced to at most

[0091] In at least some applications, the properties of the aforementioned discrete nonconvex and nonlinear optimization problems make it impossible to achieve an exact solution to problem (5). However, the aforementioned multipart problem solution method guarantees... Where (0,0,0) is a cloud-only solution. It is an edge-only solution.

[0092] Operations 44 and 46 are in Figure 4 The pseudocode 400 indicates this.

[0093] refer to Figure 2 After generating a feasible solution tuple (b) w ,b a Following the list S, selection, configuration, and deployment operations can be performed. For example, a partitioning scheme that minimizes latency and satisfies the accuracy degradation threshold constraint can be selected from the list as the implementation scheme.

[0094] After selecting an implementation scheme, a set of configuration actions can be applied to generate: (i) defining edge DNN configuration information 33 for edge DNN 30 (corresponding to the first n layers of the optimized trained DNN 12); and (ii) defining cloud DNN configuration information 34 for cloud DNN 40 (corresponding to the last N–n layers of the optimized trained DNN 12). In an exemplary embodiment, the edge DNN configuration information 33 and cloud DNN configuration information 34 can take the form of corresponding DAGs, which include the information required for edge device 88 to implement edge DNN 30 and cloud device 86 to implement cloud DNN 40. In the example, based on the selected bit-width vector b w The weights included in the edge DNN configuration information 33 will be quantized versions of the weights of the corresponding layers in the optimized trained DNN12. Similarly, the edge DNN configuration information 34 will include the bit-width vector b that implements the quantization of the selected feature map. a Required information. In at least some examples, the cloud DNN configuration information 34 will include information specifying the same bit width as the last N–n layers used in the optimized trained DNN 12. However, the weight and feature map bit widths of the cloud DNN 40 may also differ from those used in the optimized trained DNN 12.

[0095] In an exemplary embodiment, a packing interface function 36 can be added to the edge DNN 30. This packing interface function 36 organizes and packs the feature map 39 output by the last layer of the edge DNN 30 so that the feature map 39 can be efficiently sent to the cloud device 86 via the network 84. Similarly, a corresponding unpacking interface function 38 can be added to the cloud DNN 40. This unpacking interface function 38 unpacks and organizes the received feature map 39 and provides the feature map 39 to the first layer of the cloud DNN 40. If necessary, other interface functions can be included to enable the inference results generated by the cloud device 86 to be sent back to the edge device 88.

[0096] In an exemplary embodiment, the trained DNN 12 may be a DNN used to perform inference on an input image.

[0097] The partitioning module 10 is used to treat the partitioning point and bit width selection (i.e., quantization precision) as optimizations, where the goal is to identify the partitioning and bit width allocation of weights and activations to reduce the overall latency of the resulting partitioned DNN (i.e., a combination of edge DNN and cloud DNN) without sacrificing accuracy. This approach offers several advantages over existing strategies, such as being architecturally secure, deterministic, and flexible. The proposed method provides a range of options in the precision-latency tradeoff, which can be selected based on the requirements of the target application. The bit width used in different network layers can vary, allowing for mixed-precision quantization through the edge DNN 30. For example, an 8-bit integer bit width can be assigned to the weights and feature values ​​of a first set of one or more layers in the edge DNN 30, followed by a second set of one or more layers, and then a 4-bit integer bit width is assigned to the weights and feature values ​​of the second set of one or more layers in the edge DNN 30, where a 16-bit floating-point bit width is used for the layers in the cloud DNN 40.

[0098] Although the partitioning module 10 is described in the context of edge device 88 and cloud device 86 within an internet environment, the partitioning module 10 can be applied to other environments for partitioning deep learning models used to perform inference tasks across asymmetric computing platforms. For example, in alternative environments, edge device 88 could take the form of a small, simple edge device (e.g., smart glasses, fitness tracker), cloud device 86 could take the form of a relatively more powerful device (e.g., smartphone), and network 84 could take the form of Bluetooth. TM In the form of a link.

[0099] See Figures 1 to 3The performance partitioning module 10 provided in the example of the present invention can be summarized as follows. The partitioning module 10 is used to partition a trained neural network (e.g., an optimized DNN 12) into a first neural network (e.g., an edge DNN 30) for execution on a first device (e.g., an edge device 88) and a second neural network (e.g., a cloud DNN 40) for execution on a second device (e.g., a cloud device 86). The partitioning module 10 identifies a first set from the trained neural network for inclusion in one or more neural network layers in the first neural network, and a second set from the trained neural network for inclusion in one or more neural network layers in the second neural network. The partitioning module 10 then assigns weight bit widths to the weights of the first set of one or more neural network layers and assigns feature value bit widths to the feature maps generated by the first set of one or more neural network layers. Identification and assignment are performed to optimize the overall latency of the following operations within accuracy constraints: executing the first neural network on the first device to generate a feature map output based on input data, sending the feature map output from the first device to the second device, and executing the second neural network on the second device to generate an inference output based on the feature map output of the first device.

[0100] Figure 5 This is a block diagram of an exemplary simplified processing unit 100 provided in the examples disclosed herein. This processing unit 100 may be part of a system or device implementing the partitioning module 10, or as an edge device 88 implementing an edge DNN 30, or as a cloud device 86 implementing a cloud DNN 40. Other processing units suitable for implementing the embodiments described herein may be used, and these units may include components different from those discussed below. Although Figure 5 A single instance of each component is shown, but multiple instances of each component may exist in the processing unit 100.

[0101] Processing unit 100 may include one or more processing devices 102, such as a processor, microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or a combination thereof. One or more processing devices 102 may also include other processing units (e.g., neural processing unit (NPU), tensor processing unit (TPU), and / or graphics processing unit (GPU)).

[0102] Figure 5Optional components are shown in dashed lines. Processing unit 100 may also include one or more optional input / output (I / O) interfaces 104 that can support connection to one or more optional input devices 114 and / or optional output devices 116. In the example shown, one or more input devices 114 (e.g., keyboard, mouse, microphone, touchscreen, and / or keypad) and one or more output devices 116 (e.g., display, speaker, and / or printer) are shown as optional and external to processing unit 100. In other examples, one or more of the one or more input devices 114 and / or one or more output devices 116 may include components of processing unit 100. In other examples, there may be no one or more input devices 114 and one or more output devices 116, in which case one or more I / O interfaces 104 may not be required.

[0103] The processing unit 100 may include one or more optional network interfaces 106 for wired communication (e.g., Ethernet cable) or wireless communication (e.g., one or more antennas) with a network (e.g., intranet, Internet, P2P network, WAN and / or LAN).

[0104] The processing unit 100 may further include one or more storage units 108, which may include mass storage units such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives. The processing unit 100 may include one or more memories 110, which may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The one or more non-transient memories 110 may store instructions for execution by one or more processing devices 102 to implement the neural networks, equations, and algorithms described in this invention, quantize and normalize data, and approximate one or more nonlinear functions of activation functions. The one or more memories 110 may include other software instructions, such as implementing operating systems and other applications / functions.

[0105] In some other examples, one or more datasets and / or modules may be provided by external memory (e.g., an external driver that communicates with the processing unit 100 via wired or wireless communication) or by transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage devices.

[0106] A bus 112 may exist to provide communication between components of the processing unit 100, including one or more processing devices 102, one or more optional I / O interfaces 104, one or more optional network interfaces 106, one or more storage units 108, and / or one or more memories 110. The bus 112 may be any suitable bus architecture, including, for example, a memory bus, a peripheral bus, or a video bus.

[0107] Figure 6 This is a block diagram of an exemplary hardware structure of an exemplary NN processor 200 for implementing a processing device 102 (e.g., a cloud DNN 40 or an edge DNN 30) according to some exemplary embodiments of the present invention. The NN processor 200 may be disposed on an integrated circuit (also known as a computer chip). All algorithms for the layers and neurons of the NN, including piecewise linear approximations of nonlinear functions, as well as data quantization and normalization, can be implemented in the NN processor 200.

[0108] One or more processing devices 102 Figure 1 The processor 200 may include another processor 211 in combination with the NN processor 200. The NN processor 200 can be any processor suitable for NN computation, such as a neural processing unit (NPU), tensor processing unit (TPU), graphics processing unit (GPU), etc. Taking an NPU as an example, the NPU can be installed as a coprocessor on the processor 211, with the processor 211 assigning tasks to the NPU. The core of the NPU is the arithmetic circuitry 203. The controller 204 controls the arithmetic circuitry 203 to retrieve matrix data from the memory (201 and 202) and perform multiplication and addition operations.

[0109] In some implementations, the arithmetic circuit 203 internally includes multiple processing units (process engines, PEs). In some implementations, the arithmetic circuit 203 is a two-dimensional pulsating array. Furthermore, the arithmetic circuit 203 can be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 203 is a general-purpose matrix processor.

[0110] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 203 retrieves the weight data of matrix B from the weight memory 202 and caches this data in each PE of the arithmetic circuit 203. The arithmetic circuit 203 retrieves the input data of matrix A from the input memory 201 and performs matrix operations based on the input data of matrix A and the weight data of matrix B. The obtained partial or final matrix result is stored in the accumulator 208.

[0111] Unified memory 206 is used to store input and output data. Weight data is moved directly to weight memory 202 using memory cell access controller 205 (direct memory access controller, DMAC). Input data is also moved to unified memory 206 using DMAC.

[0112] The bus interface unit (BIU) 210 is used to enable interaction between the DMAC and the instruction fetch memory 209 (instruction fetch buffer). The bus interface unit 210 is also used to enable the instruction fetch memory 209 to fetch instructions from the memory 110, and to enable the memory access controller 205 to fetch the source data of the input matrix A or the weight matrix B from the memory 110.

[0113] DMAC is mainly used to move input data from memory 110 to unified memory 206 at double data rate (DDR), or to weighted data to weighted memory 202, or to input data to input memory 201.

[0114] Vector computation unit 207 includes multiple computation processing units. If necessary, vector computation unit 207 performs further processing on the output of computation circuit 203, such as vector multiplication, vector addition, exponential operation, logarithmic operation, or magnitude comparison. Vector computation unit 207 is primarily used for computation at neurons or layers (described below) of a neural network. Specifically, vector computation unit 207 can perform computation, quantization, or normalization. For example, vector computation unit 207 can apply a nonlinear or piecewise linear function of the activation function to the output matrix generated by computation circuit 203, such as a vector of accumulated values, to generate the output value of each neuron in the next NN layer.

[0115] In some implementations, the vector computation unit 207 stores the processed vector in the unified memory 206. The instruction fetch memory 209 (instruction fetch buffer), connected to the controller 204, is used to store the instructions used by the controller 204.

[0116] Unified memory 206, input memory 201, weighted memory 202, and instruction fetch memory 209 are all on-chip memories. Data memory 110 is independent of the NPU hardware architecture. (See reference) Figure 7 Now we will describe other examples of dividing a fully trained neural network (NN) into multiple partitions that can be executed on different computing platforms.

[0117] refer to Figure 7 Other examples of dividing a fully trained neural network (NN) into multiple partitions that can be executed on different computing platforms will now be described. The variable names and symbols in equations (10) through (19) may be given different meanings and terms than those used above for similar components in the following parts of the invention.

[0118] In the example, training and inference are performed using the desired bit width (also known as bit depth) of the weights and feature maps so that the behavior of the NN remains unchanged. In the example, the NN partitions are arbitrarily chosen to find the optimal balance between the workload (the computer instructions involved in executing the deep learning model) performed on edge and cloud devices and the amount of data transferred between edge and cloud devices.

[0119] More specifically, the workload-intensive portion of the neural network (NN) can be included in NN partitions executed on cloud devices to achieve lower overall latency. For example, a large floating-point NN 701 already trained using training server 702 can be partitioned into a small, low-bit-depth NN 705 for deployment on a lower-power computing device (e.g., edge device 704) and a larger floating-point NN 707 for deployment on a higher-power computing device (e.g., cloud server 706). Features (e.g., feature maps) generated by edge NN 705 based on input data are sent via network 710 to cloud server 706 for further inference processing by cloud NN 701 to generate output labels. The different bit-depth allocation can be used to account for the difference in computing resources between edge device 704 and cloud server 706. This framework, implemented by partitioning module 700, is suitable for both multi-task and single-task models, can be applied to any model architecture, and can use mixed precision. For example, the NN partition allocated to edge device 704 (edge ​​NN 705) can be stored / executed in a lower bit depth (e.g., int8 or int4) instead of using 32-bit floating-point weights / operations for the entire NN inference. Furthermore, it supports devices / chips that can only run int8 (or lower) and have low memory footprints. In the exemplary embodiment, training is end-to-end. Therefore, in the case of a cascaded model, multiple iterations of data collection, cleaning, labeling, and training are not required. Only the final output labels are sufficient to train and end-to-end models. Furthermore, compared to cascaded models, the intermediate parts of the end-to-end model are trained to help optimize the overall loss. This may improve overall accuracy.

[0120] For example, consider license plate recognition. Traditional methods use two-stage training, where a detector neural network is trained to learn a model for detecting license plates in an image, and a recognizer neural network is trained to learn a model for performing recognition on the license plates detected by the detector neural network. In this invention, a single model can perform both license plate detection and recognition simultaneously, and the detection network is learned in a way that maximizes recognition accuracy. The neural network in this method can also have mixed precision weights and activations to provide efficient inference at the edge and in the cloud. This method is secure because it does not directly send the raw data. Intermediate features cannot be recovered to the original data. The data transmission volume is much smaller than the original data size because the feature information is rich and concise. This is a deterministic method. After training the model, the separation and edge-cloud workload distribution remain unchanged. This method is applicable to many applications, such as models for smartphones, surveillance cameras, and IoT devices. This application can be used for computer vision, speech recognition, NLP, and basically anywhere neural networks can be used at the edge.

[0121] In one exemplary embodiment, end-to-end mixed-precision training is performed on training server 702. For example, a portion of NN 701 (e.g., a first subset of NN layers) is trained using 8-bit (integer) bit depth of weights and features, and another portion of NN 701 (e.g., a second subset of NN layers) is trained using 32-bit (floating-point) bit depth of weights and features. NN 701 is then partitioned such that the smaller bit depth training portion is implemented as an edge NN 705, and the larger bit depth training portion is implemented as a cloud NN 707. This allows the NN workload to be partitioned between edge device 704 and cloud server 706.

[0122] exist Figure 8 In another example shown, during end-to-end mixed-precision training, the first part of NN 701 (e.g., the first subset of NN layers) is trained with an 8-bit (integer) depth of weights and features, the second subset of NN 701 (e.g., the second subset of NN layers) is trained with a 4-bit (integer) depth of weights and features, and the third part of NN 701 (e.g., the third subset of NN layers) is trained with a 32-bit (floating-point) depth of weights and features. NN 701 is then partitioned such that the first and second parts (8-bit and 4-bit parts) are assigned to the edge NN 705, and the third part (32 bits) is assigned to the cloud NN 707. The 4-bit features result in a lower data transfer rate.

[0123] To identify the partitioning and bit width allocation values ​​of a given neural network 701, a computer program runs offline (only once). The program takes as input the characteristics of the edge device 705 (memory, CPU, etc.) and the neural network 701, and outputs the partitioning and bit width.

[0124] In neural network 701, there is L total Each layer (L) total =L+L cloud In the case of ), the first L layers of the neural network 701 are deployed as edge network 705 on the edge device 704 (e.g., including the L layers of the neural network 701). total The instructions for the software program of each layer are stored in the memory of the edge device, and the instructions are executed by the processor of the edge device 704, and the remaining layers of the neural network 701 (L... cloud Layers) are deployed as cloud NN 707 on cloud computing platforms (e.g., including L neural networks). cloud The instructions for the layer's software program are stored in the memory of one or more virtual machines instantiated by a cloud computing platform (e.g., a cloud server 706), and the instructions are executed by the virtual machine's processor. In this case, L=0 indicates that the entire model runs on the cloud, L cloud=0 indicates that the model runs on an edge device. Since the portion running in the cloud will be hosted on a GPU, it runs at a high bit width, such as 16-bit floating point (FP) or 32-bit FP. In this setup, the goal is to determine a reasonable value for L and an appropriate bit width for each layer l = 1, 2, ..., L, such that the overall latency is lower than the two extreme cases: (1) running entirely at the edge (L cloud =0, if it fits the device memory), or (2) transfer to the cloud and then execute on the cloud (L=0).

[0125] In cases where the model cannot fully run on edge devices 704 (e.g., unsuitable or too slow), Figure 7 The purpose of this system is to provide a solution that satisfies the following:

[0126]

[0127] in, and Let represent the total latency of the cloud and the proposed method, respectively. If the model is suitable for edge devices but has higher latency than the cloud, the objective of (10) remains valid. In cases where edge latency is lower than cloud latency, the scheme in (10) is found to produce lower latency than the edge; otherwise, inference on the edge is assumed. Nevertheless, (10) can be rewritten as:

[0128]

[0129] in, It is a bit width of B i The delay of layer i, This is the time required to send the input to the cloud. It is a bit width of B L The transmission delay of layer L is a characteristic. It should be noted that it is reasonable to assume that the cloud model operates with a 16-bit FP, but this can also be changed to a 32-bit FP. (11) can be simplified to:

[0130]

[0131] Then, the overall optimization problem can be formulated as:

[0132]

[0133] in, and These are the weights and activation bit width values ​​assigned to layer i. and It is the size of the weights and activations, M totalThis represents the total available memory on the edge device. The constraint in (13) ensures that the first L layers running at the edge will not exceed the total available device memory. It should be noted that in the hardware, "read-only" memory stores parameters (weights), and "read-write" memory stores activations (when they change according to input data). Due to the reuse of "read-write" memory, the activation memory slots are reused, but the weights do accumulate in memory. Therefore, the memory required for the maximum activation layer is considered in (13). Therefore, This is the maximum memory required for activation.

[0134] For a fixed value L, and This becomes a constant in (13). Then, the optimization becomes minimizing the feature transfer cost by running the first L layers at the edge, i.e. The solution with the lowest latency is usually the one with a lower bit width. However, a low bit width increases the output quantization error, which in turn reduces the accuracy of the quantized model. This means that only solutions that provide sufficiently low output quantization error are meaningful. This has always been an implicit constraint, because the goal of post-training quantization is to gain speedup without sacrificing accuracy. Therefore, for L layers running at the edges, the latency minimization problem can also be viewed as minimizing the budget of output quantization error, subject to memory and bit allocation constraints.

[0135] First, we will describe the case where the value L is fixed, and then explain how this case is suitable. Figure 7 The overall scheme provided by the system. With the model running entirely on an edge device 704 (equivalent to a fixed value L), it has been empirically and theoretically proven that if the output quantization error is evaluated using mean squared error (MSE), the overall error is the sum of the weights and activations. In this formula, the output quantization error is defined as:

[0136]

[0137] in, and B represents the bit width assigned to the weights and activations of layer i. total is the average total bit width of the network, and D is the MSE output error (on the feature vector) generated by quantizing the weights or activations of the layer.

[0138] The exemplary embodiment is based on formula (14) for the case where L is fixed. However, instead of imposing a constraint on the sum of the bit widths of different layers, this paper discloses alternative, more feasible constraints on the total memory, which in turn depend on the bit width value.

[0139] In edge-cloud workload partitioning, a two-dimensional problem arises where both the bit width B and the partition L are unknown. This is a problem that is difficult to solve in a closed-form solution. Therefore, Figure 7 The system is designed to make the search space significantly smaller.

[0140] In an exemplary embodiment, a training server 702 (or other device) is used to first find a reasonable split point. For this purpose, for B... total The average bit width value in [2,4,6] determines all schemes of (15):

[0141]

[0142] To solve (15), the Lagrange multiplier is incorporated. Equation (16) gives the per-level bit allocation of the "activation". After finding all possible schemes for various partitions, these schemes are sorted in order of activation amount, as follows:

[0143]

[0144] The sorting is done in ascending order because the largest negative value is preferred. Large negative values ​​in (16) indicate lower activation levels in the corresponding layer, which in turn leads to faster data transfer. * Provide a reasonable partition and bit allocation for the activation of the first L layers. This allocation is reasonable, but not optimal, because (15) uses L total Instead of solving for L, simulations show that data transmission has a much greater impact on overall latency than layer execution.

[0145] Next, the bit width of the weights will be identified through the following solution:

[0146]

[0147] in, It is based on S in (16) * The constraints in (17) are the same as those in (13) in the calculation of the solution. For any λ≥0, the solution to the constrained problem in (17) is also the solution to the following unconstrained problems:

[0148]

[0149] (18) can be solved in the same way as (15), using the generalized Lagrange multiplier method to optimize resource allocation.

[0150] Figure 9 The pseudocode algorithm is summarized Figure 7 Recommended methods for system implementation. Figure 9 The second step of the algorithm involves finding the results in (15). Refinement of the solution. As mentioned above, the solution provided in (15) in the first iteration is suboptimal. A better solution can be obtained by solving the following problem.

[0151]

[0152] It should be noted that the constraints have now been changed to reflect the maximum memory available for activation (now known). Solving (19) may result in higher bit width values ​​for some layers in l = 1, 2, ..., L. This, in turn, implies lower MSE values ​​and higher precision, at the cost of a negligible increase in latency. Nevertheless, a simple but quick way to achieve a reasonable solution is to start increasing the bit width values ​​of the layers until their magnitude reaches just below

[0153] The proposed approach described above is, in principle, applicable to any neural network for any task. In other words, it provides a scheme for dividing a neural network into two parts to run on different platforms. A simple approach is to run the model entirely on one platform or on another. Another approach, if available, is to run a portion of the model on each platform. However, the latter is more likely to occur when edge devices have scarce computing resources (power, memory, or speed limitations). Examples include low-power embedded devices, smartwatches, smart glasses, hearing aids, etc. It should be noted that while specialized deep learning chips are entering the market, most existing cost-friendly consumer products largely fall within the feasible scenarios considered here.

[0154] An exemplary application of the invention will now be described. In license plate recognition, consider an on-chip camera mounted on a parking lot object (e.g., a gate) to authorize entry for vehicles with certain registered license plates. The input to the camera system is frames captured from a car, and the output should be the recognized license plate (as a string).

[0155] For edge devices, a realistic consumer camera based on the Hi3516E V200 SoC was chosen. This is an affordable HD IP camera widely used in home surveillance and can connect to the cloud. The chip uses an ARM Cortex-A7 and features low memory and storage requirements.

[0156] Figure 10 A block diagram of the proposed solution is shown. (For example...) Figure 10As shown, the system of this invention ensures that the camera chip of edge device 88 or 704 has sufficient workload and securely sends features (the only required data, with no additional data) to the cloud device in cloud 82 for accurate identification. In other words, edge-cloud workload separation causes edge device 88 or 704 to send features (instead of raw data), thereby protecting the privacy of user data. The hybrid precision separable model that divides the workload between edge devices 88, 704 and cloud 82 can provide high accuracy (because it can utilize a larger neural network with higher learning capacity than an edge-only scheme) and lower latency (because it pushes the heavy workload to the cloud 82 GPU).

[0157] This invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Selected features from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and it is understood that features suitable for such combinations are within the scope of this invention.

[0158] All values ​​and sub-ranges within the scope of disclosure are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, these systems, devices, and components may be modified to include additional or fewer such elements / components. For example, although any of the disclosed elements / components may be referred to as the singular, the embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

[0159] The units described as discrete parts may or may not be physically separate. The parts shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0160] Furthermore, the functional units in the exemplary embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0161] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, server, or network device) to execute all or part of the steps of the method described in the embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a universal serial bus (USB) flash drive, a removable hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0162] The above description is merely a specific implementation method and is not intended to limit the scope of protection. Any variations or substitutions that are readily conceived by those skilled in the art within the scope of the technology should be within the scope of protection. Therefore, the scope of protection should be determined by the scope of the claims.

Claims

1. A method for dividing a trained neural network into a first neural network for execution on a first device and a second neural network for execution on a second device, wherein the computing power of the first device is lower than that of the second device, characterized in that, The method includes: Identify a first set from the trained neural network for inclusion in one or more neural network layers in the first neural network, and identify a second set from the trained neural network for inclusion in one or more neural network layers in the second neural network; Assign weight bit widths to the weights of the first set of one or more neural network layers, and assign feature map bit widths to the feature maps generated by the first set of one or more neural network layers; The identification and assignment are performed to optimize the overall latency of the following operations within accuracy constraints: executing the first neural network on the first device to generate a feature map output based on input data, sending the feature map output from the first device to the second device, and executing the second neural network on the second device to generate an inference output based on the feature map output of the first device.

2. The method according to claim 1, characterized in that, The identification and the allocation include: A set of one or more feasible schemes within the accuracy constraint is selected from a plurality of potential partitioning schemes for dividing the trained neural network into a first set of one or more neural network layers and a second set of one or more neural network layers, wherein each feasible scheme identifies: (i) a partition point indicating a layer in the trained neural network included in the first set of one or more layers; (ii) a set of weight bit widths for configuring the weights of the first set of one or more neural network layers; and (iii) a set of feature map bit widths for the feature map generated by the first set of one or more neural network layers.

3. The method according to claim 2, characterized in that, The method includes selecting an implementation scheme from the set of one or more feasible schemes; generating first neural network configuration information defining the first neural network and second neural network configuration information defining the second neural network according to the implementation scheme; providing the first neural network configuration information to the first device and providing the first and second neural network configuration information to the second device.

4. The method according to claim 2 or 3, characterized in that, The selection is also based on the memory constraints of the first device.

5. The method according to claim 4, characterized in that, include: Before selecting the set of one or more feasible schemes, the plurality of potential partitioning schemes are determined based on identifying the transmission cost associated with different possible partitioning points, which is lower than the transmission cost associated with including all layers of the trained neural network in the second neural network.

6. The method according to claim 2 or 3, characterized in that, The selection includes: For each of the plurality of potential schemes with different weight bit widths and feature map bit widths, the quantization error of the combined performance of the first neural network and the second neural network is calculated, wherein the selection of the set of one or more feasible schemes is based on selecting weight bit widths and feature map bit widths that will keep the calculated quantization error within the accuracy constraints.

7. The method according to claim 6, characterized in that, The different weight bit widths and feature map bit widths of each of the plurality of potential schemes are uniformly selected from the set of possible weight bit widths and feature map bit widths, respectively.

8. The method according to any one of claims 1 to 3, characterized in that, The accuracy constraint includes a tolerance threshold for accuracy degradation defined for the combined performance of the first neural network and the second neural network relative to the performance of the trained neural network.

9. The method according to any one of claims 1 to 3, characterized in that, The first device has a lower memory capacity than the second device.

10. The method according to any one of claims 1 to 3, characterized in that, The first device is an edge device, and the second device is a cloud-based computing platform.

11. The method according to any one of claims 1 to 3, characterized in that, The trained neural network is an optimized trained neural network represented as a directed acyclic graph.

12. The method according to any one of claims 1 to 3, characterized in that, The first neural network is a mixed-precision network, comprising at least some layers, wherein the at least some layers have weight bit widths and feature map bit widths that are different from those of the other layers.

13. A computer system, characterized in that, The system includes one or more processing devices and one or more non-transient memories, the non-transient memories storing computer-implementable instructions for execution by the one or more processing devices, wherein executing the computer-implementable instructions causes the computer system to perform the method according to any one of claims 1 to 12.

14. A non-transient computer-readable medium, characterized in that, The storage computer implements instructions for causing a computer system to perform the method according to any one of claims 1 to 12.

Images