Execution of a deep learning model

Abstract

According to implementations of the present disclosure, a scheme of execution of a deep learning model is proposed. In the scheme, in response to a convolution operation of a convolution layer of the deep learning model being triggered, a plurality of partitioned convolution operations are sequentially executed in a trusted execution environment (TEE) of a computing device based on inputs of the convolution layer and a set of parameter values. The execution of a given partitioned convolution operation includes storing an input portion of the inputs to be processed by a parameter value subset of the given partitioned convolution operation in a protected storage area of the TEE, determining a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and the parameter value subset for the given partitioned convolution operation, the parameter value subset being represented as a matrix, and removing the input portion from the protected storage area. The result of the convolution operation is determined by merging the results of the plurality of partitioned convolution operations. Thereby, the model execution speed can be accelerated and the storage space efficiency can be improved in the TEE which has high security level but limited storage resources.

Description

BACKGROUND

[0001] Deep learning, as a branch of machine learning, has been continuously developed in recent years and is currently widely used in different fields and implementations. Deep learning models can be designed to implement a wide variety of tasks, including computer vision processing, speech recognition, natural language processing, etc. Such tasks can sometimes be tasks desired to be performed in various terminals, such as mobile phones, Internet of Things (IoT) applications. The execution of deep learning models, especially the execution of complex or more precisely configured large models, will require higher computing and storage resources. SUMMARY

[0002] According to implementations of the present disclosure, a scheme for execution of a deep learning model is proposed. In the scheme, in response to a convolution operation of a convolution layer of the deep learning model being triggered, a plurality of partitioned convolution operations are sequentially executed in a trusted execution environment (TEE) of a computing device based on inputs and a set of parameter values of the convolution layer. The execution of a given partitioned convolution operation includes storing an input portion of the inputs to be processed by a subset of parameter values of the given partitioned convolution operation in a protected storage area of the TEE, determining a result of the given partitioned convolution operation through a single matrix multiplication operation of the input portion and the subset of parameter values for the given partitioned convolution operation, the subset of parameter values being represented as a matrix, and removing the input portion from the protected storage area. The result of the convolution operation is determined by merging the results of the plurality of partitioned convolution operations. Thereby, the model execution speed can be accelerated and the storage space efficiency can be improved in the TEE which is high in security but limited in storage resources.

[0003] The summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. The summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 A block diagram illustrating an environment in which multiple implementations of the present disclosure can be implemented is shown;

[0005] Figure 2 A block diagram illustrating a computing environment implemented in a computing device according to one implementation of the present disclosure is shown;

[0006] Figure 3A An example of a direct convolution operation of a convolution layer is shown;

[0007] Figure 3B An example of a fast convolution based on convolution reduction is shown;

[0008] Figure 4 An example of a fast convolution based on partitioned convolution operations according to one implementation of the present disclosure is shown;

[0009] Figure 5 a block diagram illustrating a computing environment implemented in a computing device according to another implementation of the disclosure is shown;

[0010] Figure 6 an example of parallel pipelined threads for parameter loading, parameter validation, and model execution according to one implementation of the disclosure is shown; and

[0011] Figure 7 a flow diagram illustrating a process according to one implementation of the disclosure is shown.

[0012] In these drawings, like or similar elements are referred to with like or similar reference numerals. DETAILED DESCRIPTION

[0013] The present disclosure will now be discussed with respect to several example implementations. It should be appreciated that these implementations are discussed solely for the purposes of exemplifying the present disclosure and are not intended to limit the scope of the disclosure in any manner.

[0014] As used herein, the terms “includes,” “including,” “has,” “having,” “contains,” “containing,” “comprises,” “comprising,” “is” and “are” are to be construed as open-ended terms (i.e., including but not limited to) unless specifically indicated to the contrary. The term “based on” is to be construed as “based at least in part on.” The terms “one implementation” and “an implementation” are to be construed as “at least one implementation.” The term “another implementation” is to be construed as “at least one other implementation.” The terms “first,” “second,” and the like can refer to different or identical objects. Also, the terms “exemplary” and “example” are utilized herein to mean “serving as an example, instance, or illustration,” and not “preferred” or “advantageous over other implementations.” The detailed description includes specific details for the purpose of providing an understanding of the described techniques. These techniques, however, can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the concepts of the described techniques.

[0015] Machine learning is an artificial intelligence technique. Deep learning is a machine learning algorithm that processes input and provides a corresponding output by using multiple layers of processing units. Deep learning algorithms can be implemented by multi-layer neural networks (NNs). Such multi-layer neural networks are also referred to as “deep learning models” or “deep learning models.” In this document, the terms “deep learning model,” “machine learning model,” “learning network,” “neural network,” “model,” and “network” are used interchangeably.

[0016] Generally, a neural network includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications often include many hidden layers, thus extending the depth of the network. The layers of a neural network are connected in sequence, such that the output of a preceding layer is provided as input to a following layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer. A convolutional neural network (CNN) is a type of neural network that includes one or more convolutional layers that perform convolution operations on respective inputs. CNNs can be used in a variety of scenarios, and are particularly well suited for processing image or video data.

[0017] As mentioned above, deep learning has found wide application in many tasks, which can include, for example, computer vision processing, speech recognition, natural language processing, etc. Some current mobile applications or Internet of Things (IoT) applications can already be implemented by deep learning algorithms. However, the execution of deep learning models is very computationally and storage resource intensive, both in the training of the deep learning model and in the subsequent use phase.

[0018] A possible solution is to transfer the execution of the deep learning model from a device with limited computational and / or storage resources (e.g., a mobile device, an IoT device) to another device with more resources, such as a cloud computing device, an edge server, a large computing system, etc. The results of the execution can be sent back to the respective device for use. However, this requires sending the input to the deep learning model, such as images, speech, text information, etc., to the device that implements the execution of the deep learning model, which can be sensitive or private data of the user. Transferring user data from a local device to another computing environment for use involves user privacy concerns. Public environments such as cloud computing environments are often difficult to trust, as malicious external attacks and untrusted internal management occur from time to time.

[0019] One approach that can protect user privacy is to perform deep learning models on a user's local device. In order to be able to perform deep learning models with limited computing and / or storage resources of the local device, it is often necessary to compress large deep learning models into smaller deep learning models and design new lightweight (small parameter set) models. This approach avoids the user data leaving the local device, thus can significantly reduce the possibility of privacy leakage. However, this locally performed approach has some problems. First, because there is an essential trade-off between model size and model accuracy, model compression and lightweight models are difficult to achieve the same accuracy as large deep learning models. In addition, even if it is feasible to run the redesigned model on the local device, the execution of the model can have large delays due to the limited computing and / or storage resources, affecting the user experience. Local execution also causes higher energy consumption, thus affecting the life of the device running on the battery.

[0020] As can be seen, it is a more efficient approach to transfer the execution of deep learning models to an external device with higher computing power, such as a cloud computing device, but the possible user data security problems in this approach should be solved or mitigated.

[0021] Figure 1 A block diagram of a computing system 100 capable of implementing a number of implementations of the present disclosure is shown. The computing system 100 involves a computing device 101 and a device 102. The computing device 101 is configured to perform respective tasks for the device 102 in response to a request of the device 102. Figure 1 The structure of the computing device 101 is shown in detail in the detailed description. As shown, the computing device 101 includes a computing device 101 in the form of a general purpose computing device. The components of the computing device 101 can include, but are not limited to, one or more processing units or processors 110, a memory 120, a storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160. It should be understood that Figure 1 The computing device 101 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein.

[0022] In some implementations, the computing device 101 can be implemented as various terminals or devices with computing capabilities. For example, the computing device 101 can be a cloud computing device, an edge server, a large computing system, etc. The computing device 101 can also be other devices with computing capabilities, or even a mobile terminal, a fixed terminal, or a portable terminal, etc.

[0023] The processor 110 can be a real or virtual processor and is capable of performing various processing according to a program stored in the memory 120. In a multi-processor system, a plurality of processors performs computer-executable instructions in parallel to improve parallel processing capability of the computing device 101. The processor 110 can also be referred to as a central processing unit (CPU), a microprocessor, a controller, a microcontroller.

[0024] The computing device 101 typically includes a plurality of computer storage media. Such media can be any available media that is accessed by the computing device 101 and includes both volatile and nonvolatile media, removable and non-removable media. The memory 120 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory), or some combination thereof. The storage device 130 can be a removable or non-removable media and can include machine readable media such as a memory, a flash drive, a disk drive, or any other media capable of storing information and / or data and accessible by the computing device 101.

[0025] The computing device 101 can further include additional removable / non-removable, volatile / nonvolatile storage media. Although not shown in Figure 1 FIG. 1, a disk drive for reading from or writing to a removable, non- volatile magnetic disk (e.g., a "floppy" disk), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In such instances, each drive can be connected to the bus (not shown) by one or more data media interfaces.

[0026] The communication unit 140 enables communication with another device over a communication medium. Additionally, the functionality of the components of the computing device 101 can be implemented in a single computing cluster or a plurality of computers that are capable of communicating over a communication connection. Thus, the computing device 101 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0027] Input device 150 can be one or more various input devices such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 can be one or more output devices such as a display, speakers, a printer, etc. Computing device 101 can also include, as necessary, other input or output devices such as a disk drive, modem, etc. These devices and / or systems can be connected to computing device 101 through input / output (I / O) interface(s) (not shown) via the use of appropriate connectors. Accordingly, computing device 101 includes a bus 102 or other communication mechanism for enabling communication between various components of the device. Bus 102 can be any bus conforming to a standard including the Advanced Technology Attachment (AT A) and Enhanced Industry Interface (El l) standards.

[0028] In some implementations, some or all of the components of computing device 101 can also be provided in the form of a cloud computing architecture, in addition to being integrated on a single device. In a cloud computing architecture, the components can be arranged remotely and can work together to implement the functionality described in this disclosure. In some implementations, cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various implementations, cloud computing uses appropriate protocols over a wide area network, such as the Internet, to deliver services. For example, cloud computing providers deliver applications over a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of a cloud computing architecture, and the corresponding data, can be stored on servers at remote locations. Computing resources in a cloud computing environment can be consolidated at a remote data center location or they can be dispersed. Cloud computing infrastructure can provide services through a shared data center, even though they appear as a single point of access for users. Thus, the components and functionality described herein can be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on a client device.

[0029] The computing device 101 can be used to implement execution of deep learning models in a number of implementations of the present disclosure. The model execution in the computing device 101 can be initiated in response to a request from the device 102. In executing a deep learning model, the computing device 101 can obtain an input 172 of the deep learning model from another device 102. For example, the computing device 101 can receive 172 from the device 102 via the communication unit 140. The device 102 can be, for example, a user terminal for providing the input 172 to the computing device 101 in response to a user operation. The deep learning model to be executed is stored in the computing device 101, and the computing device 101 is configured to cause the deep learning model to process the input 172 to generate an output 174 of the deep learning model. The output 174 can be provided to the device 102, for example, by the communication unit 140.

[0030] Implementations of the present disclosure propose a scheme for execution of deep learning models. The scheme proposes to execute a deep learning model, in particular, to execute convolution operations of convolution layers of the deep learning model, in a trusted execution environment (TEE) of a computing device. Before describing implementations of the present disclosure in detail, first, a TEE is briefly introduced.

[0031] A TEE is a hardware-protected secure execution environment, also referred to as a hardware-protected enclave or secure enclave. A TEE can enable isolated execution from other parts of a computing device. Code execution in a TEE can obtain a very high level of protection, ensuring confidentiality and integrity of code and data in the TEE even when the operating system, hypervisor, and basic input / output system (BIOS) and the like are infected with a virus or subjected to a malicious attack. A TEE is also able to defend against hardware attacks such as memory snooping. Thus, even a malicious administrator of the computing device cannot access code and data in the TEE. In some implementations, a TEE also provides remote attestation by a third party to verify code and data loaded into the TEE.

[0032] A TEE can be implemented in a processor of a computing device, protected by a hardware protection mechanism of the processor, with code and data therein executed and stored by a specific protected memory area of the processor. Data, information, and the like exchanged by a TEE with components or devices outside the TEE are encrypted, thereby protecting data security. A TEE can be implemented, for example, by a software security extension (SGX) technology. Other implementation manners of a TEE can be, for example, a secure virtual machine, a converged security engine (CSE), and the like.

[0033] It is thus seen that TEEs can be used in applications that require protection against privacy leakage. However, the inventors have discovered through research that there can be some performance drawbacks if a deep learning model is directly placed into a TEE for execution. For example, the execution speed of a deep learning model implemented in a TEE can be several times slower (e.g., 6.4 times slower) than the execution speed in a standard computing environment outside of the TEE. The inventors have found that there are two main reasons for this performance reduction.

[0034] In one aspect, memory reads and writes are slower in a TEE than in a standard execution environment outside of the TEE, and yet the execution of a deep learning model requires a large number of memory reads and writes. The memory operation speed is reduced because the TEE uses a special protected memory region, also known as processor reserved memory (PRM). All data in the protected memory region is encrypted by a special chipset, which causes additional data encryption and decryption at every memory access.

[0035] On the other hand, the protected memory region is typically limited and difficult to expand on the fly as needed. For example, in an Intel Skylake CPU, such a protected memory region is 128 MB. Thus, such a memory space size is typically smaller than the size of many deep learning models, e.g., some large deep learning models can require more than 1 GB of memory space, and some smaller deep learning models can also require more than 200 MB of memory space. While a TEE can support larger memory space needs through techniques such as paging, this further reduces the execution speed because paging requires frequent swapping of data from the protected memory region to an unprotected memory region or vice versa, thus incurring additional data encryption and decryption operations. While it is theoretically possible to increase the size of the protected memory region of a TEE, the size of the protected memory region is typically an engineering carefully set value in the BIOS and is difficult to modify easily. Moreover, increasing the size of the protected memory region of a TEE also reduces the memory space available in the computing device for other standard applications that do not require memory protection.

[0036] In view of the constraints in storage space and storage access, the inventors found that the convolution operation in the deep learning model needs further improvement when the deep learning model is put into execution in the TEE. Therefore, according to some implementations of the present disclosure, an improved execution scheme of the deep learning model is proposed. Specifically, according to the scheme, the convolution operation is divided into a plurality of partitioned convolution operations, and the plurality of partitioned convolution operations are sequentially executed in the TEE. For each partitioned convolution operation, the input of the convolution layer determines the input part in matrix form to be processed by the partitioned convolution operation. The input part is stored in the protected storage area of the TEE. The result of the partitioned convolution operation is determined by a single matrix multiplication operation of the input part and a parameter value subset of the input part in the parameter value set for the convolution operation. The result of the convolution operation is determined by merging the results of the plurality of partitioned convolution operations as the output of the convolution layer.

[0037] By executing the deep learning model in the TEE, the protection of data can be improved, and the user privacy can be prevented from being leaked. Thus, the deep learning model can be executed on an external computing device with strong computing power, achieving high accuracy and low latency of model execution while protecting user privacy. In addition, by dividing the convolution operation into a plurality of partitioned convolution operations sequentially executed, the model execution can be further adapted to the TEE with limited storage space, achieving a trade-off between computing speed and storage space consumption.

[0038] The basic principles and several example implementations of the present disclosure are explained below with reference to the accompanying drawings.

[0039] Firstly, reference is made to Figure 2 which shows a schematic block diagram of a computing environment implemented in a computing device according to some implementations of the present disclosure. For ease of illustration, a computing environment implemented in a computing device 101 in Figure 1 is taken as an example. According to implementations of the present disclosure, the computing device 101 establishes an execution environment 200 when in operation. The execution environment 200 includes a TEE 210. In this document, the part of the execution environment 200 other than the TEE 210 is referred to as an untrusted execution environment.

[0040] As mentioned above, TEE 210 is a hardware-assisted secure execution environment that provides the smallest possible attack surface (e.g., processor boundary). TEE 210 can be implemented within the processor 110 of computing device 101, with code and data protection provided by the processor 110's hardware protection mechanisms. TEE 210 is also allocated a protected memory area 212. The protected memory area 212 is not accessible by operations outside the TEE, including direct memory access (DMA) from peripheral devices. In some implementations, the protected memory area 212 has a predetermined size for high-speed storage within the TEE, metadata storage required for memory encryption and decryption, etc. In some implementations, TEE 210 can also be implemented to support page swapping, allowing data (e.g., rarely used data) in the protected memory area 212 to be swapped to unprotected memory space outside the protected memory area 212 when needed, or data to be retrieved from unprotected memory space. Thus, TEE 210 can use a larger storage space than the protected memory area 212 during execution. During page switching, the TEE 210 uses symmetric key encryption to protect the confidentiality and integrity of the data.

[0041] According to the implementation of this disclosure, a portion or all of the deep learning model 220 is executed in the TEE 210. The TEE 210 includes a model executor 214 for performing corresponding operations in the deep learning model 220.

[0042] For ease of understanding, the following is a brief introduction to deep learning model 220. Figure 2 The deep learning model 220 is shown as a neural network. The neural network has a hierarchical architecture, with each layer having one or more processing nodes (called neurons or filters) for processing. In the neural network, the output of the previous layer after processing becomes the input of the next layer, where the first layer in the architecture receives model input 172 for processing, and the output of the last layer is provided as model output 174. Figure 2 As shown, the deep learning model 220 includes N network layers 222-1, 222-2, ..., 222-N, where N is an integer greater than 2. Network layer 222-1 receives model input 172, and network layer 222-N provides model output 174. For ease of description, network layers 222-1, 222-2, ..., 222-N of the deep learning model 220 are sometimes collectively referred to or individually as network layer 222.

[0043] It should be understood that Figure 2 The architecture of the deep learning model shown, along with the number of network layers and processing nodes, is illustrative. In different applications, deep learning models can be designed with other architectures as needed.

[0044] Generally, the main processing operations within a neural network are interleaved linear and non-linear transformations. These processes are distributed across various processing nodes. Figure 2 An enlarged view of one node 221 in the model 220 is also shown. The node 221 receives a plurality of input values a1, a2, a3, etc., and processes the input values based on corresponding parameter values (such as weights w1, w2, w3, etc.) to generate an output z. The node 221 can be designed to process the inputs with one activation function, which can be represented as:

[0045] z = σ(w T a) (1)

[0046] where a represents an input vector of the node 221 (which includes elements a1, a2, a3, etc.); w represents a weight vector of the parameter values used by the node 221 (which includes elements w1, w2, w3, etc.), each weight is used to weight a corresponding input; N represents the number of input values; and σ() represents an activation function used by the node 221, which can be a linear function, a non-linear function. Commonly used activation functions in neural networks include sigmoid function, ReLu function, tanh function, maxout function, etc. The output of the node 221 can also be referred to as an activation value. Depending on the network design, the output (i.e., the activation value) of each network layer can be provided to one, multiple, or all nodes of the next layer as input.

[0047] The parameter values of certain types of nodes 221 can also include a bias for each input, in which case the equation (1) can be rewritten as:

[0048] z = σ(w T a + b) (2)

[0049] where b represents a bias vector of the parameter values used by the node 221 (which includes elements b1, b2, b3, etc.), each bias is used to bias the result of a corresponding input and weighting.

[0050] Each network layer in the deep learning model 220 can include one or more nodes 221, and when viewing the processing in the deep learning model 220 in units of network layers, the processing of each network layer can also be similarly represented in the form of equation (1) or equation (2), in which case a represents an input vector of the network layer, and w and possibly b represent the parameter values of the network layer, which increase in size accordingly.

[0051] In some implementations, the execution of the deep learning model 220 can be model execution in a case where the deep learning model 220 has been trained. The values of the parameters (or simply parameter values) of the trained deep learning model 220 are determined. In model execution, the input of the model is processed with the trained parameter values to determine the corresponding output. Such model execution can also be referred to as model inference or model usage.

[0052] In some other implementations, the execution of the deep learning model 220 can be model execution in a training phase of the deep learning model 220. In the training phase, data for training is input to the deep learning model 220, and the current values of the parameters are adjusted by determining the difference between the output determined by the deep learning model 220 based on the current parameter values and the real output. In training, the values of the parameters of the deep learning model 220 need to be executed constantly until the convergence condition of the training is met. Therefore, each model execution in the model training phase uses the current values of the parameters.

[0053] In the deep learning model 220, especially in the deep learning model 220 with a hierarchical architecture, processing is performed in units of layers. The model executor 214 is configured to execute the deep learning model 220 layer by layer. Considering the limited space of the protected storage area 212 of the TEE 210, in some implementations, the deep learning model 220 will be parsed, where the configuration information of the structure, connection relationship, attributes of nodes, etc. of each layer of the model is configured in the TEE 210, and the parameter values 202 of the deep learning model 220 are stored in the unprotected storage area outside the TEE 210. In this way, the model executor 214 can determine the model structure of the deep learning model 220 and the processing manner of each layer according to the configuration information. The parameter values of the deep learning model 220 need to occupy a large storage space, and the security and privacy requirements of the parameter values are lower compared to user data, so storing them in the unprotected storage area can save the storage space available to the TEE 210. In addition, this can prevent the delay caused by frequent page swapping that may be triggered in the case of storing a large amount of data in the protected storage area 212, because all parameter values of the entire model are not frequently used in the TEE 210, which can trigger the page swapping mechanism. The unprotected storage area may, for example, be the general memory of the computing device 110.

[0054] In case the parameter values are stored in a protected storage, the TEE 210 can further comprise a parameter loader 216 for loading the required parameter values from the external storage into the protected storage 212 on demand for use by the model executor 214. This can improve the efficiency of the storage space in the TEE 210. The storage addresses of the parameter values 202 of the deep learning model 220 can be provided to the parameter loader 216 so that the parameter loader 216 can fetch the required parameter values from the external storage. Since the deep learning model 220 is executed layer by layer, in some implementations, the parameter loader 216 can load the parameter values layer by layer from the first layer to the last layer of the model. The loading of the parameter values can depend on the current execution stage in the model executor 214.

[0055] In some implementations, the model execution in the model executor 214 can be triggered by an execution request. For example, the model input 172 of the deep learning model 220 can come from an external device of the computing device 101, such as a device 102 of a user. The device 102 can be a terminal device, and the provided model input 172 can be sensitive or private user information. The model input 172 can be provided into the TEE 210 via a secure channel between the device 102 and the TEE 210. In one example, the model input 172 can be encrypted, received by the communication unit 140 of the computing device 101, and provided into the TEE 210. The model input 172 is decrypted in the TEE 210 for triggering the execution of the deep learning model 210. Figure 1

[0056] In implementations of the present disclosure, the deep learning model 220 to be executed comprises one or more convolutional layers 222. Convolutional layer is a kind of network layer often adopted in deep neural networks, which can achieve very good processing effect for data such as images, videos, etc. The deep learning model 220 with convolutional layers is also sometimes referred to as convolutional neural network. Depending on the configuration of the deep learning model 220, the convolutional layers can be deployed at the first layer and / or intermediate layers of the model. As will be discussed below, the convolution operation in the deep learning model 220 requires relatively high computing resources and storage resources, and therefore improving the convolution operation helps to significantly improve the model execution in the TEE 210 with high security level but limited storage resources, speed up the model execution, and improve the storage space efficiency.

[0057] ​The convolutional layers 222 in the deep learning model 220 include one or more convolutional kernels, also called filters. Each convolutional kernel has three dimensions: length, width, and depth. The length and width of the convolutional kernel are hyperparameters of the deep learning model 220 and can be specified by the model designer. Length × width is also called the kernel size, and common sizes include 2×2, 3×3, and 5×5. The depth of the convolutional kernel is related to the input to the convolutional layer. Typically, the input to the convolutional layer is represented as one or more two-dimensional feature maps (or, for the first layer, the input is the original image). The depth of the input refers to the number of channels in the input image (i.e., the number of feature maps). The number of convolutional kernels in each convolutional layer is also a hyperparameter of the deep learning model 220 and can be pre-specified by the designer. The number of convolutional kernels is the same as the number of feature maps expected to be output by the convolutional layer, and this number is also pre-configured as a hyperparameter of the deep learning model 220. Each convolutional kernel's parameter value is multiplied by the corresponding part of the convolutional layer's input, and the kernel slides across the input with a predetermined stride, thus obtaining multiple multiplication results to form the output of that kernel, i.e., a feature map. If the convolutional layer has multiple kernels, each kernel can be used to calculate its corresponding feature map, which serves as the output of the convolutional layer.

[0058] Figure 3A The diagram illustrates a direct convolution operation of a convolutional layer 222 in an example deep learning model 220. Figure 3A In the example, the input to the convolutional layer is 3×3 feature maps 310-1 and 310-2, so the input dimension is 3×3×2, where 2 represents the number of input channels. This convolutional layer uses a single 2×2 kernel with a depth of 2. Therefore, the convolution operation is performed on feature maps 310-1 and 310-2 using two 2×2 parameter value windows, 320-1 and 320-2, respectively, with a stride of 1. For example, parameter value windows 320-1 and 320-2 start from the top left corner of feature maps 310-1 and 310-2, moving from left to right and then downwards. For instance, parameter value window 320-1 moves and overlaps with elements B1, C1, E1, and F1 of feature map 310-1, so the matrices of individual parameter values ​​K1, K2, K3, and K4 are multiplied with the matrices of elements B1, C1, E1, and F1 to obtain the first intermediate result. The parameter value window 320-2 similarly moves across feature map 310-2, and upon reaching elements B2, C2, E2, and F2 in feature map 310-2, performs matrix multiplication with these elements to obtain a second intermediate result. The first and second intermediate results are then added to form element O2 in the result 330 of the convolution operation. Other elements in the result 330 of the convolution operation are determined similarly.

[0059] It is evident that direct convolution requires many small matrix multiplication operations, which reduces processing speed and makes it unsuitable for models requiring high efficiency.

[0060] A method for accelerating convolution operations has been proposed, known as convolution reduction. Figure 3B An example of a fast convolution operation based on convolution reduction is shown. According to this convolution operation, the elements in the feature maps 310-1 and 310-2 input to the convolutional layer are rearranged, and the parameters of the convolution kernel used are also expanded, thereby transforming the multiple matrix multiplications of the convolution operation into a single matrix multiplication.

[0061] like Figure 3B As shown, based on multiple small matrix multiplications of the convolution kernel parameter value windows 320-1 and 320-2 with feature maps 310-1 and 310-2, the elements of feature maps 310-1 and 310-2 are rearranged into input matrix 340. Input matrix 340 is related to the size of the set of parameter values ​​for the matrix operation (e.g., the length, width, and depth of the convolution kernel) and the size of the input (the length, width, and depth of the feature maps). The individual parameter values ​​in parameter value windows 320-1 and 320-2 are arranged into an 8-by-1 parameter value matrix 350. Note that if multiple convolution kernels exist, the parameter values ​​of the multiple convolution kernels are also rearranged to obtain a single parameter value matrix 350. Thus, the result 330 of the convolution operation can be determined through a single matrix multiplication operation of input matrix 340 and parameter value matrix 350. For example, in a single matrix multiplication operation, multiplying the second row of input matrix 340 with the first column of parameter value matrix 350 yields element O2 in result 330. Note that although the arrangement of each element in the matrix is ​​different in result 330, the values ​​of each element are the same, and the arrangement of elements can be adjusted as needed.

[0062] Compared to multiple matrix multiplication operations, a single matrix multiplication operation after convolution can significantly improve the computation speed of convolution. However, the size of the rearranged input matrix is ​​much larger than the original size of the input to the convolutional layer. For example, if an M×M convolution kernel is used, the size of the rearranged input matrix is ​​M times the original size. 2 The speed increase is achieved by sacrificing storage overhead after the convolution operation. Since the input matrix needs to be stored after the convolution operation, this incurs significant storage costs. Therefore, convolution reduction actually sacrifices storage overhead for speed improvement.

[0063] Other methods exist for implementing fast convolution operations, such as convolution based on the Fast Fourier Transform (FFT) and convolution based on Winograd. However, these methods do not reduce storage overhead and fail to achieve a trade-off between computational speed and storage cost. Therefore, existing fast convolution operations are not suitable for use in TEEs with limited storage space.

[0064] According to implementations of the present disclosure, upon determining that a convolution operation of a convolution layer of the deep learning model 220 is triggered, the model executor 214 in the TEE 210 sequentially executes a plurality of partitioned convolution operations that collectively constitute the convolution operation of the convolution layer. Specifically, rather than converting the entire convolution operation into a single matrix multiplication as in the convolution reduction scheme, in implementations of the present disclosure, the convolution operation is divided into a plurality of partitioned convolution operations, each of which is implemented with a single matrix multiplication.

[0065] Specifically, for a given partitioned convolution operation among the plurality of partitioned convolution operations, the model executor 214 determines, from the input of the convolution layer, an input portion to be processed by the parameter value subset of the given partitioned convolution operation, which is represented in a matrix. In executing the partitioned convolution operation, the parameter value set of the current convolution layer is also divided into different parameter value subsets for different partitioned convolution operations. That is, each partitioned convolution operation is executed with different parameter values. In some implementations, the parameter values corresponding to each convolution kernel in the convolution layer are arranged as a column or a row in a parameter value matrix corresponding to the parameter value set. The parameter value matrix can be divided into parameter value subsets in the same number as the number of partitioned convolution operations, either by rows or by columns. Each parameter value subset is also in the form of a matrix. Such division can be uniform division, i.e., the number of parameter values in each parameter value subset is the same, or can be non-uniform division, i.e., the number of parameter values in different parameter value subsets is different. The model executor 214 can determine, from the input of the convolution layer, the input portion to be processed by the parameter value subset of a certain partitioned convolution operation. The input portion includes elements that will be multiplied by each parameter value in the parameter value subset in the overall convolution operation.

[0066] In some implementations, in order to implement the single matrix multiplication of the input portion and the parameter value subset, the input of the convolution layer is converted into an input matrix upon determining the input portion for the given partitioned convolution operation. This is, for example, similar to the input rearrangement in the matrix reduction scheme. The arranged input matrix is thus related to the size of the parameter value set of the matrix operation (e.g., the length, width, and depth of the convolution kernel) and the size of the input (the length, width, and depth of the feature map). Thus, by rearrangement, the elements included in the row or column of the input matrix are to be used in the convolution operation to be sequentially multiplied by the corresponding parameter values in the parameter value set of the convolution operation, if a direct convolution operation is to be performed. Note that the row or column includes the elements arranged in this way in relation to the matrix arrangement, and the row and column of the matrix can be arbitrarily converted. Then, based on the number of the plurality of partitioned convolution operations and the order of the given convolution operation among the plurality of partitioned convolution operations, the input portion to be processed by the parameter value subset of the given partitioned convolution operation is determined from the input matrix. Thus, a single input matrix is divided into the same number as the partitioned convolution operations.

[0067] Reference is made to Figure 4 better understand the partitioned convolution operation of the present disclosure. As shown in Figure 4 , the convolution operation between the input feature maps 310-1 and 310-2 of the convolution layer and the parameter value set consisting of the parameter value windows 320-1 and 320-2 is divided into two partitioned convolution operations. The parameter value set is divided into parameter value subsets 420-1 and 420-2 corresponding to the two partitioned convolution operations respectively. In this example, the parameter value set is evenly divided. In the first partitioned convolution operation, an input portion 410-1 is determined from the input feature maps 310-1 and 310-2, which is to be processed by the parameter value subset 420-1. The input portion 410-1 can be determined by dividing a single input matrix (e.g. the single matrix 340 in Figure 3B ) determined from the input feature maps 310-1 and 310-2. In this example, the input matrix 340 is divided into two portions by columns, since the elements in the rows of the single matrix 340 are to be used in the convolution operation to sequentially multiply with the corresponding parameter values in the parameter value set.

[0068] After the input portion for a given partitioned convolution operation is determined, the model executor 214 stores the input portion in the protected storage area 212 of the TEE 210. Then, the model executor 214 performs a single matrix multiplication operation on the input portion with the parameter value subset for the corresponding partitioned convolution operation, thereby determining the result of the current partitioned convolution operation. After the result of the current partitioned convolution operation is determined, the input portion for the current partitioned convolution operation is removed from the protected storage area 212, so that the input portion can be reused to store the next partitioned convolution operation. This can reduce the storage space required in the convolution process.

[0069] For example, in Figure 4 , the model executor 214 first stores the input portion 410-1, and then performs a single matrix multiplication operation on the input portion 410-1 with the parameter value subset 420-1, to determine the result 430-1 of the first partitioned convolution operation. In the single matrix multiplication operation, each row of the input portion 410-1 is multiplied with the parameter value subset 420-1, and the result of the multiplication corresponds to an element in the result 430-1 of the first partitioned convolution operation. For example, the elements B1, C1, E1 and F1 in the second row of the input portion 410-1 are multiplied with the parameter value subset 420-1, resulting in the element O2’ in the result 430-1. The input portion 410-1 can be removed from the protected storage area 212 at this time. Next, the model executor 214 determines and stores the input portion 410-2 to be processed by the next partitioned convolution operation, and performs a single matrix multiplication on the input portion 410-2 with the parameter value subset 420-2 to determine the result 430-2 of the second partitioned convolution operation.

[0070] The model executor 210 can sequentially perform the multiple partitioned convolution operations. After determining the result of each of the multiple partitioned convolution operations, the model executor 214 determines the result of the convolution operation by merging the results of the multiple partitioned convolution operations as the output of the convolution layer. For example, in the example of Figure 4 , the model executor 214 merges the result 430-1 of the first partitioned convolution operation and the result 430-2 of the second partitioned convolution operation to determine the final result 330 of the convolution operation. In implementations of the present disclosure, the convolution operation of one convolution layer is represented as the sum of multiple partitioned convolution operations, which can be represented as follows:

[0071]

[0072] where O represents the result of the convolution operation, i.e., the output of the convolution layer; I i represents the i-th input portion; K i represents the i-th parameter value subset; and n represents the number of partitioned convolution operations.

[0073] In the partitioned convolution process of the present disclosure, since each partitioned convolution operation only needs to store the input portion to be used by the current partitioned convolution operation, the required storage space in implementing the convolution is reduced by timely removing the input portion used by the previous partitioned convolution operation. Such reduction of storage space makes the deep learning model more suitable for execution in a TEE with limited storage space. In some implementations, the result of each partitioned convolution operation is also stored in the protected storage area 212 until the final result of the convolution operation is determined.

[0074] In some implementations, the number of partitioned convolution operations to be performed for a particular convolution layer can be determined based on the available storage space size of the protected storage area 212. The model executor 214 can determine the number of partitioned convolution operations to be performed according to the currently available storage space size in the TEE each time the convolution operation of a certain convolution layer is performed. The number of partitioned convolution operations can also be specific to the deep learning model to be executed, e.g., determined based on the convolution layer with the largest storage consumption in the deep learning model. The number of partitioned convolution operations can also be any fixed value.

[0075] In some implementations, the number of partitioned convolution operations to be performed can be 2 raised to the power of n, where n can be an integer greater than or equal to 1. In some implementations, the number of partitioned convolution operations to be performed can be set to be less than the number of matrix multiplication operations to be performed in a direct convolution operation, so as to make the speed of the entire convolution process improved compared to the number of direct convolution operations. For example, in the example of Figure 4 , the number of partitioned convolution operations can be less than 8.

[0076] In some implementations, the model executor 214 can determine the number of partitioned convolution operations to be performed such that the storage space required in each partitioned convolution operation performed sequentially is less than a predetermined storage space threshold. The predetermined storage space threshold can be determined in relation to the total space size of the protected storage region 212 of the TEE 210, the size of other storage space necessary for the execution of the deep learning model 220, etc. In one example, the predetermined storage space threshold can be set to 32MB.

[0077] The above discusses how the convolution operations of the convolution layers of the deep learning model 220 are performed in the TEE 210. According to the layer-by-layer execution order of the deep learning model 220, if the convolution operation of a convolution layer of the deep learning model 220 is triggered, the model executor 214 will perform the convolution operation of the current convolution layer. The triggering of the corresponding convolution operation of the convolution layer can be in response to the input of the convolution layer being determined. The input determination of the convolution layer depends on the location of the convolution layer in the deep learning model 220. For example, if the convolution layer is at the first layer of the deep learning model 220, the convolution operation is triggered upon receiving the model input 172. If the convolution layer is at an intermediate layer of the deep learning model 220, the convolution operation of the convolution layer is triggered upon the output of the network layer before the convolution layer (which can be a convolution layer or any other layer such as a pooling layer, etc.) being determined, and thus the output of the previous network layer will be used as the input of the convolution layer.

[0078] During the convolution operation, the original input of the convolution layer (rather than the individual input portions reduced by the convolution) is also stored in the protected storage region 212. After the output of the convolution layer is determined, if the original input of the convolution layer will not be used in the subsequent layers of the deep learning model, the model executor 214 or other components in the TEE 210 can remove the input of the convolution layer from the protected storage region 212 in order to further save the consumption of storage space. Similarly, after the output of the convolution layer is determined, if the current convolution layer is an intermediate layer of the deep learning model 220, whose output will be used as the input in the next layer, the model executor 214 can store the output in the protected storage region 212 to facilitate the use of the subsequent layer. If it is subsequently determined that the output is no longer used, the output can also be removed from the protected storage region 212. In addition to the convolution layers of the deep learning model 210, for other network layers, the inputs / outputs of the intermediate network layers in the protected storage region 212 can also be similarly cleaned up in time. It should be noted that the input of certain types of network layers can be used after a number of subsequent network layers. For such inputs, they can be removed only after it is determined by the model operation logic analysis that the input is no longer used. The reuse after a number of network layers typically occurs in recurrent neural networks (RNNs).

[0079] As mentioned above, in the TEE 210, since the parameter values of the deep learning model 220 are stored in a storage area outside the TEE 210, depending on the current execution stage in the model executor 214, the parameter loader 216 loads the parameter values required for model execution from outside into the TEE 210. In some implementations, since multiple partitioned convolution operations are sequentially executed, the parameter loader 216 can load the parameter values by partitioned convolution operation. For example, the parameter loader 216 loads a corresponding subset of parameter values from a storage area outside the TEE 210 into the protected storage area 212 upon determining that a given partitioned convolution operation is to be executed. In some implementations, to avoid the model executor 214 having to wait for parameter loading, the parameter loader 216 ensures that the corresponding subset of parameter values has been loaded by the time the given partitioned convolution operation is executed.

[0080] In some implementations, parameter loading and model execution can be performed in parallel to improve efficiency. For example, when performing convolution operations, while the model executor 214 is executing a certain partitioned convolution operation, the parameter loader 216 can load, in parallel, a subset of parameter values to be used by a partitioned convolution operation that is to be executed after the partitioned convolution operation that is currently being executed. This can ensure that the corresponding subset of parameter values is ready in the protected storage area 212 by the time the model executor 214 executes the subsequent partitioned convolution operation. The parameter loader 216 can similarly perform parameter loading in parallel with model execution throughout the execution of the deep learning model 220, except for convolution layers. For example, while the model executor 214 is executing operations of one network layer, the parameter loader 216 can simultaneously continue loading parameter values for a subsequent network layer.

[0081] In addition to performing parameter value loading by network layer or by partitioned convolution operation, the parameter loader 216 can perform parameter value loading by any other unit. For example, the parameter loader 216 can divide parameter values of a network layer into multiple blocks and load the parameter values block by block. The model executor 214 can sequentially perform operations based on the parameter values that have been loaded. For another example, the parameter loader 216 can load parameter values of multiple network layers or parameter values of multiple partitioned convolution operations at a time.

[0082] In some implementations, loaded parameter values can be removed from the protected storage area 212 after the corresponding operations are completed, so as to save storage space. For example, upon determining a result of a partitioned convolution operation, the subset of parameter values used by the partitioned convolution operation can be removed from the protected storage area 212.

[0083] Since the parameter values of the deep learning model 210 are stored in the unprotected external storage area, in some implementations, after obtaining the parameter values from the external storage area, an integrity check of the parameter values can also be performed in the TEE 210. Figure 5 An example of such an implementation is shown. As shown, the TEE 210 also includes a parameter checker 510. Figure 5

[0084] A set of expected integrity check values 502 of the parameter values of the deep learning model 210 are pre-stored in the protected storage area 212. The expected integrity check values in the set 502 can be stored in an initial stage when the TEE 210 is created. One expected integrity check value can be calculated for each subset of parameter values of the deep learning model 210, or the subset of parameter values can be further divided into multiple smaller subsets, and one expected integrity check value is calculated for each smaller subset. For the set of parameter values of other network layers in the deep learning model 210, a single expected integrity check value can also be similarly calculated, or the set of parameter values is divided into multiple subsets, and an expected integrity check value is calculated for each subset. The integrity check value can be determined, for example, by performing a hash operation on the corresponding parameter values, such an integrity check value is also referred to as a hash check value.

[0085] After the parameter loader 216 obtains the subset of parameter values used by a given partitioned convolution operation from the outside, the parameter checker 510 also calculates an integrity check value of the obtained subset of parameter values in a similar manner, and then compares the calculated integrity check value with the corresponding expected integrity check value. If the calculated integrity check value matches (i.e., is identical to) the expected integrity check value, the parameter checker 510 confirms the integrity of the subset of parameter values. In this case, the subset of parameter values is formally stored in the protected storage area 212. In some implementations, if the integrity check of the subset of parameter values fails, the parameter checker 510 can cause the model execution process in the TEE 210 to stop. The TEE 210 then returns an error message to the device 102 indicating that an error has occurred in the parameter values of the deep learning model 210. There can be many options for the subsequent handling of the error, which can be determined by the device 102 or its user whether to continue the model execution.

[0086] In some implementations, since the parameter values can only be used by the model executor 214 after passing the check, to improve the efficiency of the model execution, the three stages of parameter loading, parameter checking, and model execution can be performed in parallel. In some implementations, a ring buffer is configured in the protected buffer area 212 for storing the subset of parameter values. Parallel pipeline threads for the three processing stages can be created in the TEE 210. Figure 6 ​The pipelined processing of the parameter loader 216, the parameter verifier 510, and the model executor 214 is shown.

[0087] Once a parameter value subset is put into the ring buffer by the parameter loader 216, the parameter verifier 510 can immediately start computing and verifying the integrity check value of the parameter value subset. At this time, the parameter loader 216 starts loading the next parameter value subset. Similarly, after the parameter verifier 510 finishes the verification, the model executor 214 can immediately start the model execution using the parameter value subset that has been verified, while the parameter verifier 510 can start verifying the next parameter value subset. After the parameter value subset has been used for the model execution, it can be released from the ring buffer so that the ring buffer can be used for loading new parameter values.

[0088] From Figure 6 As can be seen, while the model executor 214 is executing a first partitioned convolution operation using a first parameter value subset (denoted as Block 1), the parameter verifier 510 can be verifying a second parameter value subset (denoted as Block 2) to be used for a second partitioned convolution operation following the first partitioned convolution operation, and the parameter loader 216 can be loading a third parameter value subset (denoted as Block 3) to be used for a third partitioned convolution operation following the second partitioned convolution operation. Such parallel pipelined processing can improve the model execution speed in the TEE.

[0089] Figure 7 A flowchart of a process 700 according to some implementations of the present disclosure is shown. The process 700 can be implemented by the computing device 101, e.g., can be implemented in the processor 110 of the computing device 101.

[0090] At block 710, the computing device 101 determines that a convolution operation of a convolution layer of a deep learning model is triggered. In response to the convolution operation of the convolution layer of the deep learning model being triggered, at block 720, the computing device 101 determines that there are partitioned convolution operations to be executed among a plurality of partitioned convolution operations. The plurality of partitioned convolution operations use different parameter value subsets partitioned from a parameter value set. If there are partitioned convolution operations to be executed, at block 730, the computing device 101 stores an input portion of an input of the convolution layer to be processed by a parameter value subset of a given partitioned convolution operation in a protected storage area of a TEE. The input portion is denoted as a matrix. At block 740, the computing device 101 determines a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and the parameter value subset for the given partitioned convolution operation. The parameter value subset is denoted as a matrix. At block 750, the computing device 101 removes the input portion from the protected storage area.

[0091] Then, the process 700 returns to block 720, and the computing device 101 continues to determine whether there are any partitioned convolution operations that have not been executed, and repeats blocks 730-750 to determine the result of the partitioned convolution operation if there are partitioned convolution operations to be executed, until all partitioned convolution operations are executed. When there are no partitioned convolution operations to be executed, at block 760, the computing device 101 determines the result of the convolution operation by combining the results of the plurality of partitioned convolution operations as the output of the convolution layer.

[0092] In some implementations, in the execution of the deep learning model, if the convolution operation of the convolution layer is not triggered, i.e., operations in other types of network layers are executed, the computing device 101 can also obtain the corresponding parameter values from outside the TEE according to the configuration of the model, and perform the operation of the corresponding network layer based on the obtained parameter values.

[0093] In some implementations, the number of the plurality of partitioned convolution operations is determined based on the available storage space size of the protected storage area.

[0094] In some implementations, storing the input portion in the protected storage area includes: converting the input into an input matrix based on the size of the set of parameter values and the size of the input, elements in a row or column of the input matrix are to be used in the convolution operation to sequentially multiply with corresponding parameter values in the set of parameter values; determining the input portion corresponding to the subset of parameter values from the input matrix based on the number of the plurality of partitioned convolution operations and the order of the given convolution operation in the plurality of partitioned convolution operations; and storing the determined input portion in the protected storage area.

[0095] In some implementations, the set of parameter values is stored in a storage area outside the TEE. The process 700 further includes: in response to determining that the given partitioned convolution operation is to be executed, loading the subset of parameter values from the storage area outside the TEE into the protected storage area.

[0096] In some implementations, loading the subset of parameter values into the protected storage area further includes: performing an integrity check of the subset of parameter values in the TEE, including: calculating an integrity check value of the subset of parameter values, comparing the calculated integrity check value with an expected integrity check value stored in the protected storage area, and confirming the integrity of the subset of parameter values in response to the calculated integrity check value matching the expected integrity check value; and storing the subset of parameter values into the protected storage area in response to confirming the integrity of the subset of parameter values.

[0097] In some implementations, the process 700 further includes: after determining the result of the given partitioned convolution operation, removing the subset of parameter values from the protected storage area.

[0098] In some implementations, computing the integrity check value for the subset of parameter values includes determining the integrity check value by performing a hash operation on the subset of parameter values.

[0099] In some implementations, loading the subset of parameter values into the protected storage includes loading the subset of parameter values into the protected storage in parallel with execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation.

[0100] In some implementations, performing the integrity check on the subset of parameter values in the TEE includes performing the integrity check on the subset of parameter values in the TEE in parallel with: execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation; and loading of another subset of parameter values into the protected storage, the other subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that follows the given partitioned convolution operation.

[0101] In some implementations, the set of parameter values includes: a set of trained parameter values of the deep learning model, or a set of parameter values of the deep learning model that are being trained.

[0102] In some implementations, the input to the convolution layer is stored in the protected storage, and the process 700 further includes removing the input from the protected storage in response to the input not being used in subsequent layers of the deep learning model after the output is determined.

[0103] The following lists some example implementations of the present disclosure.

[0104] In an aspect, the present disclosure provides a computer-implemented method. The method includes: in response to a convolution operation of a convolution layer of a deep learning model being triggered, sequentially performing, in a trusted execution environment (TEE) of a computing device, a plurality of partitioned convolution operations based on an input to the convolution layer and a set of parameter values, the plurality of partitioned convolution operations using different subsets of parameter values partitioned by the set of parameter values, execution of a given partitioned convolution operation of the plurality of partitioned convolution operations including: storing, in a protected storage of the TEE, an input portion of the input to be processed by the subset of parameter values of the given partitioned convolution operation, the input portion represented as a matrix, determining a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and the subset of parameter values for the given partitioned convolution operation, the subset of parameter values represented as a matrix, and removing the input portion from the protected storage; and determining a result of the convolution operation as an output of the convolution layer by merging results of the plurality of partitioned convolution operations.

[0105] In some implementations, a number of the plurality of partitioned convolution operations is determined based on a size of available storage space of the protected storage.

[0106] In some implementations, storing the input portion in the protected storage includes: converting the input into an input matrix based on a size of the set of parameter values and a size of the input, elements in a row or a column of the input matrix to be used in a convolution operation to sequentially multiply with respective parameter values in the set of parameter values; determining the input portion corresponding to the subset of parameter values from the input matrix based on a number of the plurality of partitioned convolution operations and an order of the given convolution operation in the plurality of partitioned convolution operations; and storing the determined input portion in the protected storage.

[0107] In some implementations, the set of parameter values is stored in a storage external to the TEE. The method further includes: in response to determining that the given partitioned convolution operation is to be executed, loading the subset of parameter values from the storage external to the TEE into the protected storage.

[0108] In some implementations, loading the subset of parameter values into the protected storage further includes: performing an integrity check of the subset of parameter values in the TEE, including: computing an integrity check value of the subset of parameter values, comparing the computed integrity check value with an expected integrity check value stored in the protected storage, and in response to the computed integrity check value matching the expected integrity check value, confirming an integrity of the subset of parameter values; and in response to confirming the integrity of the subset of parameter values, storing the subset of parameter values into the protected storage.

[0109] In some implementations, the method further includes: after determining a result of the given partitioned convolution operation, removing the subset of parameter values from the protected storage.

[0110] In some implementations, computing the integrity check value of the subset of parameter values includes: determining the integrity check value by performing a hash operation on the subset of parameter values.

[0111] In some implementations, loading the subset of parameter values into the protected storage includes: loading the subset of parameter values into the protected storage in parallel with an execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation.

[0112] In some implementations, performing the integrity check of the subset of parameter values in the TEE includes performing the integrity check of the subset of parameter values in the TEE in parallel with: an execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation; and a loading of another subset of parameter values into the protected storage, the another subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that succeeds the given partitioned convolution operation.

[0113] In some implementations, the set of parameter values includes: a set of trained parameter values of a deep learning model, or a set of parameter values of a deep learning model that is being trained.

[0114] In some implementations, the input to the convolution layer is stored in a protected storage area, and the method further includes removing the input from the protected storage area in response to determining that the input is not to be used in subsequent layers of the deep learning model after the output is determined.

[0115] In another aspect, the disclosure provides an electronic device. The electronic device includes a processor; and a memory coupled to the processor and containing instructions stored thereon that, when executed by the processor, cause the device to perform the following actions: in response to a convolution operation of a convolution layer of a deep learning model being triggered, sequentially performing, in a trusted execution environment (TEE) of the computing device, a plurality of partitioned convolution operations based on an input to the convolution layer and a set of parameter values, the plurality of partitioned convolution operations using different subsets of parameter values partitioned by the set of parameter values, performance of a given partitioned convolution operation of the plurality of partitioned convolution operations including: storing, in a protected storage area of the TEE, an input portion of the input to be processed by the parameter values subset of the given partitioned convolution operation, the input portion represented as a matrix, determining a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and the parameter values subset for the given partitioned convolution operation, the parameter values subset represented as a matrix, and removing the input portion from the protected storage area; and determining a result of the convolution operation by combining results of the plurality of partitioned convolution operations as an output of the convolution layer.

[0116] In some implementations, a number of the plurality of partitioned convolution operations is determined based on a size of available storage space of the protected storage area.

[0117] In some implementations, storing the input portion in the protected storage area includes: based on a size of the set of parameter values and a size of the input, converting the input into an input matrix, elements in a row or column of the input matrix to be sequentially multiplied with respective parameter values in the set of parameter values in the convolution operation; based on the number of the plurality of partitioned convolution operations and an order of the given convolution operation in the plurality of partitioned convolution operations, determining the input portion corresponding to the parameter values subset from the input matrix; and storing the determined input portion in the protected storage area.

[0118] In some implementations, the set of parameter values is stored in a storage area outside the TEE. The actions further include: in response to determining that the given partitioned convolution operation is to be performed, loading the parameter values subset from the storage area outside the TEE into the protected storage area.

[0119] In some implementations, loading the subset of parameter values into the protected storage further includes performing, in the TEE, an integrity check of the subset of parameter values, including: computing an integrity check value for the subset of parameter values, comparing the computed integrity check value to an expected integrity check value stored in the protected storage, and in response to the computed integrity check value matching the expected integrity check value, confirming the integrity of the subset of parameter values; and in response to confirming the integrity of the subset of parameter values, storing the subset of parameter values into the protected storage.

[0120] In some implementations, the actions further include: after determining the result of the given partitioned convolution operation, removing the subset of parameter values from the protected storage.

[0121] In some implementations, computing the integrity check value for the subset of parameter values includes determining the integrity check value by performing a hash operation on the subset of parameter values.

[0122] In some implementations, loading the subset of parameter values into the protected storage includes loading the subset of parameter values into the protected storage in parallel with the execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation.

[0123] In some implementations, performing, in the TEE, the integrity check of the subset of parameter values includes performing, in the TEE, the integrity check of the subset of parameter values in parallel with: the execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation; and the loading of another subset of parameter values into the protected storage, the other subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that succeeds the given partitioned convolution operation.

[0124] In some implementations, the set of parameter values includes: a set of trained parameter values of the deep learning model, or a set of parameter values of the deep learning model that are being trained.

[0125] In some implementations, the input to the convolution layer is stored in the protected storage, the actions further including: in response to the input not being used in a subsequent layer of the deep learning model after the output is determined, removing the input from the protected storage.

[0126] In yet another aspect, the disclosure provides a computer program product tangibly stored in a computer storage medium and comprising machine executable instructions that, when executed by a device, cause the device to: in response to a convolution operation of a convolutional layer of a deep learning model being triggered, sequentially perform, in a trusted execution environment (TEE) of the computing device, a plurality of partitioned convolution operations based on an input to the convolutional layer and a set of parameter values, the plurality of partitioned convolution operations using different subsets of parameter values partitioned by the set of parameter values, the performance of a given partitioned convolution operation of the plurality of partitioned convolution operations comprising: storing in a protected storage area of the TEE an input portion of the input to be processed by the parameter values subset of the given partitioned convolution operation, the input portion represented as a matrix, determining a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and the parameter values subset for the given partitioned convolution operation, the parameter values subset represented as a matrix, and removing the input portion from the protected storage area; and determining a result of the convolution operation by combining results of the plurality of partitioned convolution operations as an output of the convolutional layer.

[0127] In some implementations, the number of the plurality of partitioned convolution operations is determined based on a size of available storage space of the protected storage area.

[0128] In some implementations, storing the input portion in the protected storage area comprises: converting the input into an input matrix based on a size of the set of parameter values and a size of the input, elements in a row or column of the input matrix to be sequentially multiplied with a corresponding parameter value in the set of parameter values in the convolution operation; determining the input portion corresponding to the parameter values subset from the input matrix based on the number of the plurality of partitioned convolution operations and an order of the given convolution operation in the plurality of partitioned convolution operations; and storing the determined input portion in the protected storage area.

[0129] In some implementations, the set of parameter values is stored in a storage area outside the TEE. The acts further comprise: in response to determining that the given partitioned convolution operation is to be performed, loading the parameter values subset from the storage area outside the TEE into the protected storage area.

[0130] In some implementations, loading the parameter values subset into the protected storage area further comprises: performing an integrity check of the parameter values subset in the TEE, including: computing an integrity check value of the parameter values subset, comparing the computed integrity check value with an expected integrity check value stored in the protected storage area, and in response to the computed integrity check value matching the expected integrity check value, confirming the integrity of the parameter values subset; and in response to confirming the integrity of the parameter values subset, storing the parameter values subset into the protected storage area.

[0131] In some implementations, the acts further comprise: after determining the result of the given partitioned convolution operation, removing the parameter values subset from the protected storage area.

[0132] In some implementations, computing the integrity check value for the subset of parameter values includes determining the integrity check value by performing a hash operation on the subset of parameter values.

[0133] In some implementations, loading the subset of parameter values into the protected storage includes loading the subset of parameter values into the protected storage in parallel with execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation.

[0134] In some implementations, performing the integrity check on the subset of parameter values in the TEE includes performing the integrity check on the subset of parameter values in the TEE in parallel with: execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation; and loading of another subset of parameter values into the protected storage, the other subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that follows the given partitioned convolution operation.

[0135] In some implementations, the set of parameter values includes: a set of trained parameter values of the deep learning model, or a set of parameter values of the deep learning model that are being trained.

[0136] In some implementations, the input to the convolution layer is stored in the protected storage, and the actions further include: responsive to the input not being used in subsequent layers of the deep learning model after the output is determined, removing the input from the protected storage.

[0137] In yet another aspect, the present disclosure provides a computer-readable medium having stored thereon machine executable instructions, which when executed by a device, cause the device to perform the method of the above aspect.

[0138] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0139] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the machine to perform the functions / acts specified in the flow diagrams and / or block diagrams. The program code can execute entirely on a machine, partly on a machine, as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0140] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0141] Moreover, while operations can be depicted in a particular, serial order, this should not be understood as requiring or implying that such operations be performed in the particular order shown, or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details have been discussed, such details are not to be interpreted as limiting. Certain features that are described in the context of separate implementations can also be implemented in combination. Conversely, various features that are described in the context of a single implementation can also be implemented separately or in any suitable subcombination. It will be appreciated that various features described herein can form part of a system, an article of manufacture, and / or an article of fabrication.

[0142] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: in response to a convolution operation of a convolutional layer of a deep learning model being triggered, sequentially performing, in a trusted execution environment (TEE) of a computing device, a plurality of partitioned convolution operations based on an input to the convolutional layer and a set of parameter values, the plurality of partitioned convolution operations using different subsets of parameter values partitioned by the set of parameter values, the performing of a given partitioned convolution operation of the plurality of partitioned convolution operations comprising: storing, in a protected storage area of the TEE, an input portion of the input to be processed by the parameter value subset of the given partitioned convolution operation, the input portion represented as a matrix, determining a result of the given partitioned convolution operation through a single matrix multiplication operation of the input portion and the parameter value subset for the given partitioned convolution operation, the parameter value subset represented as a matrix, and removing the input portion from the protected storage area; and determining a result of the convolution operation as an output of the convolutional layer by merging the results of the plurality of partitioned convolution operations, wherein the storing of the input portion in the protected storage area comprises: based on a size of the set of parameter values and a size of the input, converting the input into an input matrix, elements in a row or column of the input matrix to be sequentially multiplied with respective parameter values in the set of parameter values in the convolution operation; based on a number of the plurality of partitioned convolution operations and an order of the given convolution operation in the plurality of partitioned convolution operations, determining the input portion corresponding to the parameter value subset from the input matrix; and storing the determined input portion in the protected storage area.

2. The method of claim 1, wherein the number of the plurality of partitioned convolution operations is determined based on an available storage space size of the protected storage area.

3. The method of claim 1, wherein the set of parameter values is stored in a storage area external to the TEE, the method further comprising: in response to determining that the given partitioned convolution operation is to be performed, loading the parameter value subset from the storage area external to the TEE into the protected storage area.

4. The method of claim 3, wherein the loading of the parameter value subset into the protected storage area further comprises: performing, in the TEE, an integrity check of the parameter value subset, comprising: computing an integrity check value of the parameter value subset, comparing the computed integrity check value with an expected integrity check value stored in the protected storage area, and in response to the computed integrity check value matching the expected integrity check value, confirming the integrity of the parameter value subset; and in response to confirming the integrity of the parameter value subset, storing the parameter value subset into the protected storage area.

5. The method of claim 4, further comprising: after determining the result of the given partitioned convolution operation, removing the parameter value subset from the protected storage area.

6. The method of claim 4, wherein the computing of the integrity check value of the parameter value subset comprises: determining the integrity check value by performing a hash operation on the subset of parameter values.

7. The method of claim 3, wherein loading the subset of parameter values into the protected storage area comprises: loading the subset of parameter values into the protected storage area in parallel with execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation.

8. The method of claim 4, wherein performing integrity checking of the subset of parameter values in the TEE comprises performing integrity checking of the subset of parameter values in the TEE in parallel with: execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation, and loading of another subset of parameter values into the protected storage area, the other subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that follows the given partitioned convolution operation.

9. The method of claim 1, wherein the set of parameter values comprises: a set of trained parameter values of the deep learning model, or a set of parameter values of the deep learning model that are being trained.

10. The method of claim 1, wherein the input to the convolution layer is stored in the protected storage area, the method further comprising: removing the input from the protected storage area in response to the input not being used in subsequent layers of the deep learning model after the output is determined.

11. An electronic device, comprising: a processor; and memory coupled to the processor and containing instructions stored thereon that, when executed by the processor, cause the device to perform the following actions: in response to a convolution operation of a convolution layer of a deep learning model being triggered, sequentially performing, in a trusted execution environment (TEE) of a computing device, a plurality of partitioned convolution operations based on an input to the convolution layer and a set of parameter values, the plurality of partitioned convolution operations using different subsets of parameter values partitioned from the set of parameter values, execution of a given partitioned convolution operation of the plurality of partitioned convolution operations comprising: storing, in a protected storage area of the TEE, an input portion of the input to be processed by a subset of parameter values of the given partitioned convolution operation, the input portion represented as a matrix, determining a result of the given partitioned convolution operation by a single matrix multiplication operation of the input portion and a subset of parameter values for the given partitioned convolution operation, the subset of parameter values represented as a matrix, and removing the input portion from the protected storage area; and determining a result of the convolution operation as an output of the convolution layer by combining results of the plurality of partitioned convolution operations, wherein storing the input portion in the protected storage area comprises: based on a size of the set of parameter values and a size of the input, converting the input into an input matrix, elements in a row or column of the input matrix to be sequentially multiplied with respective parameter values in the set of parameter values in the convolution operation; determining, from the input matrix, an input portion corresponding to the subset of parameter values based on the number of the plurality of partitioned convolution operations and an order of the given convolution operation among the plurality of partitioned convolution operations; and storing the determined input portion in the protected storage.

12. The device of claim 11, wherein the number of the plurality of partitioned convolution operations is determined based on an available storage size of the protected storage.

13. The device of claim 11, wherein the set of parameter values is stored in a storage external to the TEE, the acts further comprising: in response to determining that the given partitioned convolution operation is to be executed, loading the subset of parameter values from the storage external to the TEE into the protected storage.

14. The device of claim 13, wherein loading the subset of parameter values into the protected storage further comprises: performing an integrity check of the subset of parameter values in the TEE, including: computing an integrity check value of the subset of parameter values, comparing the computed integrity check value with an expected integrity check value stored in the protected storage, and in response to the computed integrity check value matching the expected integrity check value, confirming integrity of the subset of parameter values; and in response to confirming integrity of the subset of parameter values, storing the subset of parameter values into the protected storage.

15. The device of claim 14, wherein the acts further comprise: after determining a result of the given partitioned convolution operation, removing the subset of parameter values from the protected storage.

16. The device of claim 14, wherein computing an integrity check value of the subset of parameter values comprises: determining the integrity check value by performing a hash operation on the subset of parameter values.

17. The device of claim 14, wherein performing an integrity check of the subset of parameter values in the TEE comprises performing the integrity check of the subset of parameter values in the TEE in parallel with: execution of a partitioned convolution operation of the plurality of partitioned convolution operations that precedes the given partitioned convolution operation, and loading of another subset of parameter values into the protected storage, the other subset of parameter values to be used in a partitioned convolution operation of the plurality of partitioned convolution operations that follows the given partitioned convolution operation.

18. A computer program product tangibly stored in a computer storage medium and comprising machine executable instructions that, when executed by a device, cause the device to perform the method of any one of claims 1 to 10.

Images