Artificial intelligence device based on trust environment
The trust environment-based AI device addresses security and performance issues in DNN frameworks by using encrypted data processing, direct convolution, and pipelining to enhance security and accelerate DNN execution in mobile devices.
Patent Information
- Application Number
- JP2023129058
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2023-08-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-08-08
AI Technical Summary
Existing DNN frameworks in mobile and embedded devices face security vulnerabilities from physical attacks and suffer from slow memory access and high data encryption/decryption overhead, leading to increased execution time.
A trust environment-based artificial intelligence device that uses encrypted data processing, direct convolution, DNN-friendly SRAM management, cryptographic hardware offloading, and intra-layer pipelining to enhance security and accelerate DNN execution.
The device securely accelerates DNN execution by reducing memory accesses, offloading encryption/decryption tasks, and optimizing SRAM usage, resulting in significant speed and energy efficiency improvements.
Smart Images

Figure 2024160912000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an artificial intelligence neural network framework technology, and more particularly to a trust environment-based artificial intelligence device that can safely accelerate the execution of an artificial neural network in a trust environment. [Background technology]
[0002] A DNN (Deep Neural Network) is an artificial neural network that consists of an input layer, an output layer, and several hidden layers in between. DNNs are widely used in mobile and embedded applications. In particular, DNNs are useful in applications that perform biometric authentication, using a user's biological characteristics (e.g., fingerprint, iris, face, etc.) to verify the user's identity.
[0003] Because DNNs contain a lot of sensitive user data, mobile and embedded devices must provide a secure DNN execution environment that can safely protect user and DNN data from security attacks.
[0004] Previously, it was proposed to run DNNs in a Trusted Execution Environment via TrustZone, a hardware-based security technology used in ARM (Advanced RISC Machine) processors. TrustZone is a hardware-based security technology that protects important information by placing separate, independent security zones within a processor. However, running DNNs in the TrustZone alone cannot fully protect data, as the TrustZone has limited memory protection capabilities. TrustZone allows hardware and software resources to be divided into a secure general zone to isolate the execution of DNNs from other processes. The general zone is prevented by hardware from accessing data in the security zone. However, because TrustZone keeps data unencrypted in volatile memory, physical security attacks such as cold boot attacks can obtain sensitive user and DNN data despite the execution of DNNs in the TrustZone.
[0005] To protect data from physical attacks, one can choose to encrypt data and decrypt it only in secure on-chip memory. Such a scheme can not only isolate the execution of the DNN from other processes using a trust zone, but also protect user and DNN data from physical attacks through encryption. However, protecting encrypted data in memory causes problems such as a large increase in slow memory accesses and a large increase in the execution time of the DNN due to the high data encryption / decryption overhead imposed on the processor.
[0006] Therefore, a new security DNN framework is needed that not only protects sensitive user and DNN data from physical attacks, but also overcomes the slow memory access and high data encryption / decryption overhead, and reduces the execution time of DNNs. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] Korean Patent No. 10-2474875 Summary of the Invention [Problem to be solved by the invention]
[0008] The present invention has been made in consideration of the above-mentioned problems in the conventional technology, and an object of the present invention is to provide a trust environment-based artificial intelligence device that can safely accelerate the execution of an artificial neural network in a trust environment. [Means for solving the problem]
[0009] In order to achieve the above-mentioned objective, a trust environment-based artificial intelligence device according to one aspect of the present invention comprises a first type of memory that transmits encrypted input data and receives encrypted output data, and a trust artificial intelligence processing unit that operates in a trust trust space and performs artificial intelligence calculations on the encrypted input data and the encrypted output data, wherein the trust artificial intelligence processing unit includes a cryptographic processing front-end processor that generates decrypted input data by decrypting the encrypted input data and encrypts unencrypted output data to generate the encrypted output data, a second type of memory that provides a buffer for the decrypted input data and the unencrypted output data, and a processor that performs neural network calculations based on the decrypted input data to generate the unencrypted output data.
[0010] The cryptographic processing front-end processor may receive encrypted input activations and encryption filters as the encrypted input data from the first type of memory, and store decryption input activations and decryption filters in the second type of memory. The cryptographic front-end processor may receive an on-demand request by the processor during the artificial intelligence calculation and access the first type of memory to retrieve the encrypted input data. The second type of memory may have a relatively faster operating speed and a smaller storage capacity than the first type of memory. The processor may perform direct convolution-based neural network operations to reduce the number of accesses to the first type of memory. The processor may statically store decryption input activations in the second type of memory, and store decryption filters and non-encrypted output activations in a circular queue fashion. The processor may transmit and receive data to and from the first type memory and the second type memory via interrupt driven offloading of the cryptographic front-end processor. The processor can transmit and receive data between the cryptographic front-end processor and the first type memory and the second type memory via a DMA-driven offload of a Direct Memory Access (DMA) controller. The processor may perform the neural network operations to overlap with the performance of encryption and decryption operations of the cryptographic front-end processor to achieve intra-layer pipelining. The processor may perform the neural network operations uninterruptedly by having the cryptographic front-end processor perform decryption operations on the encrypted input data while the neural network operations are being performed. For the intra-layer pipelining, the processor can seamlessly perform the neural network operation by subdividing it into a data decoding step, an operation step, and a data encryption step.
[0011] In order to achieve the above-mentioned objective, a trust environment-based artificial intelligence device according to another aspect of the present invention comprises a first type of memory that transmits encrypted input data, and a trust artificial intelligence processing unit that operates in a trust trust space and performs artificial intelligence calculations on the encrypted input data, wherein the trust artificial intelligence processing unit includes a cryptographic processing front-end processor that generates decrypted input data by decrypting the encrypted input data, a second type of memory that provides a buffer for the decrypted input data and unencrypted output data, and a processor that performs neural network calculations based on the decrypted input data to generate the unencrypted output data.
[0012] The processor may perform direct convolution-based neural network operations to reduce the number of accesses to the first type of memory. The processor may statically store decryption input activations in the second type of memory, and store decryption filters and non-encrypted output activations in a circular queue fashion. The processor may perform the neural network operations to overlap with the performance of encryption and decryption operations of the cryptographic front-end processor to achieve intra-layer pipelining. Effect of the Invention
[0013] The technology disclosed in this specification has the following effects. However, this does not mean that a specific embodiment must include all of the following effects or that it must include only the following effects, and therefore the scope of the disclosed technology should not be understood to be limited thereby.
[0014] According to the trust environment-based artificial intelligence device of the present invention, it is possible to safely accelerate the execution of an artificial neural network in a trust environment.
[0015] Additionally, data encryption can be performed in the trust space to enhance security against physical attacks, and direct convolution-based neural network operations can be performed to reduce the number of memory accesses.
[0016] Additionally, the task of running a neural network can be offloaded to cryptographic hardware to take advantage of limited processor resources, and can be overlapped with data encryption and decryption via intra-layer pipelining to reduce the execution time of an artificial neural network. [Brief description of the drawings]
[0017] [Figure 1] FIG. 2 is a diagram illustrating a trust environment-based artificial intelligence device according to the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a method for executing a trust environment-based artificial intelligence operation in the artificial intelligence device of FIG. 1. [Diagram 3] FIG. 2 is a diagram illustrating a trust execution environment. [Figure 4a] FIG. 1 is a diagram illustrating the working model of the DNN framework according to the present invention compared with the prior art. [Figure 4b] FIG. 1 is a diagram illustrating the working model of the DNN framework according to the present invention compared with the prior art. [Diagram 5] 1 shows the execution speed, bandwidth, and decoding throughput of a conventional DNN framework, respectively. [Figure 6] 1 shows the execution speed, bandwidth, and decoding throughput of a conventional DNN framework, respectively. [Figure 7] 1 shows the execution speed, bandwidth, and decoding throughput of a conventional DNN framework, respectively. [Figure 8a] FIG. 1 illustrates the convolutions used to perform neural network operations. [Figure 8b] FIG. 1 illustrates the convolutions used to perform neural network operations. [Figure 9a]FIG. 1 illustrates a DNN-friendly SRAM management process. [Figure 9b] FIG. 1 illustrates a DNN-friendly SRAM management process. [Figure 10] FIG. 13 illustrates the throughput between a CPU and cryptographic hardware. [Figure 11] FIG. 1 is a diagram illustrating cryptographic hardware offloading. [Figure 12] FIG. 1 is a diagram illustrating the realization of intra-layer pipelining in the process of performing neural network operations. [Figure 13] FIG. 1 is a diagram illustrating the realization of intra-layer pipelining in the process of performing neural network operations. [Figure 14] FIG. 2 is a diagram illustrating an example of a software configuration according to the present invention. [Figure 15] 11A to 11C are diagrams illustrating experimental results according to the present invention. [Figure 16] 11A to 11C are diagrams illustrating experimental results according to the present invention. [Figure 17] 11A to 11C are diagrams illustrating experimental results according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] The present invention was filed with the support of the following research project: [National research and development project that supported this invention] [Project unique number] 1711193986 [Project Number] 2020-0-01361-004 [Ministry name] Ministry of Science, ICT and Technology [Name of issue management (specialized) organization] Information and Communications Planning and Evaluation Agency [Research Project Name] Information, Communications, and Broadcasting Innovation Human Resource Development [Research title] Artificial Intelligence Graduate School Support (Yonsei University) [Contribution rate] 1 / 2 [Name of project implementation organization] Yonsei University Industry-Academic Cooperation Group [Research period] 2023.01.01~2023.12.31 [National research and development project that supported this invention] [Project unique number] 1415177659 [Issue number] P0016150 [Ministry name] Ministry of Trade, Industry and Energy [Name of issue management (specialized) organization] Korea Institute for Industrial Technology Advancement [Research Project Name] International Cooperation in Industrial Technology (R&D) [Research Project Title] Development of attack detection and defense technologies for smart IoT network security [Contribution rate] 1 / 2 [Name of project implementing organization] Korea Electronics Research Institute [Research period] 2021.12.01~2022.11.30
[0019] The description of the present invention is merely an embodiment for structural or functional description, and therefore the scope of the present invention should not be interpreted as being limited by the embodiment described in the present specification. In other words, since the embodiment can be modified in various ways and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the objectives or effects presented in the present invention do not mean that a specific embodiment should include all of them or should include only such effects, and therefore the scope of the present invention should not be understood as being limited thereby.
[0020] Meanwhile, the meanings of the terms used in this specification should be understood as follows.
[0021] The terms "first", "second", etc. are used to distinguish one component from another component, and should not be used to limit the scope of the rights. For example, the first component may be named the second component, and similarly, the second component may be named the first component.
[0022] When referring to an element being "connected" to another element, it should be understood that it may be directly connected to the other element, but there may also be other elements in between. Conversely, when referring to an element being "directly connected" to another element, it should be understood that there are no other elements in between. Meanwhile, other expressions describing the relationship between elements, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.
[0023] Singular expressions shall be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprise" or "have" shall be understood to be intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof that are performed, and shall not be understood to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.
[0024] In each step, the identification numbers (e.g., a, b, c, etc.) are used for convenience of description, and the identification numbers do not describe the order of each step, and each step may occur in a different order than specified unless the context clearly dictates a specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.
[0025] The present invention is embodied as computer readable codes in a computer readable recording medium, and the computer readable recording medium includes all kinds of recording devices in which data read by a computer system is stored. Examples of the computer readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Also, the computer readable recording medium may be distributed among computer systems connected by a network, so that the computer readable codes may be stored and executed in a distributed manner.
[0026] All terms used herein, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries must be interpreted to be consistent with the meaning they have in the context of the relevant art, and cannot be interpreted as having an ideal or overly formal meaning unless expressly defined in this specification.
[0027] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, specific examples of the embodiments of the present invention will be described in detail with reference to the drawings.
[0028] FIG. 1 is a diagram illustrating a trust environment-based artificial intelligence device according to the present invention.
[0029] Referring to FIG. 1, a trust environment-based artificial intelligence device 100 is a framework for enhancing security and accelerating the execution of artificial neural networks on mobile or embedded devices, and is realized including a first type of memory 110 and a trust artificial intelligence processing unit 130.
[0030] The first type memory 110 transmits encrypted input data and receives encrypted output data. Here, the first type memory 110 is configured with a DRAM (Dynamic Random Access Memory), but is not necessarily limited to this.
[0031] The trust artificial intelligence processing unit 130 operates in the trust trust space and performs artificial intelligence calculations on the encrypted input data and the encrypted output data. To this end, the trust artificial intelligence processing unit 130 includes a cryptographic processing front-end processor 131, a second type of memory 133, and a processor 135.
[0032] The cryptographic processing front-end processor 131 generates decrypted input data by decrypting encrypted input data as cryptographic hardware, and encrypts non-encrypted output data to generate encrypted output data. The cryptographic processing front-end processor 131 receives encrypted input activations, which are encrypted input data, and encryption filters from the first type memory 110, and stores decrypted input activations and decryption filters in the second type memory 133. The cryptographic processing front-end processor 131 receives an on-demand request from the processor 135 during the artificial intelligence calculation process, and accesses the first type memory 110 to retrieve encrypted input data.
[0033] The second type memory 133 provides a buffer for decrypted input data and unencrypted output data. Here, the second type memory 133 has a relatively faster operating speed and smaller storage capacity than the first type memory 110. For example, if the first type memory 110 is configured with a DRAM, the second type memory 133 is configured with a Static Random Access Memory (SRAM). When data is transmitted and received between the first type memory 110 and the second type memory 133, the cryptographic processing front-end processor 131 encrypts and decrypts the data to ensure accuracy and security.
[0034] The processor 135 performs neural network operations based on the decrypted input data to generate unencrypted output data. The processor 135 performs direct convolution-based neural network operations to reduce the number of accesses to the first type memory 110. That is, the processor 135 uses direct convolution to minimize the working set size of the neural network operations, and the reduced working set size reduces accesses to the first type memory 110, which has a relatively slower operating speed than the second type memory 133, and calls for data encryption and decryption during the neural network operations.
[0035] The processor 135 stores the decrypted input activations in a fixed manner in the second type memory 133, and stores the decrypted filters and the non-encrypted output activations in a circular queue manner. The processor 135 efficiently manages the second type memory 133 based on the access patterns of the input activations, filters, and output activations of the convolutional layer in executing the neural network operation. The processor 135 stores the input activations in a fixed manner since they tend to increase data reuse, stores the filters in a circular queue manner since the same filter is used multiple times when generating an output channel but the size is much smaller than the input activations and only one output channel is required to generate, and stores the output activations in a circular queue manner since they are one-time write data with high spatial intensities.
[0036] The processor 135 transmits and receives data to and from the first type memory 110 and the second type memory 131 through interrupt driven offloading of the cryptographic front-end processor 131. The processor 135 also transmits and receives data to and from the cryptographic front-end processor 131, the first type memory 110, and the second type memory 133 through DMA driven offloading of a Direct Memory Access (DMA) controller. The processor 135 selects between interrupt driven offloading and DMA driven offloading. Interrupt driven offloading is suitable for latency-sensitive tasks, and DMA driven offloading is suitable for processing larger amounts of data at once. Here, both offloading mechanisms accelerate neural network operations.
[0037] The processor 135 executes neural network operations so as to overlap with the execution of encryption and decryption operations of the cryptographic front-end processor 131, thereby realizing intra-layer pipelining. The processor 135 executes neural network operations seamlessly by making the cryptographic front-end processor 131 execute decryption operations of encrypted input data while executing neural network operations. For intra-layer pipelining, the processor 135 executes neural network operations seamlessly by subdividing the operations into a data decryption step, an operation step, and a data encryption step. The processor 135 parallelizes the data decryption step, the operation step, and the data encryption step through intra-layer pipelining.
[0038] FIG. 2 is a flowchart illustrating a method for performing a trust environment-based artificial intelligence operation in the artificial intelligence device of FIG.
[0039] 2, the artificial intelligence device 100 receives encrypted input data from the first type memory 110 and generates decrypted input data through decryption (step S210). The artificial intelligence device 100 decrypts the encrypted input data through the cryptographic processing front-end processor 131. The artificial intelligence device 100 stores the decrypted input data in the second type memory 133.
[0040] The artificial intelligence device 100 performs neural network calculations based on the decrypted input data to generate unencrypted output data (step S230). Here, the artificial intelligence device 100 performs convolution-based neural network calculations directly through the processor 135 to reduce the number of accesses by the cryptographic front-end processor 131 to the first type memory 110 to retrieve encrypted input data. The artificial intelligence device 100 stores the unencrypted output data in the second type memory 133.
[0041] The artificial intelligence device 100 encrypts the unencrypted output data to generate encrypted output data (step S250). The artificial intelligence device 100 encrypts the unencrypted output data via the encryption processing front-end processor 310. The artificial intelligence device 100 outputs the encrypted output data to the first type memory 110.
[0042] The artificial intelligence device 100 accelerates neural network operations by executing the neural network operations of the processor 135 so as to overlap with the execution of encryption and decryption operations of the cryptographic processing front-end processor 131, thereby realizing intra-layer pipelining in the data decryption step, the operation step, and the data encryption step.
[0043] The trust environment-based artificial intelligence device according to the present invention will be described in more detail below with reference to FIGS.
[0044] FIG. 3 is a diagram illustrating a trust execution environment.
[0045] Referring to Figure 3, a Trusted Execution Environment (TEE) is a security processing environment with processing, memory, and storage capabilities that restricts sensitive tasks and data from leaving the environment, achieving high security. Tasks and data in the TEE are isolated in a Rich Execution Environment (REE) where an Operating System (OS) and applications are executed.
[0046] To realize a Trust Zone supported TEE for use in mobile and embedded devices, the Trust Zone implements a security processor mode that allows the CPU to operate exclusively in only one of the TEE and the REE at a given time. The switching and interaction between the TEE and the REE is managed by a security monitor that can be invoked by SMC (Secure Monitor Calls). The Trust Zone also divides the DRAM into secure and non-secure areas and does not allow the REE to access the secure area to protect sensitive data in the TEE.
[0047] Existing DNN frameworks running on a trust zone-supported TEE start with the REE receiving input data from a peripheral device (e.g., a fingerprint image from a fingerprint sensor). After running some initial layers of the DNN in the REE, the REE encrypts the output activations and sends them to the TEE via the SMC. The TEE then decrypts the sent activations and executes the remaining layers using the decrypted activations and pre-sent filters. After completing the execution of the remaining layers, the REE returns the predictions made by the DNN to the REE via the SMC.
[0048] This approach isolates the execution of the DNN and protects it from several security attacks.
[0049] 4a and 4b are diagrams illustrating the working model of the DNN framework of the present invention compared with the conventional one, where FIG. 4a is the conventional DNN framework and FIG. 4b is the DNN framework proposed in the present invention (hereinafter referred to as GuardianNN).
[0050] For the conventional DNN framework in Figure 4a, first all input data and filters of the DNN are encrypted and stored in DRAM. When the framework starts executing a layer of the DNN, it 1) loads the encrypted input activations and filters into SRAM and 2) uses the CPU to decrypt the data. Then, the framework 3) executes each layer using the decrypted data and stores the output activations in SRAM. After that, the framework 4) encrypts the output activations and 5) stores them in DRAM. The input activations and filters are replaced in SRAM as needed. When the SRAM is smaller than the working set of the layer, data swapping and encryption / decryption often occur. Using encrypted DRAM to execute the DNN in the TEE allows the conventional DNN framework to fully protect sensitive user and DNN data from security attacks, but it can have difficulties with slow DNN execution.
[0051] 5 to 7 show the execution speed, bandwidth, and decoding throughput of the conventional DNN framework, respectively.
[0052] As shown in Figure 5, which is a comparison of the DNN execution speed between the conventional DNN framework (OP-TEE with Pager) in Figure 4a and the insecure DarkneTZ working model that does not encrypt DRAM data, the conventional DNN framework's DNN execution speed is significantly slower than DarkneTZ. The conventional DNN framework took 3.42 times longer to run AlexNet than DarkneTZ.
[0053] In conventional DNN frameworks, the limited capacity of embedded SRAM increases the number of accesses to slow DRAM, and the performance bottleneck phenomenon due to the high data encryption and decryption overhead imposed on the CPU affects the execution speed of DNN. Specifically, conventional DNN frameworks utilize embedded SRAM as a secure on-chip memory where TEE securely loads encrypted DRAM data and decrypts the loaded data. However, since the capacity of SRAM (hundreds of KB) is generally smaller than the on-chip CPU cache (hundreds of KB) in mobile and embedded devices, using SRAM as a secure on-chip buffer reduces the effective on-chip memory size by an order of magnitude. This increases the number of accesses to slow DRAM when executing DNN, slowing down the execution speed of DNN.
[0054] As shown in Figure 6, the bandwidth comparison between the secure SRAM and off-chip DRAM for various working set sizes shows that the SRAM provides a sustained higher bandwidth than the DRAM for various working set sizes. This means that increased DRAM accesses have a negative impact on the execution speed of the DNN. The results also suggest that sequential accesses should be preferred over arbitrary accesses to achieve higher bandwidth. As a result, to minimize the negative impact of slow DRAM accesses on the execution speed of the DNN, the memory accesses of the DNN execution should be structured as sequential accesses while maximizing the reuse of fast SRAM data.
[0055] When exchanging data between embedded SRAM and encrypted DRAM, data must be encrypted and decrypted by the CPU to ensure functional correctness and high security. However, CPU-based encryption and decryption is not only slow, but also consumes a significant portion of the limited CPU bandwidth, which is all shared between the computationally intensive DNN execution and encryption and decryption, slowing down the execution speed of the DNN.
[0056] As shown in Fig. 7, the throughput of CPU-based data decryption with various payload sizes shows that data encryption and decryption are also difficult due to limited CPU resources, achieving a throughput of 4.51 MB / s for a 2 KB payload. Compared with the throughput of sequential access to 2 KB DRAM data (609.24 MB / s, Fig. 6 (a)), CPU-based encryption and decryption of 2 KB data is 135.09 times slower than sequential DRAM access. The low data encryption and decryption throughput indicates that the overhead of data encryption and decryption is considerable, making it difficult for conventional DNN frameworks to achieve fast DNN execution. As a result, conventional DNN frameworks have considerable difficulties due to the high overhead of DRAM data encryption and decryption imposed on limited CPU resources, and the overhead must be resolved for fast and secure DNN execution.
[0057] Therefore, the present invention solves the slow DNN execution problem of conventional DNN frameworks and proposes GuardiaNN, a fast and secure DNN framework for mobile and embedded devices. The GuardiaNN proposed in the present invention has the following features.
[0058] -The working set size of a DNN run can be minimized through direct convolution. Reducing the working set size can reduce slow DRAM access and data encryption and decryption requirements during the run of the DNN.
[0059] -DNN-friendly SRAM management is used to maximize data reuse in convolutional layers. Input activations are pinned to SRAM, and filters and output activations are stored in a circular queue to efficiently manage SRAM.
[0060] - Data encryption and decryption can be offloaded to cryptographic hardware. Offloading data encryption and decryption allows limited CPU resources to be dedicated to DNN.
[0061] -The DNN layer can be overlapped with data encryption and decryption operations to further accelerate the execution of DNN. This is possible because the CPU and cryptographic hardware can simultaneously perform the DNN and data encryption and decryption operations, respectively.
[0062] Looking at FIG. 4b, the working model of the DNN framework proposed in the present invention is as follows: 1) When executing the DNN layer, the cryptographic hardware used in mobile and embedded devices (the cryptographic processing front-end processor in FIG. 1) loads the encrypted input activations and filters in DRAM, decrypts the data, and stores the decrypted data in SRAM. Then, 2) the CPU (the processor in FIG. 1) uses the SRAM data to perform the work of the DNN layer and stores the output activations in SRAM (the second type of memory in FIG. 1). After that, 3) the cryptographic hardware encrypts the output activations stored in SRAM, and stores the encrypted output activations in DRAM (the first type of memory in FIG. 1).
[0063] Compared with the working model of the conventional DNN framework in Fig. 4a, the use of direct convolution and DNN-friendly SRAM management can significantly reduce the high data encryption and decryption overhead imposed on the CPU using slow DRAM access and cryptographic hardware. The impact of the main features proposed in this invention on the execution of DNN can be summarized in Table 1 below.
[0064] [Table 1]
[0065] The present invention uses direct convolution to reduce the working set size of convolutional layers, and uses DNN-friendly SRAM management to maximize SRAM data reuse, which can significantly reduce slow DRAM accesses during DNN execution.
[0066] 8a and 8b are diagrams illustrating the convolutions used to perform neural network operations, where FIG. 8a is an im2col (Image to Column) convolution and FIG. 8b is a direct convolution.
[0067] Conventional mobile and embedded DNN frameworks (e.g., DarkneTZ, TensorFlow Lite) use im2col (Image to Column) convolution to perform time-consuming convolution layers. im2col is a function that converts multidimensional data into a matrix to perform matrix operations. Convolution of multidimensional data is the same as the inner product of data converted into a matrix via im2col. im2col convolution flattens each patch (i.e., a set of input activations whose elements are mapped to elements of a filter) into a 2D matrix as shown in Figure 8a, and performs matrix multiplication between the flattened patch and the filter to achieve fast convolution layer execution. However, merging patches significantly increases the working set size of the interrupt layer because additional buffers must be allocated to store the merged patches. When the working set size increases due to the limited capacity of SRAM (up to a few hundred KB), many slow DRAM accesses occur, which significantly slows down the execution speed of DNNs. Therefore, to achieve fast DNN execution, the working set size of the convolution layer must be minimized.
[0068] In order to minimize the working set size of the convolutional layer, we use direct convolution instead of im2col convolution. Direct convolution performs a correlation operation on the input with the filter of the convolutional layer, and the result of calculating the correlation value in all regions of the filter in a sliding window manner is the output. Direct convolution does not increase the working set size because it incorporates the necessary input activations when requesting them, as shown in Figure 8b. In addition to the reduced working set size, direct convolution tends to increase the reuse of SRAM data due to the inherent temporal and spatial locality of input activations. Since direct convolution spatially slides the filter over the input activations to generate channels of output activations, input activations adjacent to recently accessed input activations are more likely to be accessed in the near future. In the case of filters, filters have high temporal locality because the same filter is used multiple times when generating output channels. The high spatial and temporal aggregation of direct convolution coupled with demand paging built-in SRAM helps to significantly reduce slow DRAM accesses.
[0069] 9a and 9b are diagrams illustrating a DNN-friendly SRAM management process.
[0070] As shown in Figure 9a, Pager, an existing open source TEE framework that applies ARM Trust Zone technology, uses demand paging to manage the limited capacity of security embedded SRAM. Demand paging loads data from encrypted DRAM to SRAM as needed and pushes the nearest unused SRAM data to DRAM to reclaim SRAM space. Demand paging efficiently exploits the temporal locality of data, but it cannot recognize the various access patterns of input activations, filters, and output activations of convolutional layers. In addition, the size of SRAM is only a few hundred KB in mobile and embedded devices, so it is not possible to store all the data required for convolutional layers in SRAM. Such inefficient SRAM management significantly increases the number of accesses to slow DRAM and hinders the execution of fast DNNs. Therefore, to prevent SRAM from inducing a large number of slow DRAM accesses, the small SRAM must be efficiently managed by integrating the various data access patterns of convolutional layers.
[0071] As shown in Fig. 9b, the present invention realizes DNN-friendly SRAM management to maximize the reuse of SRAM data. Here, the DNN-friendly SRAM management exploits the different access patterns of input activations, filters, and output activations of a convolutional layer. The present invention first pins all input activations to SRAM when repeatedly accessing them throughout the execution of a convolutional layer. An input activation accessed to generate an output channel must be accessed again for all other output channels. Then, the remaining SRAM space is configured with two circular queues, one for filters and one for output activations. Filters are frequently accessed data like input activations. However, since their size is much smaller than input activations and filters are needed to generate only one output channel, storing filters in a circular queue alone can fully exploit the high temporal locality. Output activations are one-time write data with high spatial intensities, so they are suitable for circular queues. By managing SRAM in a DNN-friendly manner, we can maximize SRAM data reuse and minimize slow DRAM accesses, accelerating the execution of DNNs.
[0072] The present invention uses cryptographic hardware to encrypt and decrypt data, allowing limited CPU resources to be fully used for running DNNs. This not only fully dedicates CPU resources to running DNNs, but also leverages the high performance of cryptographic hardware to achieve fast DNN execution. Data encryption and decryption occurs every time the SRAM loads and stores data in the encrypted DRAM, consuming a significant amount of limited CPU resources. To overcome the high overhead of data encryption and decryption, encryption and decryption work is performed using cryptographic hardware. The high overhead data encryption and decryption can be offloaded to cryptographic hardware, allowing limited CPU resources to be fully dedicated to accelerating the execution of DNNs (see FIG. 10).
[0073] FIG. 11 is a diagram illustrating cryptographic hardware offloading, where (a) shows interrupt-based offloading and (b) shows DMA (Direct Memory Access)-driven offloading.
[0074] The present invention is selected from two hardware offload mechanisms, interrupt-based offload (a) and DMA-driven offload (b) in FIG. 11. The two offload mechanisms accelerate different peripheral usage scenarios. Interrupt-based offload is suitable for latency-sensitive work on peripherals, as it generates an interrupt as soon as the peripheral completes a given task. In contrast, DMA-driven offload is suitable for processing larger amounts of data at a time. DMA handles all data transfers and peripheral calls to process data without CPU intervention, and generates an interrupt after all data is processed. Since DNN-friendly SRAM management requires large amounts of data transfers (typically tens of KB) between SRAM and DRAM, DMA-based offload can be advantageous, as it achieves higher throughput with larger payload sizes than interrupt-based offload.
[0075] Using cryptographic hardware allows further performance optimization by fully dedicating limited CPU resources to DNN execution and accelerating DNN execution. The overlapping DNN execution and data encryption and decryption are performed on the CPU and cryptographic hardware, respectively. One attribute of convolutional layers is that the work of generating different output channels is independent of each other. A convolutional layer uses one filter to generate one output channel, and the input activations are shared as read-only data between all output channels. This attribute also applies to pooling layers. Each output channel of a pooling layer requires only that input channel, making the output channel work data-parallel. Based on this, we divide the execution of a large amount of output channels of a convolutional layer into three pipeline steps: a data decryption step, a computation step, and a data encryption step. Then, we pipeline the three steps of the different bulks of the output channels to achieve faster DNN execution. This is called intra-layer pipelining. This is because the various output channel work of a convolutional layer is connected by a pipeline.
[0076] FIG. 12 illustrates the implementation of intra-layer pipelining in the execution process of neural network operations, and shows the benefits of applying intra-layer pipelining to the sixth layer of AlexNet, which generates 256 output channels (including a large number of 32 output channels due to the limited capacity of SRAM) and contributes most to the execution latency of AlexNet.
[0077] In FIG. 12, the data decryption step first decrypts the input activations and / or filters required to generate a large number of output channels and loads them into SRAM. Then, the computation step uses the input activations and filters to compute a large number of output channels. Then, the data encryption step encrypts the large number of output channels and stores them in DRAM. Without intra-layer pipelining, the three steps are serialized as in (a) and it takes 57 ms to execute the layer. In contrast, the intra-layer pipelining, which pipelines three steps of different bulks of output channels, takes only 43 ms to execute the layer as in (b), which is 24.6% faster than the case of (a). This example shows that intra-layer pipelining can achieve faster DNN execution. FIG. 13 is an algorithm showing pseudocode for applying intra-layer pipelining to a convolutional layer.
[0078] However, applying intra-layer pipelining to convolutional layers consumes more SRAM capacity than executing layers without pipelines. Larger capacity is required to guarantee the functional correctness of intra-layer pipelining. To overlap the computation step of the output channel of a large amount with the data decoding step of the output channel of the next large amount, SRAM buffers for the two large amounts must be allocated simultaneously. If a larger SRAM buffer is required, the output channels of the layer are grouped into more bulks, which may slow down the execution speed of the DNN. However, despite the need for a larger SRAM buffer, the performance advantage of intra-layer pipelining is that there are fewer output channels per bulk, which outweighs the potential performance degradation. Therefore, we basically activate intra-layer pipelining for convolutional and pooling layers. However, for mobile and embedded devices equipped with very small embedded SRAM, intra-layer pipelining can be deactivated to prevent potential performance degradation.
[0079] FIG. 14 is a diagram illustrating an example of the software configuration required to implement the present invention on a mobile or embedded device equipped with a secure embedded SRAM and cryptographic hardware.
[0080] In FIG. 14, the present invention is implemented on OP-TEE, an open source TrustZone-based implementation, which is one of the most widely used TEE implementations, but is not limited to this and can be implemented on all TrustZone-based TEE implementations. At the heart of the present invention is the GuardiaNN runtime, a trusted application that executes layers of a DNN to generate predictions for a given input in the TEE. The execution of the DNN is initiated from the REE. First, the REE sends the description (e.g., number of layers, input and output activation size per layer), encrypted input data, and filters of the DNN to the TEE memory and invokes the GuardiaNN runtime. The GuardiaNN runtime then executes each layer of the DNN using helper functions that implement different types of layers of the DNN. The GuardiaNN runtime then executes the DNN and returns the predictions made to the REE again.
[0081] Direct convolution can only be achieved within the GuardiaNN runtime, which must interact with a trusted OS (e.g., OP-TEE) to allocate SRAM and use cryptographic hardware and DMA. To achieve DNN-friendly SRAM management and intra-layer pipelining, the GuardiaNN runtime must allocate buffers in SRAM that cannot be swapped out to DRAM. For this, we extend TEE Internal Core API function for TEE memory allocation, TEE_Malloc(), with an additional input argument, isSRAM. If the value of isSRAM is true, the trusted OS allocates SRAM buffers to prevent the buffers from being swapped out to DRAM. We also extend the memory management of the trusted OS (e.g., Pager) to call TEE_Malloc() with isSRAM set to true for DRAM and prevent it from replacing the allocated SRAM buffers. The default value of isSRAM is set to false to ensure functional correctness of existing trusted applications that are not aware of isSRAM. For example, calling TEE_Malloc(1024, hint, true) will allocate a 1KB SRAM buffer in DRAM that will never be removed, where hint provides some hint about the characteristics of the buffer (e.g., it will be filled with zeros).
[0082] For DMA-driven data encryption and decryption offload, the GuardiaNN runtime calls user-specified system calls defined in the DMA device driver of the trusted OS. The GuardiaNN implementation extends the trusted OS to provide two user-specified system calls, EncryptData() and DecryptData(). The EncryptData() system call takes as input the encryption context (including encryption type, key size, etc.), SRAM start address, DRAM start address, and data size. The system call then flushes the CPU cache to remove dirty cache lines from SRAM, reads the unencrypted SRAM data at the SRAM start address, encrypts the data using the encryption context and crypto hardware, and stores the encrypted data in DRAM at the DRAM start address. In a similar manner, the DecryptData() system call takes as input four factors: encryption context, DRAM start address, SRAM start address, and data size. It then reads the encrypted DRAM data, decrypts the data, and stores the decrypted data in SRAM by a procedure similar to the EncryptData() system call.
[0083] Using two user-specified system calls together with the extended TEE_Malloc() API functions and the existing GlobalPlatform API, the present invention can be fully implemented on mobile and embedded devices.
[0084] ≪EVALUATION≫
[0085] <Experimental Setup>
[0086] To evaluate the effect of GuardiaNN on fast and secure DNN execution, we prototyped GuardiaNN on an STM32MP157C-DK2 development board and compared the DNN execution speed and energy consumption with the basic secure DNN framework. The development board is officially supported by OP-TEE, an open source TrustZone-based TEE realization, and reflects the typical hardware configuration of modern embedded devices. It consists of a dual-core ARM Cortex-A7 CPU, 256KB of security embedded SRAM, cryptographic hardware, and 512MB DDR3L DRAM. We use OP-TEE v3.11.0 as the trusted OS with Pager supporting GlobalPlatform TEE Client API v1.1 and Internal Core API v1.0. The implementation of GuardiaNN and the basic framework uses only one CPU core, as OP-TEE does not currently support multithreading within a single TEE instance. We extended the implementation of the DNN layer in DarkneTZ with ARM NEON single-command multiple data commands to exploit data parallelism between each DNN layer task. The implementation of the enhanced DNN layer is fully applied to GuardiaNN and the basic framework. We assume that the trusted OS allocates all SRAM (except for 4KB reserved by OP-TEE in shared memory between TEE and REE) to GuardiaNN and the basic framework. For energy consumption comparison, we use Monsoon HVPM (High Voltage Power Monitor) to measure the energy consumption of the whole device.
[0087] We select eight DNNs quantized using 8-bit integer quantization as benchmarks to address five representative mobile and embedded application domains related to sensitive user and DNN data. The five domains are image classification, face recognition, fingerprint recognition, gaze tracking, and emotion recognition. There are a variety of DNNs for each domain. However, we select representative lightweight DNNs in each domain that have a reasonably low execution latency to run. For example, we do not include ResNet-18, a DNN for image classification, in the benchmark because it took 192 seconds for the base framework to run the DNN on the STM32MP157C-DK2 development board. Table 2 below lists the eight DNNs and their characteristics.
[0088] [Table 2]
[0089] <Fast DNN Execution>
[0090] First, we measure the execution latency of all selected DNNs to evaluate the execution speed of GuardiaNN's DNNs. To analyze the contribution of the proposed techniques, we start from the basic framework and apply each proposed technique incrementally to measure the execution latency of the DNNs. Working on direct convolution and DNN-friendly SRAM management, cryptographic hardware, and intra-layer pipelining, we provide a total of five configurations, which are displayed with five bars in Figure 15. We use a bulk size of eight output channels for intra-layer pipelining in the experiments.
[0091] Fig. 15(a) shows the experimental results of latency. Most DNNs run extremely slow in the basic framework, with a maximum latency of 11.4 seconds. GuardiaNN reduces the latency to less than 1 second for all DNNs. Fig. 15(b) shows the relative speedup provided by GuardiaNN normalized for configurations with direct convolution and DNN-friendly SRAM management techniques (third bar). It can be observed that each proposed technique is effective. Applying direct convolution improves the geometric mean speedup by 2.58x compared to the basic framework using im2col. DNN-friendly SRAM management provides an additional geometric mean speedup of 3.19x. Offloading encryption and decryption to cryptographic hardware provides an additional geometric mean speedup of 1.73x. Applying intra-layer pipelining further improves this, increasing the geometric mean speedup by 1.07x. GuardiaNN achieves a geometric mean speedup of 15.3x compared to the baseline. Of the eight DNNs evaluated here, GuardiaNN accelerates Smart Doll the most, making it 31.4x faster than the baseline. Most of this speedup is due to the application of DNN-friendly SRAM management with direct convolution, resulting in fewer accesses to the slow DRAM. AlexNet benefits most from GuardiaNN's use of cryptographic hardware and intra-layer pipelines. Relative to layer operations, AlexNet's encryption and decryption overhead is larger than the other DNNs, which is significantly reduced by using cryptographic hardware and intra-layer pipelines. Table 3 below shows the impact of GuardiaNN on the execution speed of DNNs.
[0092] [Table 3]
[0093] As shown in Table 3 above, the GuardiaNN proposed in this invention can accelerate the execution of a wide range of DNNs without compromising security guarantees.
[0094] <High Energy Efficiency>
[0095] Here, we investigate the energy consumption of DNN execution in all configurations for each DNN in the benchmark. First, we measure the average power of the whole device in the idle state and while running the DNN, and subtract both values to calculate the average power increase due to the DNN execution. Then, we calculate the energy consumption of the DNN execution by multiplying the average power increase by the latency. The normalized results are shown in Fig. 15(b). It can be observed that the energy consumption is reduced by applying each of the proposed techniques. Compared to the baseline framework, the energy consumption of GuardiaNN is reduced by a geometric mean of 92.3%, and the energy efficiency is improved by a factor of 15.2. This is due to the significant latency reduction offered by GuardiaNN. After applying the proposed techniques, the overall power of the device is maintained at the same level during the execution of the DNN. GuardiaNN combined with the reduced latency achieves a much higher energy efficiency than the baseline framework.
[0096] <Sensitivity Studies>
[0097] To study the impact of intra-layer pipelining bulk size on the execution speed of DNNs, we measure the execution latency of GuardiaNN DNNs with four bulk sizes (4, 8, 16, and 32 output channels). Figure 16 shows the measurements normalized to a bulk size of 4. Besides the four bulk sizes, we calculated a set of optimal bulk sizes by layer that lead each DNN to the lowest latency and measured the latency as follows: It can be observed that intra-layer pipelining improves the latency by up to 14.24% at the optimal bulk size (the fifth bar in Figure 16). Compared with the bulk size with the highest latency. For most DNNs, the latency tends to decrease as the bulk size increases. This is because a larger bulk size reduces the number of encryption and decryption requests (and interrupts) to the cryptographic hardware. This effectively reduces the latency of the pipeline operation steps and reduces the overall latency. However, the bulk size is bound by the SRAM capacity. For example, using a bulk size larger than 8 for FgptAuth causes the required memory size to exceed the SRAM size. This is why the execution latency of FgptAuth with bulk sizes of 16 and 32 is empty in FIG.
[0098] The DMA-capable cryptographic hardware used in GuardiaNN supports multiple block ciphers and working modes. GuardiaNN uses AES for enhanced security, and here we compare two working modes, AES-ECB and AES-CBC. We measure the encryption and decryption throughput of AES-ECB and AES-CBC with various key sizes, running (a) on the cryptographic hardware and (b) on a CPU with LibTomCrypt, the base CPU-based cryptographic library of OP-TEE. Figure 17 shows the measured throughput. It is observed that AES-ECB and AES-CBC run much faster on the cryptographic hardware than on the CPU with LibTomCrypt. Also, the throughput of the CPU with LibTom-Crypt decreases as the key size increases, since a longer key size results in more rounds in AES encryption and decryption. Comparing the working modes, AES-CBC has a slightly lower throughput than AES-ECB due to chaining. Conversely, on the cryptographic hardware, the key size and working mode do not have a noticeable impact on the throughput. This is because encryption and decryption performance is limited by the cryptographic hardware and DMA buffer management costs. Nevertheless, the cryptographic hardware used in GuardiaNN provides much higher encryption and decryption efficiency than the CPU in all cases.
[0099] As interest in running DNNs on mobile and embedded devices grows, various techniques have emerged to accelerate the execution of DNNs on such devices. However, there are still many privacy issues as data processed on-device, such as federated learning, is based on differential privacy. Therefore, utilizing a trusted execution environment for DNNs is a reasonable direction. For example, SecureTF is a TensorFlow-based distributed secure machine learning framework that utilizes a trusted execution environment. PPFL utilizes a trusted execution environment for local training and aggregation, multi-part ML to accelerate secure federated learning. Chiron and Myelin activate the trusted execution environment as a service within machine learning.
[0100] DarkneTZ, Infenclave, and Slalom all propose to execute part of the DNN within a trusted execution environment. However, applying the access method straightforwardly while leaving the remaining layers exposed to attacks results in a significant amount of performance overhead. To solve such problems, HybridTEE proposes to request the execution of the DNN in the trusted execution environment of a remote server. However, the acceleration effect of HybridTEE is not very significant because it does not optimize the execution of the DNN in the local trusted execution environment.
[0101] The present invention can not only propose complete protection of the DNN in the local trusted execution environment, but also provide a surprising speed improvement that enables it to be executed in actual mobile and embedded environments.
[0102] <DNN Encryption>
[0103] Another direction for protecting the DNN is to encrypt the DNN data. For example, SecureML utilized secure multi-party computation to build an extensible DNN framework for protecting personal information. SoftME provides a trustworthy environment and executes trustworthy operations composed of encryption, decryption, and calculation steps. Although the confidentiality is guaranteed when executing the DNN in SoftME, using the CPU for data encryption and decryption incurs a large performance overhead. Among them, applying homomorphic encryption is promising because it enables operations on encrypted data. CryptoNets demonstrated that such an idea is feasible and extended the idea to security education. MiniONN proposed a technique to transform a pre-trained DNN so that it cannot be recognized. In addition, homomorphic encryption is utilized to build a secure federated transfer learning protocol. However, homomorphic encryption is often considered not practical for mobile and embedded devices because of its low computational throughput.
[0104] TEEs have attracted attention due to their high security assurance, with ARM TrustZone and Intel SGX being widely used commercial implementations. Software-based security solutions have been successfully exploited to protect various applications. However, there are many threats targeting TEE systems, such as cache architecture, dual instance apps, or overlapping apps. As a result, there have been several recent proposals to enhance security. Also, many works have been proposed to alleviate the difficulties of utilizing TEEs. Minimal kernel builds a small kernel to solve the limited memory problem of TEE. CoSMIX allows for application-level security page error handler. TEEMon is a performance monitoring framework for TEEs.
[0105] The trust environment-based artificial intelligence device according to the present invention can isolate the execution of an artificial neural network in a trust execution environment and can encrypt data stored in a slow-operating DRAM to enhance security. In addition, the number of DRAM accesses can be reduced through direct convolution and SRAM management, and the encryption and decryption of data can be offloaded to cryptographic hardware, and pipelining can be realized to execute the execution of neural network operations so as to overlap with the encryption and decryption of data, thereby accelerating the execution of the artificial neural network.
[0106] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical spirit of the present invention. [Explanation of symbols]
[0107] 100 (Trust Environment-Based) Artificial Intelligence Device 110 Memory of the first type 130 Trust AI Processing Unit 131 Cryptographic front-end process 133 Second type of memory 135 processors
Claims
1. a first type of memory for transmitting encrypted input data and receiving encrypted output data; A trust artificial intelligence processing unit that operates in a trust trust space and performs artificial intelligence calculations on the encrypted input data and the encrypted output data, The trust artificial intelligence processing unit includes: a cryptographic front-end processor for decrypting the encrypted input data to generate decrypted input data and for encrypting unencrypted output data to generate the encrypted output data; a second type of memory providing a buffer for the decrypted input data and the unencrypted output data; and a processor that performs a neural network operation based on the decrypted input data to generate the unencrypted output data.
2. The trust environment-based artificial intelligence device of claim 1, wherein the cryptographic processing front-end processor receives encrypted input activations and encryption filters as the encrypted input data from the first type of memory and stores decryption input activations and decryption filters in the second type of memory.
3. The trust environment-based artificial intelligence device of claim 1, wherein the cryptographic processing front-end processor receives an on-demand request by the processor during the artificial intelligence calculation process and accesses the first type of memory to retrieve the encrypted input data.
4. The trust environment-based artificial intelligence device of claim 1 , wherein the second type of memory has a relatively faster operating speed and a smaller storage capacity than the first type of memory.
5. The trust environment-based artificial intelligence device of claim 1 , wherein the processor performs direct convolution-based neural network operations to reduce the number of accesses to the first type of memory.
6. The trust environment based artificial intelligence device of claim 5, wherein the processor stores decryption input activations in a fixed manner in the second type of memory, and stores decryption filters and non-encrypted output activations in a circular queue manner.
7. The trust environment-based artificial intelligence device of claim 1, wherein the processor transmits and receives data to and from the first type memory and the second type memory through an interrupt-driven offload of the cryptographic processing front-end processor.
8. The processor transmits and receives data between the cryptographic processing front-end processor and the first type memory and the second type memory through a DMA (Direct Memory Access) controller's DMA-driven offload.
9. The trust environment-based artificial intelligence device according to claim 1, characterized in that the processor executes the neural network operations so as to overlap with the execution of encryption and decryption operations of the cryptographic processing front-end processor, thereby realizing intra-layer pipelining.
10. The trust environment-based artificial intelligence device of claim 9, wherein the processor performs the neural network operations uninterruptedly by causing the cryptographic processing front-end processor to perform decryption operations on the encrypted input data while the neural network operations are being executed.
11. The trust environment-based artificial intelligence device according to claim 9, wherein the processor performs the neural network operation seamlessly by subdividing the operation into a data decryption step, an operation step, and a data encryption step for the intra-layer pipelining.
12. a memory of a first type to which encrypted input data is transmitted; A trust artificial intelligence processing unit that operates in a trust trust space and performs an artificial intelligence calculation on the encrypted input data, The trust artificial intelligence processing unit includes: a cryptographic front-end processor for generating decrypted input data through decryption of the encrypted input data; a second type of memory providing a buffer for the decrypted input data and the unencrypted output data; and a processor that performs a neural network operation based on the decrypted input data to generate the unencrypted output data.
13. The trust environment based artificial intelligence device of claim 12, wherein the processor performs a direct convolution based neural network operation to reduce the number of accesses to the first type of memory.
14. The trust environment based artificial intelligence device of claim 12, wherein the processor stores decryption input activations in a fixed manner in the second type of memory, and stores decryption filters and non-encrypted output activations in a circular queue manner.
15. The trust environment-based artificial intelligence device of claim 12, wherein the processor executes the neural network operations to overlap with the execution of encryption and decryption operations of the cryptographic processing front-end processor, thereby realizing intra-layer pipelining.
Citation Information
Patent Citations
Stable volatile data acquisition method and system in ARM processor-based mobile devices
KR102474875B1
