System on chip for supporting low power edge ai and electronic device comprising system on chip

Abstract

An electronic device according to one embodiment comprises: a system on chip (SoC); and a NAND flash connected to the SoC and storing weights and instructions of an artificial neural network model, wherein the SoC may include: a memory that stores input data of the artificial neural network model; an interface that performs communication between the NAND flash and the SoC; and an accelerator configured to perform an artificial neural network operation between input data acquired from the memory and weights acquired through the interface according to the instructions.

Description

Electronic devices including systems-on-chips and systems-on-chips for low-power edge AI support

[0001] The embodiments relate to a system-on-chip and an electronic device including a system-on-chip, and more particularly to a system-on-chip providing edge AI.

[0002] Edge computing is a distributed computing paradigm that brings computing and data storage closer to the devices collecting it. Therefore, compared to relying on centralized sources like the cloud, edge computing avoids the bandwidth and latency issues that impact application performance due to real-time data. Simply put, edge computing runs processes locally, such as on computers, IoT devices, or edge servers, instead of in the cloud. By bringing computation to the network edge, long-distance communication between clients and servers is reduced.

[0003] Edge AI combines two technologies: edge computing and artificial intelligence (AI). Currently, most AI processes are run in the cloud, as running deep learning models requires enormous computing power. However, running AI in the cloud has its drawbacks. For example, it requires an internet connection, and performance can be affected by bandwidth and latency constraints.

[0004] Edge AI, also known as "on-device AI," is a distributed computing paradigm where AI algorithms run directly on devices using data generated by the device. Running AI at the edge of a local network eliminates the need to connect devices to centralized servers like the internet or the cloud. However, current AI computing is power-intensive and computationally expensive.

[0005] The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the present application.

[0006] An electronic device according to one embodiment includes a system on chip (SoC); and a NAND flash connected to the SoC and storing weights and instructions of an artificial neural network model, wherein the SoC may include a memory storing input data of the artificial neural network model, an interface performing communication between the NAND flash and the SoC, and an accelerator configured to perform an artificial neural network operation between the input data obtained from the memory and the weights obtained through the interface according to the instructions.

[0007] The memory may further include buffers for storing the input data; and memory banks for loading and storing a portion of the weight from the interface according to the instruction.

[0008] The above memory bank may include a first memory bank storing a first weight required for the corresponding operation cycle; and a second memory bank storing a second weight required for the next operation cycle of the corresponding operation cycle.

[0009] The accelerator may include a data selection unit that selects first data and second data from among the outputs of the buffers and the output of the first memory bank.

[0010] The data selection unit may include a first multiplexer that selects first data among the outputs of the buffers and the output of the first memory bank; and a first multiplexer that selects second data among the outputs of the buffers and the output of the first memory bank.

[0011] The above instruction includes a data alignment instruction, and the accelerator may include a data alignment unit that aligns the first data and the second data according to the data alignment instruction.

[0012] The first data may include continuous data selected from the first multiplexer, the second data may include continuous data selected from the second multiplexer, and the data alignment unit may include a first shifter that aligns the continuous data selected from the first multiplexer; and a second shifter that aligns the continuous data selected from the second multiplexer.

[0013] The accelerator may include a computation unit that performs the artificial neural network operation between the first data and the second data.

[0014] The above instruction includes a convolution operation instruction, and the operation unit can perform a convolution operation between the output of the first shifter and the output of the second shifter according to the convolution operation instruction.

[0015] The above operation unit may include a MAC that performs a MAC operation between the output of the first shifter and the output of the second shifter in one cycle according to the convolution operation instruction.

[0016] The number of the above buffers can be determined based on an algorithm corresponding to the artificial neural network operation.

[0017] The above memory may include SRAM (Static Random Access Memory).

[0018] The above interface may include a Quad Serial Peripheral Interface (QSPI).

[0019] A system on chip, comprising: a memory storing input data of an artificial neural network model; an interface communicating with an external memory storing weights of the artificial neural network model; and an accelerator configured to perform an artificial neural network operation between the input data obtained from the memory and the weights obtained through the interface according to an instruction, wherein the instruction includes a data alignment instruction and a calculation instruction, and the accelerator may include: a data alignment unit that aligns the input data and the weights according to the data alignment instruction; and a calculation unit that performs an operation between the input data and the weights aligned by the data alignment unit according to the calculation instruction.

[0020] FIG. 1a illustrates a block diagram of an electronic device according to one embodiment.

[0021] Figure 1b illustrates a block diagram of a SoC according to one embodiment.

[0022] Referring to FIG. 2, the structure of an electronic device in which various embodiments can be implemented is described.

[0023] FIG. 3 is a diagram for explaining SoC operation according to one embodiment.

[0024] FIG. 4 is a diagram for explaining a method for performing a MAC operation according to one embodiment.

[0025] FIG. 5 illustrates a block diagram of an electronic device according to various embodiments.

[0026] The specific structural or functional descriptions disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to technical concepts, and the actually implemented form may have various different appearances and is not limited to the embodiments described in this specification.

[0027] While terms like "first" and "second" may be used to describe various components, these terms should be understood solely to distinguish one component from another. For example, a "first" component may be referred to as a "second" component, and similarly, a "second" component may also be referred to as a "first" component.

[0028] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0029] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0031] The embodiments can be implemented in various forms of products, such as personal computers, laptop computers, tablet computers, smartphones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. The embodiments are described in detail below with reference to the attached drawings. Like reference numerals in each drawing represent like elements.

[0032] FIG. 1a illustrates a block diagram of an electronic device according to one embodiment.

[0033] An electronic device according to various embodiments of the present document may include at least one of, for example, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device. According to various embodiments, the wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or a tattoo), or a bio-implant type (e.g., an implantable circuit).

[0034] In some embodiments, the electronic device may be a home appliance. The home appliance may include, for example, at least one of a television, a digital video disk player (DVD player), an audio device, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console (e.g., Xbox™, PlayStation™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

[0035] In another embodiment, the electronic device may be a medical device (e.g., various portable medical measuring devices (e.g., blood glucose meter, heart rate meter, blood pressure meter, or body temperature meter, etc.), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), camera, or ultrasound machine, etc.), navigation device, satellite navigation system (Global Navigation Satellite System (GNSS)), event data recorder (EDR), flight data recorder (FDR), automobile infotainment device, electronic equipment for ships (e.g., marine navigation device, gyrocompass, etc.), avionics, security device, head unit for vehicles, industrial or home robot, automatic teller's machine (ATM) of financial institution, point of sales (POS) of store, or internet of things device (e.g., light bulb, various sensors, electric or gas meter, sprinkler device, fire alarm, It may include at least one of the following: a thermostat, a streetlight, a toaster, exercise equipment, a hot water tank, a heater, a boiler, etc.

[0036] According to some embodiments, the electronic device may include at least one of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., water, electricity, gas, or radio wave measuring devices). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. The electronic device according to some embodiments may be a flexible electronic device. In addition, the electronic device according to the embodiments of the present document is not limited to the devices described above, and may include new electronic devices developed according to technological advancements.

[0037] Hereinafter, electronic devices according to various embodiments are described with reference to the attached drawings. In this document, the term "user" may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0038] An electronic device (100) according to one embodiment may include an edge AI including a system-on-chip (110) and a NAND flash (120). For example, referring to FIG. 1A, the electronic device may include a system-on-chip (110) and a NAND flash (120). The configuration of the electronic device illustrated in FIG. 1A is exemplary, and various modifications are possible to implement various embodiments disclosed in the present document. For example, the electronic device may include a configuration such as the electronic device of FIG. 2A or the electronic device (501) illustrated in FIG. 5, or may be appropriately modified by utilizing these configurations.

[0039] An electronic device (100) according to one embodiment may be an edge AI that executes AI algorithms locally on a hardware device using edge computing. Edge AI may perform artificial neural network operations.

[0040] For example, the electronic device (100) can perform TTS (Text to Speech) based on an artificial neural network model. The artificial neural network model for performing TTS may include a Text-to-Mel model that generates text as a spectrogram, and a vocoder model that converts the generated spectrogram into a sound wave (Waveform), which is actual speech. For example, the Text-to-Mel model may include a phoneme encoder, a duration predictor, and a Mel decoder, and the vocoder model may be a HiFi-GAN. However, the AI ​​algorithm that can be executed in the electronic device (100) is not limited to the above-described matters.

[0041] Below, artificial neural network operations refer to the operations required to execute AI algorithms, and can be referred to as neural network operations. A neural network can refer to the entire model, where artificial neurons (nodes) form a network through synaptic connections, which change the strength of the synaptic connections through learning, thereby gaining problem-solving capabilities.

[0042] Neurons in a neural network can contain a combination of weights or biases. A neural network can include one or more layers, each consisting of one or more neurons or nodes. A neural network can infer a desired outcome from any input by changing the weights of its neurons through learning.

[0043] Neural networks may include deep neural networks. Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It can include ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), BNN (Binarized Neural Network), and AN (Attention Network).

[0044] An electronic device (100) according to one embodiment may include an edge AI including a system-on-chip (110) that performs artificial neural network operations and NAND flash (120). However, not all of the illustrated components are essential components. The electronic device (100) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0045] Edge AI may include chips that perform artificial neural network computations and off-chip memory that stores the weights, biases, and intermediate results of these computations. Because the amount of weights, biases, and intermediate results required for these computations is extremely large, conventional edge AI uses DRAM as external memory. High bandwidth is required for fast computations, and DDR4 can transmit data at 16 bits or more at a clock rate of approximately 1 GHz, while DDR5 can transmit data at a clock rate of approximately 3 GHz.

[0046] An electronic device (100) according to one embodiment may include NAND flash (120) as external memory. Since NAND flash (120) has a lower bandwidth than DRAM (e.g., 4-bit data at 100 MHz when interfaced with QSPI), efficient data supply for fast operations is required. NAND flash (120) can store weights and instructions required for artificial neural network operations.

[0047] Accordingly, as will be described in detail below, an electronic device (100) according to one embodiment uses a system-on-chip (110) designed to enable efficient data supply, and new instructions for this purpose are defined as shown in Table 1.

[0048]

[0049]

[0050] 1b shows a block diagram of a SoC according to one embodiment.

[0051] Referring to FIG. 1B, a SoC (110) according to one embodiment may include a memory (111), an interface (112), and at least one accelerator (113). However, not all of the illustrated components are essential components. The SoC (110) may be implemented with more components than the illustrated components, or may be implemented with fewer components.

[0052] The memory (111) can store input data of an artificial neural network model. According to one embodiment, the input data may include activation values, such as activation values ​​of an intermediate layer and activation values ​​of an output layer. The memory (111) may be a buffer memory or a local memory. The memory (111) may be a static random access memory (SRAM).

[0053] An interface (112) according to one embodiment may relay a resource request or transfer from one component to another. The interface (112) may relay a resource request between the SoC (110) and the NAND flash (120). The interface (112) may transmit a data processing request between the SoC (110) and the NAND flash (120). The interface (112) may be connected to a memory (111) and at least one accelerator (113).

[0054] At least one accelerator (113) according to one embodiment may be a hardware configuration that performs some functions of the electronic device (100). Hereinafter, the accelerator may be referred to as a hardware accelerator, an inference accelerator, an IX, etc. At least one hardware accelerator (113) may perform some functions of the electronic device (100) more quickly than a software method implemented in a specific processor (e.g., CPU). As an example, at least one accelerator (113) may include at least one of a CPU, a GPU, a DSP, an ISA, a graphics card (or a video card).

[0055] Referring to FIG. 2, the structure of an electronic device capable of implementing various embodiments is described. The contents described with reference to FIGS. 1A and 1B can be equally applied to FIG. 2. The configuration of the electronic device illustrated in FIG. 2 is exemplary, and various modifications are possible to implement the various embodiments disclosed in this document.

[0056] Referring to FIG. 2, an electronic device (200) according to one embodiment may include an SoC (210), a NAND flash (220), sensor(s) (230), a microphone (240), and audio (250). The SoC (210) according to one embodiment may include an IX, an SRAM, an LDO, a POR / BOD, an OSC, an MCU, a DMA, a QSPI, an SPI, an I2C / UART, a PDM, a DDAC, an I2S, a GPIO, and an ADC.

[0057] In terms of the functions performed by each configuration, IX, SRAM, and QSPI of FIG. 2 can be understood to correspond to the accelerator (113), memory (111), and interface (112) of FIG. 1b, respectively. Descriptions of corresponding or overlapping contents are omitted below. The operations of LDO, POR / BOD, OSC, MCU, DMA, SPI, I2C / UART, PDM, DDAC, I2S, GPIO, and ADC can be understood by those of ordinary skill in the art related to the present embodiment, and thus, a detailed description thereof will be omitted.

[0058] FIG. 3 is a diagram illustrating the operation of an SoC according to one embodiment. More specifically, FIG. 3 is a diagram illustrating a specific operating method of the SoC when a data alignment instruction and a convolution operation instruction are given. The contents described with reference to FIGS. 1A to 2 can be equally applied to FIG. 3.

[0059] Referring to FIG. 3, a SoC (300) according to one embodiment may include a memory (310), an interface (320), and an accelerator (330). The memory (310) according to one embodiment may include buffers (311-1 to 311-6) and memory banks (315-1, 315-2). The accelerator (330) according to one embodiment may include a data selection unit (331), a data alignment unit (332), and an operation unit (333).

[0060] The term "-part" as used in this document can mean a unit that includes one or a combination of two or more of hardware, software, or firmware, for example. "-part" can be used interchangeably with terms such as unit, logic, logical block, component, or circuit, for example. "-part" can be the smallest unit of an integrally formed part or a part thereof. "-part" can also be the smallest unit that performs one or more functions or a part thereof. "-part" can be implemented mechanically or electronically. For example, "-part" can include at least one of an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs), or programmable-logic device that performs certain operations, whether known or to be developed in the future.

[0061] The interface (320) may store the first weight required for the corresponding operation cycle in the first memory bank (315-1) according to the instruction, and may store the second weight required for the next operation cycle of the corresponding operation cycle in the second memory bank (315-2). That is, while performing an operation with the weight and bias stored in the first memory bank (315-1), the second memory bank (315-2) may store the weight and bias for the next operation. The weights stored in the first memory bank (315-1) and the second memory bank (315-2) may have a size of 1152-bit * 2048-bit (depth x width).

[0062] Buffers (311-1 to 311-6) can store input data for artificial neural network operations. For example, each of buffers (311-1 to 311-6) can store input data TTS0 to TTS5 for TTS operations. TTS0 to TTS5 may have a size of 384-bit*2048-bit (depth x width). The width of all internal signals may be 128 x 16-bit = 2048-bit. In the artificial neural network operation process according to the above-described TTS algorithm, a total of six buffers (311-1 to 311-6) may be required for input, intermediate results, and final results. However, the number of buffers (311-1 to 311-6) is no longer limited.

[0063] According to one embodiment, a data selection unit (331) can select first data (e.g., x0) and second data (e.g., x1) from among the outputs of the buffers (311-1 to 311-6) and the outputs of the first memory banks (315-1, 315-2). The data selection unit (331) can read two operands (e.g., x0, x1) simultaneously from a single port memory. X0 and x1 are 2048-bit operands, which can be 128 data of 16-bit size. Since the data must be in different areas in order to read two operands simultaneously, the data selection unit (331) can include two multiplexers (331-1, 331-2). The first multiplexer (331-1) can select first data among the outputs of the buffers (311-1 to 311-6) and the outputs of the first memory banks (315-1, 315-2). The second multiplexer (331-2) can select second data among the outputs of the buffers (311-1 to 311-6) and the outputs of the first memory banks (315-1, 315-2).

[0064] According to one embodiment, the data alignment unit (332) may align the first data and the second data, respectively, according to the data alignment instruction. More specifically, the first data may include consecutive data (e.g., x0, x0_d) selected from the first multiplexer (331-1). Similarly, the second data may include consecutive data (e.g., x1, x1_d) selected from the second multiplexer (331-2).

[0065] The data sorting unit (332) may include a first shifter (332-1) and a first shifter (332-2). The first shifter (332-1) may sort the continuous data selected from the first multiplexer (331-1), and the second shifter (332-2) may sort the continuous data selected from the second multiplexer (331-2). For example, the first shifter (332-1) may sort 128 data among two data of x0 and x0_d read in succession, and the first shifter (332-1) may sort 128 data among two data of x1 and x1_d read in succession. Through this, it is also possible to process cases where the 128 data are not exactly sorted.

[0066] The operation unit (333) according to one embodiment can perform an artificial neural network operation between the first data and the second data. The operation unit (333) can perform a convolution operation between the output of the first shifter (332-1) and the output of the second shifter (332-2) according to a convolution operation instruction. The operation unit (333) can include a MAC that performs a MAC operation between the output of the first shifter (332-1) and the output of the second shifter (332-2) in one cycle. A method for performing the MAC operation according to one embodiment is described in detail below with reference to FIG. 4. Although not shown in FIG. 3, the operation unit (333) according to one embodiment can include operation devices that can perform a nonlinear function operation (e.g., ReLu operation) in addition to the convolution operation.

[0067] FIG. 4 is a diagram for explaining a method for performing a MAC operation according to one embodiment.

[0068] Referring to FIG. 4, a MAC operation according to one embodiment may be divided into multiple steps. For example, a MAC operation between 128 input data and weights may be divided into 9 steps. In step 1, a multiplication operation may be performed between 128 FP16 precision input data and weights. Based on the operation result of step 1, the FP16 precision may be changed to FP32 precision. To minimize bits discarded during the multiplication and addition operations, the intermediate data may be expanded to 32 bits. Thereafter, addition may be repeated in an adder-tree form until step 9, after which it may be changed back to FP16. An electronic device according to one embodiment may perform the above-described operation in one cycle.

[0069] FIG. 5 illustrates a block diagram of an electronic device according to various embodiments.

[0070] Referring to FIG. 5, the electronic device (501) may include, for example, all or part of the electronic device (100) illustrated in FIG. 1A. The electronic device (501) may include one or more processors (e.g., AP) (510), a communication module (520), a subscriber identification module (524), a memory (530), a sensor module (540), an input device (550), a display (560), an interface (570), an audio module (580), a camera module (591), a power management module (595), a battery (596), an indicator (597), and a motor (598).

[0071] The processor (510) may control a plurality of hardware or software components connected to the processor (510) by, for example, driving an operating system or an application program, and may perform various data processing and calculations. The processor (510) may be implemented as, for example, a system on chip (SoC) (e.g., SoC (110)). According to one embodiment, the processor (510) may further include a graphic processing unit (GPU) and / or an image signal processor. The processor (510) may also include at least some of the components illustrated in FIG. 5 (e.g., a cellular module (521)). The processor (510) may load and process commands or data received from at least one of the other components (e.g., a non-volatile memory) into a volatile memory, and store various data in the non-volatile memory.

[0072] The communication module (520) may include, for example, a cellular module (521), a Wi-Fi module (522), a Bluetooth module (523), a GNSS module (524) (e.g., a GPS module, a Glonass module, a Beidou module, or a Galileo module), an NFC module (525), an MST module (526), ​​and an RF (radio frequency) module (527).

[0073] The cellular module (521) may provide, for example, voice calls, video calls, text services, or Internet services through a communication network. According to one embodiment, the cellular module (521) may use a subscriber identification module (e.g., SIM card) (529) to distinguish and authenticate the electronic device (501) within the communication network. According to one embodiment, the cellular module (521) may perform at least some of the functions that the processor (510) may provide. According to one embodiment, the cellular module (521) may include a communication processor (CP).

[0074] Each of the Wi-Fi module (522), the Bluetooth module (523), the GNSS module (524), the NFC module (525), or the MST module (526) may include, for example, a processor for processing data transmitted and received through the corresponding module. According to some embodiments, at least some (e.g., two or more) of the cellular module (521), the Wi-Fi module (522), the Bluetooth module (523), the GNSS module (524), the NFC module (525), or the MST module (526) may be included in a single integrated chip (IC) or IC package.

[0075] The RF module (527) can transmit and receive, for example, a communication signal (e.g., an RF signal). The RF module (527) can include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module (521), the Wi-Fi module (522), the Bluetooth module (523), the GNSS module (524), the NFC module (525), and the MST module (526) can transmit and receive an RF signal through a separate RF module.

[0076] The subscriber identification module (529) may include, for example, a card including a subscriber identification module and / or an embedded SIM, and may include unique identification information (e.g., an integrated circuit card identifier (ICCID)) or subscriber information (e.g., an international mobile subscriber identity (IMSI)).

[0077] The memory (530) may include, for example, built-in memory (532) or external memory (534). The built-in memory (532) may include, for example, at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).

[0078] The external memory (534) may further include a flash drive, for example, a compact flash (CF), a secure digital (SD), a Micro-SD, a Mini-SD, an extreme digital (xD), a MultiMediaCard (MMC), or a memory stick. The external memory (534) may be functionally and / or physically connected to the electronic device (501) through various interfaces.

[0079] The security module (536) is a module that includes a storage space with a relatively higher security level than the memory (530), and may be a circuit that ensures secure data storage and a protected execution environment. The security module (536) may be implemented as a separate circuit and may include a separate processor. The security module (536) may include, for example, a removable smart chip, an SD (secure digital) card, or an embedded secure element (eSE) embedded in a fixed chip of the electronic device (501). In addition, the security module (536) may be driven by an operating system (OS) different from the operating system (OS) of the electronic device (501). For example, the security module (536) may operate based on the JCOP (Java Card Open Platform) operating system.

[0080] The sensor module (540) can measure a physical quantity or detect an operating state of an electronic device (501), for example, and convert the measured or detected information into an electrical signal. The sensor module (540) can include at least one of, for example, a gesture sensor (540A), a gyro sensor (540B), a pressure sensor (540C), a magnetic sensor (540D), an acceleration sensor (540E), a grip sensor (540F), a proximity sensor (540G), a color sensor (540H) (e.g., an RGB sensor), a biometric sensor (540I), a temperature / humidity sensor (540J), an illuminance sensor (540K), or a UV (ultra violet) sensor (540M). Additionally or alternatively, the sensor module (540) may include, for example, an olfactory sensor (E-nose sensor), an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module (540) may further include a control circuit for controlling at least one sensor included therein. In some embodiments, the electronic device (501) further includes a processor configured to control the sensor module (540), either as part of or separately from the processor (510), such that the sensor module (540) may be controlled while the processor (510) is in a sleep state.

[0081] The input device (550) may include, for example, a touch panel (552), a (digital) pen sensor (554), a key (556), or an ultrasonic input device (558). The touch panel (552) may use, for example, at least one of a capacitive method, a pressure-sensitive method, an infrared method, or an ultrasonic method. In addition, the touch panel (552) may further include a control circuit. The touch panel (552) may further include a tactile layer to provide a tactile response to the user.

[0082] The (digital) pen sensor (554) may be, for example, part of a touch panel or may include a separate recognition sheet. The key (556) may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device (558) may detect ultrasonic waves generated from an input tool through a microphone (e.g., a microphone (588)) and confirm data corresponding to the detected ultrasonic waves.

[0083] The display (560) (e.g., the display (460)) may include a panel (562), a holographic device (564), or a projector (566). The panel (562) may be, for example, implemented to be flexible, transparent, or wearable. The panel (562) may also be configured as a single module with the touch panel (552). The holographic device (564) may use light interference to display a three-dimensional image in the air. The projector (566) may project light onto a screen to display an image. The screen may be located, for example, inside or outside the electronic device (501). According to one embodiment, the display (560) may further include a control circuit for controlling the panel (562), the holographic device (564), or the projector (566).

[0084] The interface (570) may include, for example, an HDMI (572), a USB (574), an optical interface (576), or a D-sub (D-subminiature) (578). The interface (570) may include, for example, an MHL (mobile high-definition link) interface, an SD card / MMC interface, or an IrDA (infrared data association) standard interface.

[0085] The audio module (580) can, for example, bidirectionally convert sound and electrical signals. The audio module (580) can process sound information input or output through, for example, a speaker (582), a receiver (584), earphones (586), or a microphone (588).

[0086] The camera module (591) is, for example, a device capable of capturing still images and moving images, and according to one embodiment, may include one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (e.g., an LED or a xenon lamp).

[0087] The power management module (595) can manage the power of the electronic device (501), for example. According to one embodiment, the power management module (595) can include a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery or fuel gauge. The PMIC can have a wired and / or wireless charging method. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and can further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, or a rectifier. The battery gauge can measure, for example, the remaining capacity of the battery (596), voltage, current, or temperature during charging. The battery (596) can include, for example, a rechargeable battery and / or a solar battery.

[0088] The indicator (597) may indicate a specific status of the electronic device (501) or a part thereof (e.g., the processor (510)), such as a booting status, a message status, or a charging status. The motor (598) may convert an electrical signal into a mechanical vibration and may generate vibration, a haptic effect, or the like. Although not shown, the electronic device (501) may include a processing unit (e.g., a GPU) for supporting mobile TV. The processing unit for supporting mobile TV may process media data according to standards such as, for example, DMB (Digital Multimedia Broadcasting), DVB (Digital Video Broadcasting), or MediaFLOTM.

[0089] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0090] Software may include computer programs, codes, instructions, or any combination thereof, which may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage media or devices, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0091] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0092] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0093] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, SoC(system on chip); and Connected to the above SoC and including a NAND flash that stores weights and instructions of an artificial neural network model, The above SoC is, A memory for storing input data of the above artificial neural network model; An interface for performing communication between the NAND flash and the SoC; and An accelerator configured to perform an artificial neural network operation between the input data obtained from the memory and the weight obtained through the interface according to the above instructions. An electronic device comprising:

2. In paragraph 1, The above memory Buffers for storing the above input data; and Memory banks for loading and storing a portion of the weight from the interface according to the above instructions. An electronic device further comprising:

3. In paragraph 2, The above memory bank A first memory bank storing the first weight required for the corresponding operation cycle; and A second memory bank that stores the second weight required for the next operation cycle of the above operation cycle. An electronic device comprising:

4. In paragraph 3, The above accelerator A data selection unit that selects first data and second data among the outputs of the above buffers and the outputs of the first memory bank. An electronic device comprising:

5. In paragraph 4, The above data selection section A first multiplexer for selecting first data among the outputs of the above buffers and the output of the first memory bank; and A second multiplexer that selects second data among the outputs of the above buffers and the output of the first memory bank. An electronic device comprising:

6. In paragraph 5, The above instructions Contains data sorting instructions, The above accelerator A data alignment unit that aligns the first data and the second data according to the above data alignment instructions. An electronic device comprising:

7. In paragraph 6, The above first data Contains continuous data selected from the above first multiplexer, The above second data is Contains continuous data selected from the second multiplexer, The above data sorting section A first shifter for aligning the continuous data selected from the first multiplexer; and A second shifter that aligns the continuous data selected from the second multiplexer. An electronic device comprising:

8. In paragraph 7, The above accelerator A computation unit that performs the artificial neural network operation between the first data and the second data An electronic device comprising:

9. In paragraph 8, The above instructions Contains convolution operation instructions, The operation department An electronic device that performs a convolution operation between the output of the first shifter and the output of the second shifter according to the convolution operation instruction.

10. In paragraph 9, The above operation unit According to the above convolution operation instruction, MAC performs MAC operation between the output of the first shifter and the output of the second shifter in one cycle. An electronic device comprising:

11. In paragraph 2, The number of the above buffers is An electronic device determined based on an algorithm corresponding to the above artificial neural network operation.

12. In paragraph 1, The above memory An electronic device containing SRAM (Static Random Access Memory). In the first paragraph, The above interface is An electronic device including a Quad Serial Peripheral Interface (QSPI).

13. As a system on chip, Memory that stores input data for an artificial neural network model; An interface for communicating with an external memory that stores the weights of the artificial neural network model; and An accelerator configured to perform an artificial neural network operation between the input data obtained from the memory and the weight obtained through the interface according to the instructions. Including, The above instructions Contains data sorting instructions and operation instructions, The above accelerator A data alignment unit that aligns the input data and the weight according to the above data alignment instructions; and An operation unit that performs an operation between input data and weight sorted by the data sort unit according to the above operation instructions. A system on a chip, comprising:

Images