Electronic device performing calculation using artificial intelligence model, and method for operating electronic device

Abstract

This electronic device for performing a multiply and accumulate (MAC) operation comprises at least one MAC unit, a static random access memory (SRAM), one 8bx8b operator, two 8bx4b operators, one 4bx4b operator, a bit shifter, an adder, at least one accumulator, and a processor. The processor may receive first bit string data and second bit string data which are composed of 12 bits, divide the first bit string data into 4 bits and 8 bits, divide the second bit string data into 4 bits and 8 bits, output two 8-bit results corresponding to activation 8-bit, weight 8-bit (A8W8) by using the bit shifter and a first accumulator on the basis of determining to output an A8W8 result, and output one 12-bit result corresponding to activation 12-bit, weight 12-bit (A12W12) on the basis of determining to output an A12W12 result.

Description

Electronic devices and operating methods for electronic devices that perform calculations using artificial intelligence models

[0001] Various embodiments of this document relate to an electronic device that performs calculations using an artificial intelligence model and a method of operating the electronic device.

[0002] An artificial neural network (ANN) refers to a computational architecture that models the biological brain. Deep learning and machine learning can be implemented based on ANNs. As an example of an ANN, a deep neural network (DNN) or deep learning can have a multilayer structure containing multiple layers.

[0003] In the fields of visual and speech analysis, artificial intelligence (AI) models are being utilized in various ways. To ensure effective operation of AI models on mobile devices, active research and development is underway on hardware technologies related to AI models. For example, research is being conducted on optimizing the MAC (multiply-accumulate) operation performed in deep learning AI models, as well as on improving hardware architectures that take AI models into account.

[0004] Additionally, the data processing system may include at least one processor, commonly known as a central processing unit (CPU). Such a data processing system may also include at least one other processor, such as a neural processing unit (NPU), used for various types of specialized processing.

[0005] Data corresponding to A8W8 can be 8 bits in size. A8W8 can mean 8 bits of activation and 8 bits of weight. Activation can refer to the size of the input data. Weight can refer to a variable that controls the influence of the input data on the result.

[0006] Data corresponding to A12W12 can be 12 bits in size. Since typical computer systems use 32 or 64 bits, when receiving 12-bit data, bits from both 32 and 64 bits may remain. For example, when a computer system receives 12-bit data in a 32-bit format, 8 bits may remain. Also, when a computer system receives 12-bit data in a 64-bit format, 4 bits may remain.

[0007] For example, when training data in the form of A12W12, A12W12 may be more efficient than other systems (e.g., A8W8), but this may result in wasted bits.

[0008] An electronic device may include a memory storing instructions and a processor. The processor may include at least one 8bx8b operator, at least one 8bx4b operator, at least one 4bx4b operator, a bit shifter, an adder, and at least one accumulator. The instructions, when executed by the processor, can control the electronic device to receive first bit-string data and second bit-string data composed of 12 bits, divide the first bit-string data into 4 bits and 8 bits, divide the second bit-string data into 4 bits and 8 bits, and output two 8-bit results corresponding to A8W8 using a bit shifter and a first adder based on a determination to output a result of A8W8 (activation 8-bit, weight 8-bit), and output one 12-bit result corresponding to A12W12 based on a determination to output a result of A12W12 (activation 12-bit, weight 12-bit).

[0009] An electronic device includes an arithmetic circuit, a memory, and a processor, wherein the arithmetic circuit includes a plurality of multiplexers, a bit shifter, an adder, and an accumulator, and the processor can select a first path capable of outputting a result of A8W8 (activation 8-bit, weight 8-bit) or a second path capable of outputting a result of A12W12 (activation 12-bit, weight 12-bit) based on a user input within the arithmetic circuit.

[0010] An operating method of an electronic device that performs a calculation using an artificial intelligence model may include an operation of receiving first bit-string data and second bit-string data composed of 12 bits, an operation of dividing the first bit-string data into four bits and eight bits, an operation of dividing the second bit-string data into four bits and eight bits, an operation of outputting two 8-bit results corresponding to A8W8 using a bit shifter and a first adder based on a determination that a result of A8W8 (activation 8-bit, weight 8-bit) is to be output, and an operation of outputting one 12-bit result corresponding to A12W12 based on a determination that a result of A12W12 (activation 12-bit, weight 12-bit) is to be output.

[0011] An electronic device that performs calculations using an artificial intelligence model according to this document can perform calculations by dividing 12-bit data into 4-bit and 8-bit data using one MAC (multiply and accumulate) and optionally output results corresponding to A8W8 and A12W12.

[0012] An electronic device that performs calculations using an artificial intelligence model according to this document can process 8-bit data, and, if the user selects, 12-bit data.

[0013] An electronic device that performs calculations using an artificial intelligence model according to this document can perform calculations efficiently without wasting bits on a computer system that operates at 32 or 64 bits while processing 12-bit data.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0016] FIG. 3a illustrates the structure of a MAC (multiply and accumulate) that performs multiplication and result accumulation according to various embodiments.

[0017] Figure 3b illustrates a multiplication method between 4-bit data according to various embodiments.

[0018] Figure 4a illustrates a process of dividing 12-bit data into four sections according to various embodiments.

[0019] FIG. 4b illustrates a process of obtaining an A8W8 result through multiplication and sum operations of data separated into four sections according to various embodiments.

[0020] Figure 4c illustrates a process of obtaining an A12W12 result through multiplication and sum operations of data separated into four sections according to various embodiments.

[0021] FIG. 5a illustrates a process of obtaining an A8W8 result through multiplication and addition operations using MAC according to various embodiments.

[0022] FIG. 5b illustrates a first embodiment of sharing input values ​​among multiple MACs when there are multiple MACs as illustrated in FIG. 5a.

[0023] FIG. 6a illustrates a process of obtaining an A12W12 result through multiplication and addition operations using MAC according to various embodiments.

[0024] FIG. 6b illustrates a second embodiment of sharing input values ​​between multiple MACs according to various embodiments.

[0025] FIG. 7a and FIG. 7b illustrate a process of obtaining A8W8 and A12W12 results by packing data into 8-bit units and 4-bit units according to various embodiments.

[0026] FIG. 7c illustrates a process of performing a 3x3 convolution operation according to one embodiment.

[0027] FIG. 8 is a flowchart illustrating an operation method of an electronic device that performs a calculation using an artificial intelligence model according to one embodiment.

[0028] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0029] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0030] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0031] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0035] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0036] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0037] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0041] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0043] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0046] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0047] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0048] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0049] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0051] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0052] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0053] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0054] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0055] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0056] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0057] According to FIG. 2, the electronic device (200) may include a processor (210) and a memory (220), and some of the illustrated configurations may be omitted or replaced. The electronic device (200) may further include at least some of the configurations and / or functions of the electronic device (101) of FIG. 1. At least some of the respective configurations of the illustrated (or not illustrated) electronic device (200) may be operatively, functionally, and / or electrically connected to each other.

[0058] According to one embodiment, the processor (210) may be configured as one or more processors capable of performing calculations or data processing related to control and / or communication of each component of the electronic device (200). The processor (210) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1.

[0059] According to one embodiment, there is no limitation to the operation and data processing functions that the processor (210) can implement on the electronic device (200), but below, a feature of controlling 12-bit data by dividing it into 8-bit and 4-bit data for deep learning operations will be described in detail. The operations of the processor (210) can be performed by loading instructions stored in the memory (220).

[0060] According to one embodiment, the electronic device (200) includes one or more memories (220), and the memories (220) may include main memory and storage. The main memory may be composed of volatile memory such as dynamic random access memory (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM). Alternatively, the memory (220) may be a non-volatile memory and include a large-capacity storage device. The storage may include at least one of a one-time programmable ROM (OTPROM), a PROM, an EPROM, an EEPROM, a mask ROM, a flash ROM, a flash memory, a hard drive, or a solid-state drive (SSD). The memory (220) may store various file data, and the stored file data may be updated according to the operation of the processor (210).

[0061] Deep learning can be performed through the sum of the product of weights and inputs. Applying an activation function to the calculated result obtained through the sum of the product of weights and inputs yields the final result (activation).

[0062] A camera image sensor can perform noise removal and distortion correction on raw data in a 12-bit data format using an image signal processor (ISP). The ISP can receive 12-bit data from the camera image sensor and improve the quality of the image through deep learning pixel processing. If the processor (e.g., NPU) (210) performs Neural ISP processing (deep learning pixel processing) with A8W8 precision, loss may occur in the 12-bit Bayer data. If the NPU processes 12-bit Bayer data with A16W8 precision, there is no data loss, but since the weights use 8 bits, the bitwidth is insufficient, which may result in image quality degradation. If the NPU supports A12W12, compatibility may be good because 12-bit weights can be utilized without loss of Bayer data. Therefore, an NPU that can process A12W12 data may be required. According to one embodiment, the processor (210) may include or be implemented as a neural processing unit (NPU).

[0063] According to one embodiment, the processor (210) may output two 8-bit results corresponding to A8W8 using a bit shifter and a first adder based on a determination to output a result of A8W8 (activation 8-bit, weight 8-bit), and output one 12-bit result corresponding to A12W12 based on a determination to output a result of A12W12 (activation 12-bit, weight 12-bit).

[0064] According to one embodiment, the processor (210) receives first bit string data and second bit string data composed of 12 bits, divides upper 4 bits of the first bit string data into a first section, divides lower 8 bits of the first bit string data into a second section, divides upper 4 bits of the second bit string data into a third section, divides lower 8 bits of the second bit string data into a fourth section, performs multiplication of data of the first section to the fourth section based on an input signal, and adds the multiplication results to output a result of A8W8 (activation 8-bit, weight 8-bit) or output a result of A12W12 (activation 12-bit, weight 12-bit).

[0065] According to one embodiment, the processor (210) can output a result of A8W8 (activation 8-bit, weight 8-bit) or a result of A12W12 (activation 12-bit, weight 12-bit) using a multiplexer and a bit shifter.

[0066] According to one embodiment, the processor (210) may perform a multiplication on the second section and the fourth section to determine a first result value, calculate a multiplication on the fourth section and the third section, shift 4 bits in an upward direction using the bit shifter to determine a second result value, calculate a multiplication on the fourth section and the first section to determine a third result value, output the first result value, perform a sum operation on the second result value and the third result value, and output the result.

[0067] According to one embodiment, the processor (210) may perform a multiplication on a bit corresponding to a second section and a bit corresponding to the fourth section to determine a first result value, perform a multiplication on a bit corresponding to the second section and a bit corresponding to the third section, and shift the result value obtained by the multiplication by 8 bits in the upper direction using the bit shifter to determine a second result value, perform a multiplication on a bit corresponding to the first section and a bit corresponding to the third section, and shift the result value obtained by the multiplication by 8 bits in the upper direction using the bit shifter to determine a third result value, perform a multiplication on a bit corresponding to the first section and a bit corresponding to the fourth section, and shift the result value obtained by the multiplication by 8 bits in the upper direction using the bit shifter to determine a fourth result value, and perform a sum operation on the first to fourth result values ​​to output a result.

[0068] According to one embodiment, the processor (210) may include or be implemented as a neural processing unit (NPU).

[0069] According to one embodiment, the processor (210) may perform a multiplication operation on the bits corresponding to the second section and the bits corresponding to the fourth section and accumulate the result obtained by the multiplication.

[0070] According to one embodiment, the processor (210) may perform a sum operation on a result of performing a product operation between a bit corresponding to the second section or a bit corresponding to the fourth section and a bit corresponding to the third section, and a result of performing a product operation between a bit corresponding to the fourth section and a bit corresponding to the first section, and may perform a product operation on a bit corresponding to the third section and a bit corresponding to the first section and accumulate the result.

[0071] According to one embodiment, the processor (210) may perform a product operation of sections divided into 8 bits on first bit string data and second bit string data composed of 12 bits to determine a first value, perform a product operation of sections divided into 8 bits and 4 bits to determine a second value, perform a product operation of sections divided into 4 bits to determine a third value, and accumulate and store the result by adding the first value, the second value, and the third value.

[0072] According to one embodiment, the processor (210) may perform multiplication of data of the first to fourth sections using a MAC unit and add the multiplication results. The processor (210) may determine whether to output a result of A8W8 (activation 8-bit, weight 8-bit) or a result of A12W12 (activation 12-bit, weight 12-bit) based on an external input.

[0073] Figure 3a illustrates the structure of a MAC (multiply and accumulate) that performs multiplication and result accumulation.

[0074] According to FIG. 3a, the MAC (multiply and accumulate) can perform a multiplication operation on input data and a sum operation. The MAC can include three multiplexers (mux) (310) and one bit shifter (312). The multiplexer (mux) (310) can be used to select one of a plurality of inputs. The bit shifter (312) can be used to shift (or change) the digit of at least one bit during a bit-to-bit operation. For example, in a bit-string data composed of 12 bits, the first bit-string data composed of 4 bits located at 0 to 3 can be changed to be located at 4 to 7 through a 4-bit shift. The bit-to-bit operation process will be described in detail in FIG. 3b.

[0075] MAC (multiply and accumulate) can store the result of performing an 8-bit multiplication operation on the first accumulator (320).

[0076] MAC (multiply and accumulate) can perform an 8-bit and 4-bit multiplication operation using two multiplexers (mux) (310, 314) and a bit shifter (312). MAC (multiply and accumulate) can perform a 4-bit and 4-bit multiplication operation. MAC can perform a sum operation on the result of the 8-bit and 8-bit multiplication operation, the result of the 8-bit and 4-bit multiplication operation, and the result of the 4-bit and 4-bit multiplication operation using an adder (316). MAC can store the result of the sum operation and the result of the 8-bit and 4-bit multiplication operation in the second accumulator (330) through the third multiplexer (318).

[0077] Figure 3b shows a multiplication method between 4-bit data.

[0078] MAC can perform a 4-bit multiplication operation. Figure 3b is only an example; MAC can also perform a multiplication operation on data with different bits.

[0079] For example, a MAC can perform a multiplication operation between 4-bit data having '1101' and 4-bit data having '1011'. A MAC can perform a multiplication operation on 4-bit data having '1101' by digit by digit. A MAC can multiply 1 by the bit corresponding to the first digit from the right of 1011, which is 1. A MAC can multiply 1 by the bit corresponding to the second digit from the right of 1011, which is 1, and shift the result of the multiplication one bit higher. A MAC can multiply 1101 by the 0 in the third digit from the right of 1011, which is 0, and shift the result of the multiplication two bits higher. A MAC can multiply 1101 by the 1 in the fourth digit from the right of 1011, which is 1, and shift the result of the multiplication three bits higher. MAC can determine the final value by summing the results of the multiplication operation for each digit. In Figure 3b, the result of the multiplication operation between the 4-bit data with "1101" and the 4-bit data with "1011" can be expressed as "10001111."

[0080] Figure 4a illustrates the process of dividing 12-bit data into four sections.

[0081] According to FIG. 4A, a processor (e.g., processor (210) of FIG. 2) may separate the upper four bits of the first bit string data into a first section (401) (IN1) and separate the lower eight bits of the first bit string data into a second section (402) (IN0). The processor (210) may separate the upper four bits of the second bit string data into a third section (403) (IN2) and separate the lower eight bits of the second bit string data into a fourth section (404) (IN3). Alternatively, the processor (210) may separate the upper eight bits of the first bit string data into a first section (401) (IN1) and separate the lower four bits of the first bit string data into a second section (402) (IN0). The processor (210) may separate the upper 8 bits of the second bit string data into a third section (403) (IN2), and may separate the lower 4 bits of the second bit string data into a fourth section (404) (IN3). That is, the first bit string data and the second bit string data may be divided into 8 bits and 4 bits, but 8 bits may be arranged at the upper level and 4 bits may be arranged at the lower level, or conversely, 8 bits may be arranged at the lower level and 4 bits may be arranged at the upper level. At least one of the first bit string data or the second bit string data may include data having 12 bits (e.g., image data).

[0082] The processor (210) can perform multiplication of data of the first section (401), the second section (402), the third section (403), and the fourth section (404), and add the multiplication results to output a final result. The processor (210) can output a result of A8W8 (activation 8-bit, weight 8-bit) or a result of A12W12 (activation 12-bit, weight 12-bit) based on an external input (e.g., user input). The processor (210) can also perform multiplication of data, and before adding the multiplication results, determine in advance which data among A8W8 or A12W12 to output based on an external input.

[0083] The process of outputting at least one of A8W8 or A12W12 data will be described in FIGS. 4B and 4C. In FIG. 4A, a bit string data consisting of 12 bits is described as being divided into upper 4 bits and lower 8 bits, but this is only for convenience of explanation and the data division method is not limited to this. The bit string data can be divided into, for example, upper 8 bits and lower 4 bits, and this division method can vary depending on the settings.

[0084] FIG. 4b illustrates a process of obtaining an A8W8 result through multiplication and addition operations of data separated into four sections. The processor (210) can perform calculations to obtain two A8W8 results in one cycle by combining IN0, IN1, IN2, and IN3. X1 and X2 may represent 8-bit input values. IN may represent the name of an input of MAC. One MAC may execute X1*W and X2*W in one cycle. W may represent an 8-bit weight. The processor (210) may assign the first input X1 to an 8-bit separated section, and assign the weight W to another 8-bit separated section. The processor (210) may assign the lower 4 bits of the second input X2 to an 4-bit separated section (e.g., IN1), and assign the upper 4 bits of X2 to another 4-bit separated section (e.g., IN2). The processor (210) may assign the first input X1 to IN0 and assign the weight W to IN3. Or, conversely, the processor (210) may assign the first input X1 to IN3 and assign the weight W to IN0. Whether X1 and W are assigned may vary depending on the setting.

[0085] The processor (210) may assign the lower four bits of the second input X2 to IN1 and the upper four bits of X2 to IN2. Or, conversely, the processor may assign the lower four bits of the second input X2 to IN2 and the upper four bits of X2 to IN1.

[0086] The method of assigning X1 and X2 and W to IN0, IN1, IN2, and IN3 may vary depending on the setting and is not limited to that illustrated in Fig. 4b. Fig. 4b illustrates an example of assigning the first input X1 to IN0, the weight W to IN3, the lower four bits of the second input X2 to IN2, and the upper four bits of X2 to IN1 according to one embodiment.

[0087] According to FIG. 4B, the processor (210) can perform a multiplication operation of the second section (402) and the fourth section (404) having a size of 8 bits. The processor (210) can obtain a first 8-bit result value corresponding to A8W8 by performing the multiplication operation of the second section (402) and the fourth section (404) having a size of 8 bits. The 8-bit multiplication can be performed as described in FIG. 3B. FIG. 3B illustrates a process of performing a 4-bit multiplication to obtain 8 bits. Similarly, performing an 8-bit multiplication can obtain a 16-bit result. The processor (210) can convert (quantize) the 16-bit value obtained as a result of the 8-bit multiplication to 8 bits. The processor (210) can perform the conversion by multiplying a scaling factor (constant value) and adding a bias when converting from INT16 to INT8. The processor (210) can provide a value converted to 8 bits as input to the next layer.

[0088] In Figure 410, the processor (210) can obtain the first A8W8 result value by performing a product operation of X1 (8 bits) and W (8 bits). This can be performed in the same way as the process in which a 4-bit product operation is performed in Figure 3b.

[0089] In addition, the processor (210) can perform a product operation on the 8-bit data that is the sum of the first section (401) and the third section (403) for the fourth section (404). The processor (210) can perform a product operation on the fourth section (404) and the first section (401) to determine a first result value. The processor (210) can perform a product operation on the fourth section (404) and the third section (403), and determine a second result value that is shifted upward by 4 bits using a bit shifter for the result of the product operation. The processor (210) can obtain a second 8-bit result value corresponding to A8W8 by adding the first result value and the second result value.

[0090] In Figure 412, the processor (210) can perform a product operation of X2 (8 bits) and W (8 bits). X2 can be composed of IN1 including the first to fourth bits from the right and IN2 including the fifth to eighth bits. The processor (210) can perform a product operation of W (8 bits) and IN1. In addition, the processor (210) can perform a product operation of W (8 bits) and IN2. However, since IN2 includes the fifth to eighth bits, the first to fourth bits from the right may be omitted.

[0091] Figure 420 illustrates a process of performing a product operation with W (8 bits) by adding digits to IN2, where the first to fourth bits are omitted from the right. The processor (210) can perform a product operation of IN3 and IN1, and a product operation of IN3 and IN2. Thereafter, the processor (210) can add the product operation value of IN3*IN2 to the product operation value of IN3 and IN1 with a four-bit shift in the upper bit direction. The processor (210) can determine the product operation result value of X2 and W by adding the product operation value of IN3*IN2 to the product operation value of IN3 and IN1 with a four-bit shift in the upper bit direction. The processor (210) can obtain the second A8W8 result value by multiplying each of the four-bit values ​​by an eight-bit value and then adding them with a four-bit shift in the upper bit direction. That is, in Figure 4b, the processor (210) can output two 8-bit result values.

[0092] Figure 4c illustrates the process of obtaining the A12W12 result through multiplication and sum operations of data separated into four sections.

[0093] According to FIG. 4c, the processor (210) can perform a multiplication operation by dividing two bit string data having a size of 12 bits into a first section (401) to a fourth section (404). X can mean a 12-bit input. W can mean a 12-bit weight. One MAC can perform a multiplication operation between X and W within one cycle. When the processor (210) assigns X to IN0 and IN1, it can assign W to IN2 and IN3. Conversely, when W is assigned to IN0 and IN1, the processor (210) can assign X to IN2 and IN3. For convenience of explanation, the following description and drawings only illustrate an embodiment in which X is assigned to IN0 and IN1 and W is assigned to IN2 and IN3, but this is only one embodiment and the positions to which X and W are assigned may vary depending on the setting. The processor (210) can assign the upper four bits of the input X to IN1. In addition, the processor (210) can assign the lower 8 bits of the input X to IN0. The processor (210) can assign the upper 4 bits of W to IN2. The processor (210) can assign the lower 8 bits of W to IN3. The processor (210) can perform a product operation of IN0 and IN3 and add IN2*IN1 with a 16-bit shift in the upper bit direction and IN0*IN2 and IN3*IN1 with an 8-bit shift in the upper bit direction to the result value to obtain the result value of the product operation of X and W. The processor (210) can obtain the result value of the product operation of X and W without using a bit shift.

[0094] In Figure 420, the processor (210) can perform a multiplication operation of IN0 and IN3, which have a size of 8 bits. The multiplication operation of IN0 and IN3 can be performed as described in Figure 3b, with multiplication between 8 bits.

[0095] In Figure 422, the processor (210) can perform a product operation of IN2 and IN1 having a size of 4 bits. IN2 can include the upper 4 bits of W, and IN1 can include the upper 4 bits of input X. The electronic device according to this document (e.g., the electronic device (200) of Figure 2) can perform a product operation of IN0 and IN3, and add IN0*IN2, IN3*IN1 to the result value with an upper bit shift of eight bits, and IN2*IN1 with an upper bit shift of sixteen bits, to obtain a result value of the product operation of X and W. The electronic device (200) can obtain a result value of the product operation of X and W without using a bit shift.

[0096] In Figure 424, the processor (210) can perform a product operation of IN3 having a size of 8 bits and IN1 having a size of 4 bits. The processor (210) can perform a product operation with IN3 by shifting IN1 by 8 bits in the upper bit direction.

[0097] In Figure 426, the processor (210) can perform a product operation of IN0 having a size of 8 bits and IN2 having a size of 4 bits. The processor (210) can perform a product operation with IN0 by shifting IN2 by 8 bits in the upper bit direction.

[0098] The processor (210) can perform the sum operation of Figures 420 and 422, and the sum operation of Figures 424 and 426. The processor (210) can determine a 12-bit output value corresponding to A12W12 by adding the result values ​​of the sum operation of Figures 420 and 422 and the result values ​​of the sum operation of Figures 424 and 426.

[0099] That is, in FIG. 4c, the processor (210) can output one 12-bit result value in one cycle.

[0100] Figure 5a illustrates the process of obtaining the A8W8 result through multiplication and addition operations using MAC.

[0101] The process of obtaining an 8-bit result corresponding to A8W8 has been described in the previous Figure 4b. The structure and operation of the MAC (multiply and accumulate) have been described in Figure 3a.

[0102] In Fig. 5a, it is explained that one bit string data consisting of 12 bits is divided into upper 4 bits and lower 8 bits, but this is only for convenience of explanation and the data division method is not limited to this. The bit string data may be divided into, for example, upper 8 bits and lower 4 bits, and this division method may vary depending on the settings. The process of determining the input values ​​of IN0 (8 bits) and IN1 (4 bits) is described in Fig. 4b, and Figs. 5a and 5b are one of various embodiments and are not limited thereto.

[0103] Figure 501 illustrates a process in which an electronic device (e.g., the electronic device (200) of FIG. 2) performs convolution under the control of a processor (e.g., the processor (210) of FIG. 2). In cycle 0, X1 may be assigned to Xooo, X2 may be assigned to Xo1o, and W may be assigned to Woo. One MAC unit may perform a product operation of Xooo and Woo, and a product operation of Xo1o and Woo in one cycle (e.g., cycle 0). The electronic device (200) may obtain two A8W8 result values ​​in one cycle by using the result values ​​of the two product operations.

[0104] The electronic device (200) can perform a MAC (multiply and accumulate) operation with Xoo1, Xo11, and Wo1 in cycle 1. The electronic device (200) can perform a MAC (multiply and accumulate) operation with Xooi, Xo1i, and Woi in cycle i. i can mean 0 and a natural number.

[0105] Figure 503 illustrates a circuit diagram that performs MAC operations.

[0106] The electronic device (200) can perform a product operation of Woo and Xooo in cycle 0 and store the result on the first accumulator (510). The electronic device (200) can perform a product operation of Woi and Xooi in cycle i and store the result on the first accumulator (510). Here, i can include a natural number. The electronic device (200) can perform the product operation from cycle 0 to cycle i and accumulate and store the results on the first accumulator (510). The first accumulator (510) can include an adder (512) and a register (514). The first accumulator (510) can add a result value of the product operation determined in cycle 0 and a result value of the product operation determined in cycle i using the adder (512). Thereafter, the first accumulator (510) can add the result values ​​and register the added result in the register (514).

[0107] The electronic device (200) can perform a multiplication operation of Woo and Xo1o in cycle 0 and store the result on the second accumulator (520). The electronic device (200) can perform a multiplication operation of Woi and Xo1i in cycle i and store the result on the second accumulator (520). Here, i can include a natural number. The electronic device (200) can perform a multiplication operation from cycle 0 to cycle i and accumulate and store the results on the second accumulator (520). The multiplication between 8 bits and 4 bits is as described in the above FIG. 3A. In FIG. 503, the section corresponding to the 4-bit multiplication (4b x 4b) is not used and can be disabled. The multiplexer (530) can select one of the two inputs based on the control signal. In FIG. 503, the input corresponding to the 4-bit multiplication (4b x 4b) is not selected and can be disabled.

[0108] FIG. 5b illustrates a first embodiment of sharing input values ​​among multiple MACs when there is one or more MACs as illustrated in FIG. 5a.

[0109] According to FIG. 5b, an electronic device (e.g., the electronic device (200) of FIG. 2) may include multiple MACs. Each of the multiple MACs may perform the computational process described in FIG. 5a. Each of the multiple MACs may output at least one A8W8 result value in one cycle.

[0110] The electronic device (200) can share input values ​​between multiple MACs to reduce communication overhead. For example, in the case of A8W8, the electronic device (200) can share a total of 16 bits, including IN0 (8 bits), IN1 (4 bits), and IN2 (4 bits).

[0111] The MAC k unit can receive different 1x1 filter weights for each cycle starting from the MAC 0 unit. k can represent a natural number. The electronic device (200) can provide IN0 (8 bits), IN1 (4 bits), and IN2 (4 bits) as the same input values ​​to each MAC. For example, the electronic device (200) can determine IN0 (8 bits), IN1 (4 bits), and IN2 (4 bits) using inputs X1 and X2. X1 and X2 can represent 8-bit input values. The process of dividing IN0, IN1, IN2, and IN3 into 8-bit and 4-bit sections and determining input values ​​has been described in FIGS. 4A and 4B. The electronic device (200) can determine IN0 (8 bits), IN1 (4 bits), and IN2 (4 bits) corresponding to X1 and X2, and share the determined values ​​with multiple MACs.

[0112] In addition, the electronic device (200) can determine an 8-bit IN3 value by assigning different weights (e.g., Woo, Wo1) for each cycle using filter 0 and transmit it on MAC 0. However, the IN3 value may not be shared with other MACs, unlike IN0 (8 bits), IN1 (4 bits) and IN2 (4 bits). The electronic device (200) can determine an 8-bit IN3 value by assigning different weights (e.g., W11, W1o) for each cycle using filter 1 and transmit it on MAC 1. The electronic device (200) can determine an 8-bit IN3 value by assigning different weights (e.g., Wk1, Wko) for each cycle using filter k and transmit it on MAC k. Here, k means 0 and a natural number.

[0113] Figure 540 illustrates a process in which an electronic device (200) performs convolution under the control of a processor (210). In cycle 0, Xooo may be assigned X1, Xo1o may be assigned X2, and Woo may be assigned W. One MAC unit may perform a product operation of Xooo and Woo and a product operation of Xo1o and Woo in one cycle (e.g., cycle 0). The electronic device (200) may obtain a result value of A8W8 using the result values ​​of the two product operations. In cycle 1, the electronic device (200) may perform a MAC (multiply and accumulate) operation with Xoo1, Xo11, and Wo1. The electronic device (200) may perform a MAC (multiply and accumulate) operation with Xooi, Xo1i, and Woi in cycle i. i may mean 0 and a natural number. The characters representing natural numbers may include i, j, and k, but this is only a difference in notation, and the meaning of each character may be the same as representing 0 and natural numbers.

[0114] The electronic device (200) can perform calculations of multiple filters in parallel for the same input value. Unlike the serial connection method in which calculations are performed on one MAC and then the result value is waited for before the calculation is performed, the electronic device (200) according to this document can perform calculations simultaneously on multiple MACs, thereby reducing the latency time due to calculation performance. The intermediate output results of the convolution can be accumulated in the register of the accumulator in the MAC. The MAC can store the accumulated result value in memory (e.g., SRAM).

[0115] FIG. 6a illustrates a process of obtaining an A12W12 result through multiplication and addition operations using MAC according to various embodiments.

[0116] According to FIG. 6A, a processor (e.g., processor (210) of FIG. 2) may separate the upper four bits of the first bit string data into a first section (e.g., first section (401) of FIG. 4A) (IN1) for two 12-bit data, and separate the lower eight bits of the first bit string data into a second section (e.g., second section (402) of FIG. 4A) (IN0). The processor (210) may separate the upper four bits of the second bit string data into a third section (e.g., third section (403) of FIG. 4A) (IN2), and separate the lower eight bits of the second bit string data into a fourth section (e.g., fourth section (404) of FIG. 4A) (IN3). The process of dividing IN0, IN1, IN2, and IN3 into 8-bit and 4-bit sections and determining the input values ​​is described in FIGS. 4A and 4B . In FIG. 6A , for convenience, the upper 4 bits and the lower 8 bits are described as being separated, but the separation method is not limited to this and may vary depending on the settings. The first bit string data and the second bit string data may include data having 12 bits (e.g., image data).

[0117] Figure 610 illustrates the process in which an electronic device (200) performs convolution.

[0118] In cycle 0, input value X1 can be assigned to Xooo, and weight W can be assigned to Woo. X1 and W can contain 12-bit data. The 12-bit data can be divided into upper 4 bits and lower 8 bits, or upper 8 bits and lower 4 bits. In Fig. 6a, it is assumed that 12-bit data is divided into upper 4 bits and lower 8 bits, but the division method is not limited to this and can vary depending on the setting.

[0119] One MAC unit can perform a multiplication operation of Xooo and Woo in one cycle (e.g., cycle 0). The electronic device (200) can perform a MAC (multiply and accumulate) operation on X1 and W. The electronic device (200) can perform an 8-bit multiplication of X1 and W and obtain a first result value.

[0120] The electronic device (200) can perform a product between 8 bits of X1 and 4 bits of W. And the electronic device (200) can perform a product between 8 bits of W and 4 bits of X1. The electronic device (200) can perform a product between 8 bits and 4 bits and add the respective result values ​​to obtain a second result value.

[0121] The electronic device (200) can perform a product between the four bits of X1 and the four bits of W and obtain a third result value. The electronic device (200) can add the first result value, the second result value, and the third result value to obtain the A12W12 result for one cycle (e.g., cycle 0).

[0122] The electronic device (200) can perform a MAC (multiply and accumulate) operation on Xooi and Woi in cycle i. i can mean 0 and a natural number.

[0123] One MAC unit can perform one A12W12 multiply and accumulate operation by multiplying Xooi and Woi in cycle i, and accumulate and store the result on the second accumulator (620). One MAC unit can repeatedly perform multiply and accumulate operations from cycle 0, cycle 1 to cycle i, and accumulate and store the result on the second accumulator (620). The first accumulator can be disabled.

[0124] A may represent the result of an 8-bit and 4-bit multiplication operation. C may represent the result of a 4-bit multiplication operation. The electronic device (200) may perform a sum operation by shifting the upper bits of A by 8 bits and C by 16 bits. The electronic device (200) may perform a sum operation by shifting the upper bits of A by 8 bits and C by 16 bits without hardware for a bit shifter. The multiplexers (612, 614, 616) may select one of the two inputs according to a control signal. In FIG. 6A, an input path that is not selected on the multiplexers (612, 614, 616) may be indicated as inactive.

[0125] FIG. 6B illustrates a second embodiment of sharing input values ​​among multiple MACs according to various embodiments. According to FIG. 6B, a processor (e.g., processor (210) of FIG. 2) may separate the upper four bits of the first bit string data into a first section (e.g., first section (401) of FIG. 4A) (IN1) for two 12-bit data, and separate the lower eight bits of the first bit string data into a second section (e.g., second section (402) of FIG. 4A) (IN0). The processor (210) may separate the upper four bits of the second bit string data into a third section (e.g., third section (403) of FIG. 4A) (IN2), and separate the lower eight bits of the second bit string data into a fourth section (e.g., fourth section (404) of FIG. 4A) (IN3). The first bit string data and the second bit string data may include data having 12 bits (e.g., image data). Each of the multiple MACs may perform the operation process described in FIG. 6A. Each of the multiple MACs may output one A12W12 result value in one cycle.

[0126] The electronic device (200) can share input values ​​between multiple MACs to reduce communication overhead. For example, in the case of A12W12, the electronic device (200) can share a total of 12 bits, including IN0 (8 bits) and IN1 (4 bits) corresponding to the X value.

[0127] The MAC k unit can receive different 1x1 filter weights for each cycle starting from the MAC 0 unit. k can represent a natural number. The electronic device (200) can provide IN0 (8 bits) and IN1 (4 bits) as the same input values ​​to each MAC. For example, the electronic device (200) can determine IN0 (8 bits) and IN1 (4 bits) using the input X1. X1 can represent a 12-bit input value. The process of determining the input values ​​of IN0 and IN1 has been described in FIG. 4b, and FIG. 6 is one of various embodiments and is not limited thereto. The electronic device (200) can determine IN0 (8 bits) and IN1 (4 bits) corresponding to X1 and share the determined values ​​with multiple MACs. In addition, the electronic device (200) can determine a 4-bit IN2 value and an 8-bit IN3 value by assigning different weights (e.g., Woo, Wo1) to each cycle using filter 0 and transmit them on MAC 0. However, the IN2 and IN3 values ​​may not be shared with other MACs, unlike IN0 (8 bits) and IN1 (4 bits). The electronic device (200) can determine a 4-bit IN2 value and an 8-bit IN3 value by assigning different weights (e.g., W11, W1o) to each cycle using filter 1 and transmit them on MAC 1. The electronic device (200) can determine a 4-bit IN2 value and an 8-bit IN3 value by assigning different weights (e.g., Wk1, Wko) to each cycle using filter k and transmit them on MAC k. Here, k means 0 and a natural number.

[0128] The electronic device (200) can perform calculations of multiple filters in parallel for the same input value. Unlike the serial connection method in which calculations are performed on one MAC and then the result value is waited for before the calculation is performed, the electronic device (200) according to this document can perform calculations simultaneously on multiple MACs, thereby reducing the latency time due to calculation performance. The intermediate output results of the convolution can be accumulated in the register of the accumulator in the MAC. The MAC can store the accumulated result value in memory (e.g., SRAM).

[0129] Figures 7a and 7b illustrate the process of obtaining A8W8 and A12W12 results by packing data into 8-bit units and 4-bit units, respectively.

[0130] In Figure 7a, the data corresponding to A8W8 may have an 8-bit size. A8W8 may represent an 8-bit activation and an 8-bit weight. Activation may represent the size of the input data. Weight may represent a variable that controls the influence of the input data on the result.

[0131] Data corresponding to A12W12 can have a size of 12 bits. Since typical computer systems use 32 or 64 bits, when receiving 12-bit data, both 32 and 64 bits may be left over. For example, when receiving 12-bit data in a 32-bit computer system, 8 bits may be left over. Also, when receiving 12-bit data in a 64-bit computer system, 4 bits may be left over. For example, when training data in the A12W12 format, A12W12 may be more efficient than other systems (e.g., A8W8), but this may result in wasted bits.

[0132] The electronic device disclosed in FIG. 7A (e.g., the electronic device (200) of FIG. 2) can perform operations by dividing 12-bit data into 4-bit and 8-bit data. The electronic device (200) can perform operations on 12 bits and 8 bits simultaneously by adding only three multiplexers (mux) and one bit shifter without adding a separate operation device. According to one embodiment, the processor (e.g., the processor (210) of FIG. 2) can determine whether to output the result of A8W8 (activation 8-bit, weight 8-bit) or the result of A12W12 (activation 12-bit, weight 12-bit) based on an external input. The electronic device (200) according to the present document can selectively output either an 8-bit result or a 12-bit result by only varying the calculation method on one MAC. The electronic device (200) can selectively perform deep learning based on learning for 8-bit data and learning for 12-bit data.

[0133] According to FIG. 7a, the NPU SRAM (710) can load data for input X and weight W from DRAM (not shown) internally. The loaded data can be moved to a buffer (720) before being transmitted to MAC units. The electronic device (200) can move at least some of the data of the SRAM (710) to the buffer (720).

[0134] According to FIG. 7b, the electronic device (200) can load three rows of data lines on the SRAM (710) for A8W8 and four rows of data lines on the SRAM (710) for A12W12.

[0135] The electronic device (200) can provide data contained in the buffer (720) as inputs of the MAC using a wire or a multiplexer (mux). For example, the electronic device (200) can provide X010[0:3] as an input of MAC_0_IN1 for A8W8 and X000[8:11] as an input of MAC_0_IN1 for A12W12. For example, the electronic device (200) can provide X010[4:7] as an input of MAC_0_IN2 for A8W8 and W00[8:11] as an input of MAC_0_IN2 for A12W12.

[0136] FIG. 7c illustrates a process of performing a 3x3 convolution operation for A8W8 according to one embodiment.

[0137] According to FIG. 7c, an electronic device (e.g., electronic device (200) of FIG. 2) can additionally accumulate 8 times while moving the feature map corresponding to 1x1 to perform a 3x3 convolution operation.

[0138] The electronic device (200) can perform operations by accumulating 1x1x(j+1) MAC (multiply-and-accumulate) operations from cycle 0 to cycle j under the control of a processor (e.g., processor (210) of FIG. 2), and accumulating operations from cycle j+1 to cycle 2j+1.

[0139] The electronic device (200) can obtain a total of two 3x3x(j+1) convolution result values ​​by moving the feature map corresponding to 1x1 and accumulating it a total of nine times.

[0140] FIG. 7c illustrates an embodiment of an operation in which, in the case of A8W8, one MAC simultaneously applies a convolution operation to two feature map regions with the same 3x3x(j+1) filter, thereby obtaining two values ​​of an output feature map in parallel from two accumulators of the MAC.

[0141] The first accumulator of MAC accumulates 1x1x(j+1) multiply-and-accumulate operations from cyc 0 to cyc j to obtain one value of the output feature map, and then moves the feature map region by one space to perform another accumulation operation from cyc j+1 to cyc 2j+1. The first accumulator of MAC can accumulate these 1x1x(j+1) convolution operations a total of 9 times in a 3x3 region to obtain the result of a 3x3x(j+1) convolution operation.

[0142] The second accumulator of the MAC can perform operations on the 3x3x(j+1) region one space adjacent to the feature map region where the operation was performed on the first accumulator. The second accumulator of the MAC can perform operations in the same process as the first accumulator. The second accumulator of the MAC can obtain the second result of the 3x3x(j+1) convolution operation.

[0143] FIG. 8 is a flowchart illustrating an operation method of an electronic device that performs a calculation using an artificial intelligence model according to one embodiment.

[0144] The operations described through FIG. 8 may be implemented based on instructions that may be stored in a computer recording medium or memory (e.g., memory (220) of FIG. 2). The illustrated method (800) may be executed by an electronic device (e.g., electronic device (200) of FIG. 2) described above through FIGS. 1 to 7, and the technical features described above will be omitted below. The order of each operation of FIG. 8 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0145] In operation 810, a processor (e.g., processor (210) of FIG. 2) may receive first bit-string data and second bit-string data composed of 12 bits. The processor (210) may receive image data composed of 12 bits from a sensor (e.g., a camera). The 12-bit bit-string data may be composed of 12 bits from the beginning. Alternatively, the 12-bit bit-string data may be generated when other bit (e.g., 8-bit or 16-bit) data is converted (preprocessed) into a 12-bit format.

[0146] In operation 820, the processor (210) may divide the first bit string data and the second bit string data into four sections. In one embodiment, the processor (210) may divide the upper four bits of the first bit string data into a first section, and divide the lower eight bits of the first bit string data into a second section. The processor (210) may divide the upper four bits of the second bit string data into a third section, and divide the lower eight bits of the second bit string data into a fourth section. Alternatively, in one embodiment, the processor (210) may divide the upper eight bits of the first bit string data into a first section, and divide the lower four bits of the first bit string data into a second section. The processor (210) can separate the upper 8 bits of the second bit string data into a third section and separate the lower 4 bits of the second bit string data into a fourth section. In the following description, it is assumed that the operation is performed by separating the upper 4 bits of the first bit string data into a first section and the lower 8 bits into a second section, but the separation method is not limited thereto and may vary depending on the setting.

[0147] In operation 830, the processor (210) may perform a multiplication of four interval data to output an A8W8 or A12W12 result. The processor (210) may perform a multiplication of data of bits corresponding to the first interval or bits corresponding to the fourth interval, add the multiplication results, and output an A8W8 (activation 8-bit, weight 8-bit) result or an A12W12 (activation 12-bit, weight 12-bit) result.

[0148] According to one embodiment, the processor (210) may perform multiplication of data of the first to fourth sections using a MAC unit and add the multiplication results. The processor (210) may determine whether to output the result of A8W8 (activation 8-bit, weight 8-bit) or the result of A12W12 (activation 12-bit, weight 12-bit) based on an external input (e.g., user input).

[0149] According to one embodiment, the arithmetic circuit may include a plurality of multiplexers, a bit shifter, an adder, and an accumulator. The processor (210) may select a first path capable of outputting a result of A8W8 (activation 8-bit, weight 8-bit) or a second path capable of outputting a result of A12W12 (activation 12-bit, weight 12-bit) based on a user input within the arithmetic circuit.

[0150] According to one embodiment, the processor (210) may select a third path that may output the result of A16W16 (activation 16-bit, weight 16-bit) based on user input.

[0151] The electronic device (200) may include an A8W8-A12W12 MAC array and an A16W16 (or FP16) MAC array, respectively. The electronic device (200) may selectively use either the A8W8-A12W12 MAC array or the A16W16 (or FP16) MAC array based on user input.

[0152] An electronic device (200) that performs calculations using an artificial intelligence model according to this document can output the result of A8W8 or the result of A12W12 for 12-bit data input by only changing the calculation method for the same circuit. The electronic device (200) can support both A8W8 calculations and A12W12 calculations by adding only three multiplexers and one bit shifter, thereby increasing the efficiency of deep learning for image data.

[0153] An electronic device performing a MAC (multiply and accumulate) operation includes at least one MAC unit, a static random access memory (SRAM), one 8bx8b operator, two 8bx4b operators, one 4bx4b operator, a bit shifter, an adder, at least one accumulator, and a processor, wherein the processor receives first bit-string data and second bit-string data composed of 12 bits, divides the first bit-string data into 4 bits and 8 bits, divides the second bit-string data into 4 bits and 8 bits, and outputs two 8-bit results corresponding to A8W8 using the bit shifter and the first adder based on which it is determined to output a result of A8W8 (activation 8-bit, weight 8-bit), and outputs a result of A12W12 (activation 12-bit, weight 12-bit). It can output one 12-bit result corresponding to A12W12.

[0154] The processor may perform a product operation between 8 bits of the first bit string data and 4 bits of the first bit string data using at least one 8bx4b operator, perform a product operation between 8 bits of the first bit string data and 4 bits of the second bit string data using at least one 8bx4b operator, and output a first result corresponding to A8W8 using a bit shifter and a first accumulator. The processor may perform a product operation between 8 bits of the first bit string data and 8 bits of the second bit string data using at least one 8bx8b operator, and output a second result corresponding to A8W8 using a second accumulator. The processor may perform a product operation between 8 bits of the first bit string data and 8 bits of the second bit string data using at least one 8bx8b operator based on a determination that the processor outputs a result of A12W12 (activation 12-bit, weight 12-bit), perform a product operation between 4 bits of the first bit string data and 4 bits of the second bit string data using at least one 4bx4b operator, perform a product operation between 8 bits of the first bit string data and 4 bits of the second bit string data using at least one 8bx4b operator, perform a product operation between 8 bits of the second bit string data and 4 bits of the first bit string data using at least one 8bx4b operator, and output a third result corresponding to A12W12 using at least one accumulator.

[0155] The processor may assign a weight input to eight bits of the first bit-string data, assign first data to eight bits of the second bit-string data, assign second data to four bits of the first bit-string data and the second bit-string data based on a determination that the processor outputs a result of A8W8 (activation 8-bit, weight 8-bit), perform a product operation between eight bits of the first bit-string data and four bits of the first bit-string data and the second bit-string data, respectively, output a first result using a bit shifter and a first accumulator, and perform a product operation between eight bits of the first bit-string data and eight bits of the second bit-string data to output a second result using a second accumulator.

[0156] The processor may assign weight inputs to eight bits of the first bit-string data and four bits of the first bit-string data based on a determination to output a result of A12W12 (activation 12-bit, weight 12-bit), assign third data to eight bits and four bits of the second bit-string data, perform a product operation between eight bits of the first bit-string data and eight bits of the second bit-string data, perform a product operation between eight bits of the first bit-string data and four bits of the second bit-string data, perform a product operation between eight bits of the second bit-string data and four bits of the first bit-string data, perform a product operation between four bits of the first bit-string data and four bits of the second bit-string data, and output a third result using a bit shifter and at least one accumulator.

[0157] The first section refers to four bits of the first bit string data, the second section refers to eight bits of the first bit string data, the third section refers to four bits of the second bit string data, and the fourth section refers to eight bits of the second bit string data, and the processor can perform a multiplication on the bits corresponding to the second section and the bits corresponding to the fourth section to determine a first result value, perform a multiplication on the bits corresponding to the fourth section and the bits corresponding to the third section, shift four bits in an upward direction using a bit shifter for the result value obtained by the multiplication to determine a second result value, perform a multiplication on the bits corresponding to the fourth section and the bits corresponding to the first section to determine a third result value, output the first result value, perform a sum operation on the second result value and the third result value, and output the result.

[0158] The processor can perform a product operation on the bits corresponding to the second section and the bits corresponding to the fourth section and store the result.

[0159] The processor can perform a product operation of 8-bit divided sections on first bit string data and second bit string data composed of 12 bits to determine a first value, perform a product operation of 8-bit and 4-bit sections to determine a second value, perform a product operation of 4-bit divided sections to determine a third value, and accumulate and store the result by adding the first value, the second value, and the third value.

[0160] An electronic device includes an arithmetic circuit, a memory, and a processor, wherein the arithmetic circuit includes a plurality of multiplexers, a bit shifter, an adder, and an accumulator, and the processor can select a first path capable of outputting a result of A8W8 (activation 8-bit, weight 8-bit) or a second path capable of outputting a result of A12W12 (activation 12-bit, weight 12-bit) based on a user input within the arithmetic circuit.

[0161] An operating method of an electronic device that performs a calculation using an artificial intelligence model may include an operation of receiving first bit-string data and second bit-string data composed of 12 bits, an operation of dividing the first bit-string data into four bits and eight bits, an operation of dividing the second bit-string data into four bits and eight bits, an operation of outputting two 8-bit results corresponding to A8W8 using a bit shifter and a first adder based on a determination that a result of A8W8 (activation 8-bit, weight 8-bit) is to be output, and an operation of outputting one 12-bit result corresponding to A12W12 based on a determination that a result of A12W12 (activation 12-bit, weight 12-bit) is to be output.

Claims

1. In electronic devices, Memory for storing instructions; and comprising at least one processor, At least one of the above processors comprising at least one 8bx8b operator, at least one 8bx4b operator, at least one 4bx4b operator, a bit shifter, an adder and at least one accumulator, The above instructions, when executed by the at least one processor, cause the electronic device to: Receive first bit string data and second bit string data consisting of 12 bits, Separate the above first bit string data into 4 consecutive bits and 8 consecutive bits, The above second bit string data is divided into 4 consecutive bits and 8 consecutive bits, Based on the decision to output the result of A8W8 (activation 8-bit, weight 8-bit) Using at least one 8bx4b operator above A product operation is performed between 8 bits of the first bit string data and 4 bits of the first bit string data, and a product operation is performed between 8 bits of the first bit string data and 4 bits of the second bit string data. Using the above bit shifter and the first accumulator, the first result corresponding to A8W8 is output, Using at least one 8bx8b operator above Using at least one operator, a product operation is performed between 8 bits of the first bit string data and 8 bits of the second bit string data. Control to output the second result corresponding to A8W8 using the second accumulator, Based on the decision to output the result of A12W12 (activation 12-bit, weight 12-bit) Using at least one 8bx8b operator, a product operation is performed between 8 bits of the first bit string data and 8 bits of the second bit string data, Using at least one 4bx4b operator, a product operation is performed between four bits of the first bit string data and four bits of the second bit string data, Using at least one 8bx4b operator, a product operation is performed between 8 bits of the first bit string data and 4 bits of the second bit string data, and a product operation is performed between 8 bits of the second bit string data and 4 bits of the first bit string data, An electronic device that controls at least one accumulator to output a third result corresponding to A12W12.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the decision to output the result of A8W8 (activation 8-bit, weight 8-bit) A weight input is assigned to the eight bits of the first bit string data above, The first data is allocated to the eight bits of the above second bit string data, Second data is allocated to four bits of the first bit string data and the second bit string data, Using at least one 8bx4b operator, a product operation is performed between eight bits of the first bit string data and four bits of the first bit string data and the second bit string data, and a first result is output using the bit shifter and the first accumulator. An electronic device that performs a product operation between eight bits of the first bit string data and eight bits of the second bit string data using at least one 8bx8b operator, and controls the output of a second result using a second accumulator.

3. In paragraph 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the decision to output the result of A12W12 (activation 12-bit, weight 12-bit) A weight input is assigned to eight bits of the first bit string data and four bits of the first bit string data, Third data is allocated to eight bits and four bits of the above second bit string data, Using at least one 8bx8b operator, a product operation is performed between eight bits of the first bit string data and eight bits of the second bit string data, Using at least one 8bx4b operator, a product operation is performed between eight bits of the first bit string data and four bits of the second bit string data, Using at least one 8bx4b operator, a product operation is performed between eight bits of the second bit string data and four bits of the first bit string data, Using at least one 4bx4b operator, a product operation is performed between four bits of the first bit string data and four bits of the second bit string data, An electronic device controlling the bit shifter and at least one accumulator to output a third result.

4. In paragraph 1, The first section refers to four bits of the first bit string data, The second section refers to eight bits of the first bit string data, The third section refers to four bits of the second bit string data, The fourth section refers to eight bits of the second bit string data, The above instructions, when executed by the at least one processor, cause the electronic device to: A first result value is determined by performing a multiplication on the bit corresponding to the second section and the bit corresponding to the fourth section, A multiplication is performed on the bit corresponding to the fourth section and the bit corresponding to the third section, and the result value obtained by the multiplication is shifted by 4 bits in the upper direction using the bit shifter to determine a second result value, and a multiplication is performed on the bit corresponding to the fourth section and the bit corresponding to the first section to determine a third result value. Output the first result value above, An electronic device that controls to perform a sum operation on the second result value and the third result value and output the result.

5. In paragraph 4, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that controls to perform a product operation on bits corresponding to the second section and bits corresponding to the fourth section and to store the result.

6. In paragraph 4, The above instructions, when executed by the at least one processor, cause the electronic device to: A product operation is performed on the 8-bit divided sections for the first bit string data and the second bit string data consisting of 12 bits, and the first value is determined. Performs a product operation of 8-bit and 4-bit intervals and determines the second value, Performs a product operation on the sections divided into 4 bits and determines the third value. An electronic device that controls the accumulation and storage of a result by adding the first value, the second value, and the third value.

7. In electronic devices, Operation circuit; Memory for storing instructions; and comprising at least one processor, The above operation circuit It includes multiple multiplexers, bit shifters, adders and accumulators. The above instructions, when executed by the at least one processor, cause the electronic device to: Based on the user input, select the first path that can output the result of A8W8 (activation 8-bit, weight 8-bit) within the above operation circuit, or Or an electronic device that controls selection of a second path that can output the result of A12W12 (activation 12-bit, weight 12-bit).

8. In paragraph 7, The above instructions, when executed by the at least one processor, cause the electronic device to: An electronic device that controls selection of a third path that can output the result of A16W16 (activation 16-bit, weight 16-bit) based on user input.

9. In paragraph 7, An electronic device wherein the above instructions, when executed by the at least one processor, control the electronic device to output two 8-bit results corresponding to A8W8 or one 12-bit result corresponding to A12W12 (activation 12-bit, weight 12-bit).

10. A method of operating an electronic device that performs a calculation using an artificial intelligence model, An operation of receiving first bit string data and second bit string data consisting of 12 bits; An operation of dividing the above first bit string data into four bits and eight bits; An operation of dividing the above second bit string data into four bits and eight bits; Based on the decision to output the result of A8W8 (activation 8-bit, weight 8-bit) An operation that outputs two 8-bit results corresponding to A8W8 using a bit shifter and a first adder; Based on the decision to output the result of A12W12 (activation 12-bit, weight 12-bit) A method comprising an operation that outputs one 12-bit result corresponding to A12W12.

11. In Article 10, An operation of dividing the upper four bits of the first bit string data into a first section, and dividing the lower eight bits of the first bit string data into a second section; An operation of dividing the upper four bits of the second bit string data into a third section, and dividing the lower eight bits of the second bit string data into a fourth section; An operation of performing a product operation between the second section and the fourth section corresponding to the lower eight bits and outputting the result based on a decision to output the result of A8W8 (activation 8-bit, weight 8-bit); A method including an operation of generating the third section as eight-bit data using a bit shifter and performing a multiplication operation with the fourth section, and performing a multiplication operation between the first section and the fourth section.

12. In paragraph 11, Based on the decision to output the result of A12W12 (activation 12-bit, weight 12-bit) An operation of performing a product operation between the second section and the fourth section corresponding to the lower eight bits; and Perform a product operation between the sections corresponding to the lower eight bits and the upper four bits, A method further comprising an operation of performing a product operation between the first section and the third section corresponding to the upper four bits.

13. In paragraph 11, A first result value is determined by performing a multiplication on the bit corresponding to the second section and the bit corresponding to the fourth section, A multiplication is performed on the bit corresponding to the fourth section and the bit corresponding to the third section, and the result value obtained by the multiplication is shifted by 4 bits in the upper direction using the bit shifter to determine a second result value, and a multiplication is performed on the bit corresponding to the fourth section and the bit corresponding to the first section to determine a third result value. Output the first result value above, A method further comprising an operation of performing a sum operation on the second result value and the third result value and outputting the result.

14. In paragraph 11, A first result value is determined by performing a multiplication on the bit corresponding to the second section and the bit corresponding to the fourth section, Performing a multiplication on the bits corresponding to the second section and the bits corresponding to the third section, and shifting the result of the multiplication operation by 8 bits in the upper direction using the bit shifter to determine the second result value, Performing a multiplication on the bits corresponding to the first section and the bits corresponding to the third section, and shifting the result of the multiplication operation by 8 bits in the upper direction using the bit shifter to determine the third result value, A multiplication is performed on the bits corresponding to the first section and the bits corresponding to the fourth section, and the result of the multiplication operation is shifted upward by 8 bits using the bit shifter to determine the fourth result value. A method further comprising an operation of performing a sum operation on the first result value to the fourth result value and outputting the result.

15. In paragraph 11, A method further comprising an operation of performing a product operation on bits corresponding to the second section and bits corresponding to the fourth section and storing the result.

Images