Neural network device, method of operating a neural network device, and application processor

Abstract

A neural network apparatus, a method of operating a neural network apparatus, and an application processor are disclosed. The neural network apparatus includes a floating point operation circuit configured to perform a dot product operation and an accumulation operation, and a buffer configured to store first accumulated data generated by the floating point operation circuit, wherein the floating point operation circuit is further configured to perform the dot product operation and the accumulation operation by identifying a maximum value from a mantissa value of the first accumulated data and a plurality of mantissa addition results obtained by respectively adding mantissas of a plurality of floating point data pairs, performing, based on the maximum value, an aligned shift of a mantissa part of the first accumulated data and a plurality of aligned mantissa multiplication results obtained by respectively multiplying mantissas of the plurality of floating point data pairs, and performing a summation of the plurality of aligned mantissa multiplication results and the aligned mantissa part of the first accumulated data.

Description

[0001] This application is based on and claims priority to Korean Patent Application No. 10-2020-0042408, filed on April 7, 2020, with the Korean Intellectual Property Office, and Korean Patent Application No. 10-2021-0001056, filed on January 5, 2021, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] The disclosed example embodiments relate to the field of artificial intelligence, and more specifically, to neural network devices, methods of operating neural network devices, and application processors. Background Technology

[0003] Neural networks represent computational architectures that simulate the biological networks of an animal's brain. With the recent advancements in neural network technology, extensive research has been conducted on using neural network-based devices in various electronic systems to analyze input data and extract useful information.

[0004] Neural network devices require extensive computation to process complex input data. To enable these devices to analyze input data and extract information in real time, techniques are needed to efficiently process the operations of the neural network. In particular, low-power, high-performance systems (such as smartphones) have limited resources. Therefore, techniques are desired to reduce the computational load required to process complex input data and improve the performance of artificial neural networks. Summary of the Invention

[0005] One or more example embodiments disclosed provide a method and apparatus for performing floating-point calculations in a neural network apparatus including floating-point arithmetic circuitry.

[0006] According to one aspect of the disclosure, a neural network apparatus for performing neural network operations is provided. The neural network apparatus includes: a floating-point arithmetic circuit configured to perform dot product and accumulation operations; and a buffer configured to store first accumulated data generated by the floating-point arithmetic circuit, wherein the floating-point arithmetic circuit is further configured to perform the dot product and accumulation operations by: identifying a maximum value from the exponent value of the first accumulated data and a plurality of exponent addition results obtained by adding the exponents of a plurality of floating-point data pairs respectively; based on the maximum value, performing an alignment shift of the mantissa portion of the first accumulated data and a plurality of mantissa multiplication results obtained by multiplying the mantissas of the plurality of floating-point data pairs respectively; and performing a summation of the plurality of aligned mantissa multiplication results with the aligned mantissa portion of the first accumulated data.

[0007] According to another aspect of the disclosure, a method for operating a neural network device is provided. The method includes: receiving a plurality of floating-point data pairs and first accumulated data; identifying a maximum value from the exponent value of the first accumulated data and a plurality of exponent addition results obtained by adding the exponents of the plurality of floating-point data pairs respectively; based on the maximum value, performing an alignment shift of the mantissa portion of the first accumulated data and a plurality of mantissa multiplication results obtained by multiplying the mantissas of the plurality of floating-point data pairs respectively; adding the plurality of aligned mantissa multiplication results to the aligned mantissa portion of the first accumulated data; and generating second accumulated data based on the result of the addition.

[0008] According to another aspect of the disclosure, an application processor including a neural network device is provided, wherein the neural network device is configured to perform dot product and accumulation operations by: performing an alignment shift of the mantissa portion of the first accumulated data and a plurality of mantissa multiplication results obtained by adding the exponents of a plurality of floating-point data pairs respectively, based on the exponent value of a first accumulated data and a plurality of exponent addition results obtained by adding the exponents of a plurality of floating-point data pairs respectively; and performing a summation of the plurality of aligned mantissa multiplication results with the aligned mantissa portion of the first accumulated data. Attached Figure Description

[0009] The above and / or other aspects will become clearer by describing specific example embodiments with reference to the accompanying drawings, in which:

[0010] Figure 1 This is a schematic block diagram of a neural network system according to an example embodiment;

[0011] Figure 2 An example of a neural network architecture is shown;

[0012] Figures 3A to 3C This is an example diagram showing floating-point data format types. Figure 3D This is a diagram illustrating an example of integer data;

[0013] Figure 4 This is a flowchart of an operation method of a neural network device according to an example embodiment;

[0014] Figure 5 This is a diagram illustrating the operation of a neural network system according to an example embodiment;

[0015] Figure 6 This is a diagram of a neural network device according to an example embodiment;

[0016] Figure 7A This is a diagram of a floating-point arithmetic circuit that performs dot product operations according to an example embodiment;

[0017] Figure 7BIt is a diagram showing the format type of floating-point data pairs input to the floating-point arithmetic circuit;

[0018] Figure 8A and Figure 8B This is a diagram illustrating a bit-extended multiplier that performs operations based on the type of floating-point data format.

[0019] Figures 9 to 12 This is a diagram illustrating the operation of a floating-point arithmetic circuit according to an example embodiment;

[0020] Figure 13 This is a block diagram of computational circuitry included in a neural network device according to an example embodiment; and

[0021] Figure 14 This is a block diagram of an application processor according to an example embodiment. Detailed Implementation

[0022] In the following text, exemplary embodiments will be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is a schematic block diagram of a neural network system 100 according to an example embodiment.

[0024] The neural network system 100 can train the neural network or infer information from the input data by analyzing the input data using the neural network. The neural network system 100 can determine situations or control components of electronic devices equipped with the neural network system 100 based on the inferred information. For example, the neural network system 100 can be applied to smartphones, tablets, smart TVs (TVs), augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robots, medical devices, drones, advanced driver assistance systems (ADAS), image display devices, measuring equipment, etc., that use neural networks to perform speech recognition, image recognition, image classification, etc. The neural network system 100 can be installed on any other type of electronic device. In one embodiment, Figure 1 The neural network system 100 can correspond to an application processor.

[0025] Reference Figure 1The neural network system 100 may include a central processing unit (CPU) 110, a neural network device 120, a memory 130, and a sensor module 140. The neural network system 100 may also include an input / output module, a security module, a power controller, etc., and may also include various processors. In some embodiments, some or all of the components of the neural network system 100 (e.g., CPU 110, neural network device 120, memory 130, and sensor module 140) may be formed on a single semiconductor chip. For example, the neural network system 100 may be implemented as a system-on-a-chip (SoC). The components of the neural network system 100 may communicate with each other via a bus 150. Although... Figure 1 Not shown, but the neural network system 100 may also include an interface (not shown) for communicating with other intellectual property (IP) blocks.

[0026] CPU 110 typically controls the operation of neural network system 100. CPU 110 may include single-core or multi-core processors. CPU 110 may process or execute programs and / or data stored in storage areas (such as memory 130).

[0027] For example, CPU 110 can execute an application and control neural network device 120 to perform neural network-based tasks involved in the execution of the application. The neural network may include at least one of various neural network models, such as convolutional neural networks (CNNs), region-based CNNs (R-CNNs), region candidate networks (RPNs), recurrent neural networks (RNNs), stacked deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks.

[0028] The neural network device 120 can perform neural network operations based on input data. The neural network device 120 can generate information signals based on the results of performing the neural network operations. The neural network device 120 may include a neural network operation accelerator, a coprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. In some embodiments, the neural network device 120 may include at least one cluster of neural processing circuits.

[0029] According to an example embodiment, the neural network device 120 may include a floating-point arithmetic circuit 20 and perform neural network operations based on floating-point calculations. The floating-point arithmetic circuit 20 may perform dot product operations on multiple floating-point data and accumulate the dot product results. The multiple floating-point data may be divided into multiple floating-point data pairs for dot product operations and input to the floating-point arithmetic circuit 20.

[0030] Neural networks can include real numbers (e.g., floating-point data represented as floating-point numbers). For example, floating-point data can include feature maps, kernels (or weight maps), biases, etc. When data is represented using floating-point, the data can be represented with a wide range of values, and approximations of real numbers can be appropriately represented.

[0031] In one example embodiment, the floating-point circuit 20 can support dot product operations on floating-point data with various types of formats. The floating-point circuit 20 can expand at least one of the exponent (also known as the index) bit field and the fraction bit field of the floating-point data to fit a predetermined bit width, so that dot product operations can be performed on multiple pieces of floating-point data with different types of formats together.

[0032] In one example embodiment, the floating-point arithmetic circuit 20 may perform exponent addition (e.g., adding the exponents of the floating-point data pairs separately) and mantissa multiplication (e.g., multiplying the mantissas of the floating-point data pairs separately) on the floating-point data pairs in the dot product operation, thereby generating an exponent addition result and a mantissa multiplication result corresponding to each floating-point data pair.

[0033] In one example embodiment, the floating-point circuit 20 can perform an accumulation operation by adding the accumulated data to the result of the mantissa multiplication in the dot product operation. The accumulated data can be generated by accumulating the dot product result of the neural network device 120 and stored in a buffer (not shown) of the neural network device 120.

[0034] In one example embodiment, before adding the accumulated data to the mantissa multiplication result, the floating-point circuitry 20 may perform an alignment shift to align the floating-point values ​​of the mantissa multiplication result and the accumulated data with each other. The floating-point circuitry 20 may perform the alignment shift of the mantissa portion of the accumulated data and the mantissa multiplication result based on the maximum value identified from the exponent value of the accumulated data and the exponent addition result.

[0035] The shift width of the alignment shifter included in the floating-point circuitry 20 used for dot product operations can be reduced to a degree that does not affect the computational accuracy required by the neural network system 100. Therefore, the size of the alignment shifter and the power consumption of the neural network device 120 can be reduced, while the speed of dot product operations can be increased. When the neural network system 100 requires relatively high computational accuracy, the alignment shifter of the floating-point circuitry 20 can be designed to align the data with a relatively long bit width. When the computational accuracy required by the neural network system 100 is relatively low, the alignment shifter of the floating-point circuitry 20 can be designed to align the data with a relatively short bit width.

[0036] In one example embodiment, the floating-point circuitry 20 can perform dot product and accumulation operations together by adding the aligned mantissa multiplication result to the aligned mantissa portion of the accumulated data. To perform this operation, the floating-point circuitry 20 may include an adder (not shown) that simultaneously receives the aligned mantissa multiplication result and the aligned mantissa portion of the accumulated data.

[0037] In one example embodiment, the adder of the floating-point arithmetic circuit 20 may include a first adder circuit and a second adder circuit that are separate from each other. For example, the first adder circuit may add the higher-order bits of the aligned mantissa multiplication result to the higher-order bits of the aligned mantissa portion of the accumulated data, and the second adder circuit may add the lower-order bits of the aligned mantissa multiplication result to the lower-order bits of the aligned mantissa portion of the accumulated data. The configuration of the first and second adder circuits being separate from each other is suitable for leading one detection of the addition result data output from the adder and can optimize the logic after the adder (e.g., the post-adder). An example embodiment of leading one detection will be described below.

[0038] The neural network device 120 may also include an integer multiplier (not shown) that performs integer multiplication operations, and one of the first adder circuit and the second adder circuit may be shared by the integer multiplier. Therefore, no additional adder is required for integer multiplication operations, and thus the size and power consumption of the neural network device 120 can be reduced. An example embodiment relating to the operation of the floating-point arithmetic circuit 20 in integer calculation mode will be described below.

[0039] In one example embodiment, the floating-point circuit 20 can generate new accumulated data by performing operations such as normalization and rounding on the addition result data output from the adder. The new accumulated data can be stored in a buffer (not shown) of the floating-point circuit 20.

[0040] In one example embodiment, the alignment shifter (not shown), adder (not shown), etc., included in the floating-point circuitry 20 can be relatively small in size. Therefore, the floating-point circuitry 20 can operate relatively quickly, has low power consumption, and reduces the amount of resources used when performing dot product operations.

[0041] The neural network device 120 can receive multiple floating-point data FPDs as input data from the memory 130, and can perform dot product operations on the floating-point data FPDs using the floating-point arithmetic circuit 20. The neural network device 120 can accumulate multiple dot product results using the floating-point arithmetic circuit 20. The accumulated data can be stored in a buffer (not shown) of the neural network device 120.

[0042] The neural network device 120 can perform neural network operations using floating-point data (FPD) and store the operation results in memory 130. The neural network device 120 can output the generated floating-point data FPD as the operation result. In one example embodiment, the format type of the input data to the floating-point arithmetic circuit 20 may differ from the format type of the output data output from the floating-point arithmetic circuit 20. For example, the input data may have a floating-point 16 (FP16) format or a brainfloat 16 (BF16) format, and the output data may have a floating-point 32 (FP32) format. In other words, 16-bit data can be input to the floating-point arithmetic circuit 20, and 32-bit data can be output as the result of dot product and accumulation operations. However, this is only an example embodiment, and the disclosed embodiments are not limited thereto. The floating-point arithmetic circuit 20 can receive data with a first type format according to a standard applied to the neural network system 100 and output data with a second type format after performing dot product and accumulation operations.

[0043] The neural network device 120 can use the floating-point arithmetic circuit 20 to perform dot product and accumulation operations on the floating-point data FPD received from the memory 130, and store the floating-point data FPD corresponding to the operation results in the memory 130. In some embodiments, the neural network device 120 can exchange floating-point data FPD with the memory 130 via the bus 150 without the intervention of the CPU 110. In other words, the neural network device 120 can directly exchange floating-point data FPD with the memory 130.

[0044] The memory 130 may store programs and / or data used in the neural network system 100. The memory 130 may also store operating parameters (e.g., weights, biases, etc.) for the neural network, quantization parameters (e.g., scaling factors, biases, etc.) for the neural network, input data (e.g., input feature maps), and output data (e.g., output feature maps). The operating parameters, quantization parameters, input data, and output data may be included in floating-point data (FPD).

[0045] Memory 130 may include, but is not limited to, dynamic random access memory (DRAM). Memory 130 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), and ferroelectric RAM (FRAM). Volatile memory may include DRAM, static RAM (SRAM), and synchronous DRAM (SDRAM). In one embodiment, memory 130 may include at least one of a hard disk drive (HDD), solid-state drive (SSD), compact flash (CF) card, secure digital card (SD) card, micro SD card, mini SD card, extreme digital (xD) card, and memory stick.

[0046] Sensor module 140 can collect information about the surrounding environment of the electronic device on which neural network system 100 is installed. Sensor module 140 can sense or receive signals (e.g., image signals, sound signals, magnetic signals, biosignals, and touch signals) from outside the electronic device and convert the sensed or received signals into sensing data. To perform this operation, sensor module 140 may include at least one of various sensing devices (such as microphones, imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, biosensors, touch sensors, etc.).

[0047] Sensing data can be provided as input data to the neural network device 120 or stored in the memory 130. Sensing data stored in the memory 130 can be provided to the neural network device 120. In one embodiment, the neural network system 100 may further include a graphics processor (GPU) for processing image data, and the sensing data may be processed by the GPU and then provided to the memory 130 or the neural network device 120.

[0048] For example, sensor module 140 may include an image sensor and may capture images of the external environment of the electronic device and generate image data. Image data output from sensor module 140 or image data processed by GPU may include floating-point data and may be provided directly to floating-point circuitry 20 or stored in memory 130 before being provided to floating-point circuitry 20. In some embodiments, neural network system 100 may also include a preprocessor or a postprocessor, wherein the preprocessor processes data to be input to neural network device 120 and the postprocessor processes data output from neural network device 120.

[0049] In some embodiments, Figure 1The neural network system 100 may not include the sensor module 140, and may receive input data from a separately configured sensor module (not shown) or the like via an interface.

[0050] Figure 2 An example of a neural network architecture is shown. (See reference...) Figure 2 A neural network (NN) may include multiple layers (e.g., layers L1, L2, ..., Ln). A neural network (NN) with such a multi-layered architecture may be called a deep neural network (DNN) or a deep learning architecture. Each of the layers L1 to Ln may include a linear layer or a non-linear layer. In one embodiment, at least one linear layer may be combined with at least one non-linear layer to form a single layer. For example, a linear layer may include convolutional layers and fully connected layers, and a non-linear layer may include pooling layers and activation layers.

[0051] For example, the first layer L1 can correspond to a convolutional layer, the second layer L2 can correspond to a pooling layer, and the nth layer Ln can correspond to a fully connected layer as the output layer. The neural network NN may also include activation layers, and may also include other layers that perform other types of operations.

[0052] Each of the first layer L1 to the Nth layer Ln can receive an image frame or feature map generated in the previous layer as an input feature map, and can generate an output feature map or recognition signal REC by performing operations on the input feature map. Here, the feature map represents data representing various features of the input data. For example, the first feature map to the nth feature map FM1, FM2, FM3, ..., FMn can have a two-dimensional matrix form or a three-dimensional matrix (or tensor form) form including multiple feature values. The first feature map FM1 to the nth feature map FMn can have a width W (or column), a height H (or row), and a depth D, where the width W (or column), height H (or row), and depth D can correspond to the x-axis, y-axis, and z-axis in the coordinate system, respectively. Here, the depth D can be referred to as the number of channels.

[0053] The first layer L1 generates the second feature map FM2 by performing convolution on the first feature map FM1 and the weight map WM. The weight map WM can be in two-dimensional or three-dimensional matrix form, including multiple weights. The weight map WM can be referred to as the kernel. The weight map WM can filter the first feature map FM1 and can be referred to as the filter or kernel. The depth (i.e., the number of channels) of the weight map WM can be the same as the depth (i.e., the number of channels) of the first feature map FM1. Convolution can be performed on the same channels in both the weight map WM and the first feature map FM1. The weight map WM can be shifted on the first feature map FM1 by using a sliding window to traverse the first feature map FM1. During the shift, each weight included in the weight map WM can be multiplied and added with all feature values ​​in the region where the weight map WM overlaps with the first feature map FM1. One channel of the second feature map FM2 can be generated by performing convolution on the first feature map FM1 and the weight map WM. Although in Figure 2 Only one weight map WM is shown, but multiple weight maps WM can be convolved with the first feature map FM1 to generate multiple channels of the second feature map FM2. In other words, the number of channels in the second feature map FM2 can correspond to the number of weight maps.

[0054] The second layer L2 can generate a third feature map FM3 by modifying the spatial size of the second feature map FM2 through pooling. Pooling can be referred to as sampling or downsampling. The two-dimensional pooling window PW can be shifted on the second feature map FM2 in units of the size of the pooling window PW, and the maximum value among the eigenvalues ​​(or the average of the eigenvalues) in the region where the pooling window PW overlaps with the second feature map FM2 can be selected. In this way, the third feature map FM3 can be generated by modifying the spatial size of the second feature map FM2. The number of channels in the third feature map FM3 can be the same as the number of channels in the second feature map FM2.

[0055] The nth layer Ln can combine features from the nth feature map FMn and classify the category CL of the input data. The nth layer Ln can also generate a recognition signal REC corresponding to the category CL. For example, when the input data corresponds to image data and the neural network NN performs image recognition, the nth layer Ln can extract the category corresponding to the object in the image represented by the image data based on the nth feature map FMn provided by the previous layer, identify the object, and generate a recognition signal REC corresponding to the object.

[0056] As referenced above Figure 2 The neural network (NN) can have a complex architecture, and the neural network device performing the neural network operations can perform a large number of operations ranging from hundreds of millions to tens of billions. According to an example embodiment, Figure 1The neural network device 120 simplifies the configuration of alignment shifters, adders, etc. to the point that the accuracy of neural network operations is not affected, and performs efficient network operations using a simplified configuration, thereby reducing power consumption and design area, and increasing its operating speed.

[0057] Figures 3A to 3C This is an example diagram showing floating-point data format types. Figure 3D This is a diagram illustrating an example of integer data.

[0058] Reference Figure 3A Floating-point data can be represented by signs and 1.a×2 b The format is represented as , where "b" is the exponent and "a" is the mantissa. According to IEEE 754-2008, the standard for floating-point arithmetic, in the Type I format, 32-bit floating-point data includes one bit representing the sign, eight bits representing the exponent, and 23 bits representing the mantissa. For example... Figure 3A As shown, the most significant bit (MSB) represents the sign Sn, the 8 bits following the MSB represent the exponent En, and the remaining 23 bits represent the mantissa Fn (or significant digits). In the following text, floating-point data with the first type format may be referred to as floating-point 32 (FP32).

[0059] Reference Figure 3B In the second type format, according to the IEEE 754-2008 standard, 16-bit floating-point data includes one bit representing the sign Sn, five bits representing the exponent En, and ten bits representing the mantissa Fn. In other words, the five bits after the MSB represent the exponent, and the remaining ten bits represent the mantissa (or significant digits). Floating-point data with the second type format will be referred to as Floating-Point 16 (FP16) in the following text.

[0060] Reference Figure 3C In the third type format, 16-bit floating-point data includes one bit representing the sign Sn, eight bits representing the exponent En, and seven bits representing the mantissa Fn. In other words, the eight bits after the MSB represent the exponent, and the remaining seven bits represent the mantissa (or significant digits). Floating-point data in the third type format will be referred to as Brain Floating-16 (BF16) in the following text.

[0061] Reference Figure 3D Data in integer format can be represented in various types depending on the presence or absence of symbols and the data size (e.g., the number of bits). For example, such as Figure 3D As shown, integers typically used for calculations can be represented as 8 bits of data including the sign (e.g., two's complement). In the case of signed numbers, the MSB represents the sign, the following 7 bits represent the integer INTn, and the signed number has... The size of the number. In the case of unsigned numbers, unsigned numbers are positive and have... Size.

[0062] According to an example embodiment, the neural network device can receive floating-point data corresponding to at least one of a first to a third type format, and perform dot product and accumulation operations. For example, the neural network device can receive floating-point data corresponding to at least one of an FP16 type format, a BF16 type format, and an FP32 type format, and perform dot product and accumulation operations on the floating-point data to output accumulated data in an FP32 type format. However, this is only an example embodiment, and embodiments are not limited thereto. The neural network device can receive floating-point data in various other type formats and perform dot product and accumulation operations to output accumulated data in a pre-defined format required by the neural network system.

[0063] Figure 4 This is a flowchart of an operation method of a neural network device according to an example embodiment. Figure 4 The operation method can be provided by Figure 1 The neural network device 120 in the description performs operations for neural network operations. For understanding, reference will also be made to... Figure 1 .

[0064] Reference Figure 4 During operation S100, the neural network device 120 may receive multiple floating-point data pairs from the memory 130. These floating-point data pairs may include input feature values, weights, function coefficients, etc., used in the neural network operation. When the neural network device 120 processes a quantized neural network, the floating-point data pairs may include quantization parameters. For example, quantization parameters may include scaling factors (or inverse scaling factors), biases, etc.

[0065] In operation S110, the neural network device 120 can expand the bit fields of the floating-point data pairs. The neural network device 120 can expand one of the exponent bit field and the mantissa bit field of the data to a specified bit width to perform a dot product operation on the floating-point data pairs together. This will refer to... Figure 8A and Figure 8B Detailed description.

[0066] In operation S120, the neural network device 120 can perform dot product and accumulation operations on each floating-point data pair. The neural network device 120 can perform exponent addition and mantissa multiplication operations on the floating-point data pairs. The neural network device 120 can identify the maximum value from the exponent values ​​of the exponent addition results and the accumulated data corresponding to the floating-point data pairs, respectively. Based on the maximum value, the neural network device 120 can perform alignment shifts on the mantissa multiplication results and the mantissa portions of the accumulated data corresponding to the floating-point data pairs, respectively, and add the aligned mantissa multiplication results to the aligned mantissa portions of the accumulated data. The neural network device 120 can use the addition result data to generate new accumulated data. In addition to the dot product operation in operation S120, the neural network device 120 can also perform neural network operations including convolution, pooling, etc.

[0067] In operation S130, the neural network device 120 may store the final data generated in operation S120 in memory 130.

[0068] Figure 5 This is a diagram illustrating the operation of a neural network system 100 according to an example embodiment.

[0069] Reference Figure 5 In operation S1, the neural network device 120 may receive floating-point input data FPID from memory 130. In some embodiments, the floating-point input data FPID may be sent from memory 130 to the neural network device 120 via bus 150 without the intervention of CPU 110. For example, the neural network device 120 may include a direct memory access (DMA) controller that can access memory 130 and read the floating-point input data FPID. The neural network device 120 may use floating-point arithmetic circuitry 20 to perform neural network operations on the floating-point input data FPID, including dot product and accumulation operations. In operation S2, the neural network device 120 may generate floating-point output data FPOD as a result of the neural network operation and send the floating-point output data FPOD to memory 130.

[0070] As described above, according to the example embodiment, the neural network device 120 can perform dot product and accumulation operations quickly and efficiently with low power consumption.

[0071] Figure 6 This is a diagram of a neural network device 120 according to an example embodiment. For ease of description, a memory 130 is also shown.

[0072] Reference Figure 6The neural network device 120 may include a DMA controller 10 and a neural network processor 30. In one example embodiment, the neural network processor 30 may include a floating-point arithmetic circuit 20 according to an example embodiment, which is capable of performing dot product and accumulation operations.

[0073] The DMA controller 10 can communicate directly with the memory 130. The DMA controller 10 can receive input data (e.g., floating-point input data FPID) from the memory 130 and send output data (e.g., floating-point output data FPOD) generated as a result of neural network operations to the memory 130 without intervention from other processors (such as CPU and GPU). For example, the floating-point input data FPID may include an input feature map IFM, operation parameters PM, or a weight map (or kernel parameters). The floating-point input data FPID may also include quantization parameters.

[0074] The floating-point arithmetic circuit 20 can perform dot product and accumulation operations on the floating-point input data FPID. The floating-point arithmetic circuit 20 can perform dot product operations on each floating-point data pair, where the floating-point input data FPID is divided into a pair of floating-point data for dot product operations. The floating-point arithmetic circuit 20 can perform dot product operations on each floating-point data pair, including exponent addition and mantissa multiplication.

[0075] In one example embodiment, in the dot product and accumulation operations, the floating-point circuit 20 can perform alignment shifts on the mantissa portion of the accumulated data and the mantissa multiplication results corresponding to the floating-point data pairs, based on the exponent value of the accumulated data and the exponent addition results respectively, and then add the mantissa multiplication results to the mantissa portion of the accumulated data. In one example embodiment, the floating-point circuit 20 can identify the maximum value from the exponent addition results and the exponent value of the accumulated data; perform alignment shifts on each mantissa multiplication result bit by bit along the direction corresponding to the difference between each exponent addition result and the maximum value, where the bit bit corresponds to the difference between each exponent addition result and the maximum value; and perform alignment shifts on the mantissa portion of the accumulated data bit by bit along the direction corresponding to the difference between the exponent value of the accumulated data and the maximum value, where the bit bit corresponds to the difference between the exponent value of the accumulated data and the maximum value. In one example, when the difference between the exponent value of the exponent addition result or the maximum value of the accumulated data is zero, the alignment shift of the mantissa portion of the corresponding mantissa multiplication result or accumulated data is not performed. When the difference between the exponent value of the exponent addition result or the maximum value of the accumulated data is not zero, the mantissa portion of the corresponding mantissa multiplication result or accumulated data is shifted towards the least significant bit. The floating-point circuit 20 can simultaneously perform dot product and accumulation operations by adding the aligned mantissa multiplication result to the aligned mantissa portion of the accumulated data, thereby generating new accumulated data.

[0076] Before or after the operation of the floating-point arithmetic circuit 20, in addition to floating-point calculations, the neural network processor 30 may also perform neural network operations (such as convolution or pooling) and generate floating-point output data FPOD. For example, the floating-point output data FPOD may include an output feature map OFM.

[0077] The neural network processor 30 may include an array of processing elements, which may include a plurality of processing elements. Although not shown, the neural network processor 30 may include a controller and a buffer for storing neural network parameters, such as biases, weights, input features, and output features. The processing elements may form a floating-point circuit 20, which the neural network processor 30 may use to perform floating-point-based neural network operations according to an example embodiment.

[0078] The configuration and operation of the floating-point circuit 20 will be described below.

[0079] Figure 7A This is a diagram of a floating-point arithmetic circuit 20 that performs dot product operations according to an example embodiment. Figure 7B This is a diagram illustrating the format types of floating-point data pairs P1, P2, and P3 input to the floating-point arithmetic circuit 20.

[0080] Reference Figure 7A The floating-point arithmetic circuit 20 may include a multiplier 21 (or a floating-point multiplier), a sign and exponent calculator 22, an alignment shifter 23, an adder 24, a post-adder 25, and a buffer 26. The logic following the adder 24 is collectively referred to as the post-adder 25. The post-adder 25 may include a leading 1 detection circuit 25_1, a normalization and rounding circuit 25_2, and an accumulated data generation circuit 25_3.

[0081] The floating-point circuit 20 can receive the first accumulated data ACC1 from the buffer 26. The sign ACC1_S and the exponent ACC1_E of the first accumulated data ACC1 can be input to the sign and exponent calculator 22, and the mantissa ACC1_F of the first accumulated data ACC1 can be input to the alignment shifter 23.

[0082] The floating-point arithmetic circuit 20 can perform dot product operations on the first to fourth floating-point data pairs. The first floating-point data pair may include first floating-point data IN_A1 and second floating-point data IN_B1, the second floating-point data pair may include third floating-point data IN_A2 and fourth floating-point data IN_B2, the third floating-point data pair may include fifth floating-point data IN_A3 and sixth floating-point data IN_B3, and the fourth floating-point data pair may include seventh floating-point data IN_A4 and eighth floating-point data IN_B4. The corresponding symbols Sa1, Sb1, Sa2, Sb2, Sa3, Sb3, Sa4, and Sb4 of the first to eighth floating-point data IN_A1, IN_B1, IN_A2, IN_B2, IN_A3, IN_B3, IN_A4, and IN_B4, as well as the corresponding exponents Ea1, Eb1, Ea2, Eb2, Ea3, Eb3, Ea4, and Eb4, can be input to the sign and exponent calculator 22. The corresponding mantissas Fa1, Fb1, Fa2, Fb2, Fa3, Fb3, Fa4 and Fb4 of the first to eighth floating-point data IN_A1, IN_B1, IN_A2, IN_B2, IN_A3, IN_B3, IN_A4 and IN_B4 can be input into multiplier 21.

[0083] Multiplier 21 generates first mantissa multiplication results to fourth mantissa multiplication results F_mul1, F_mul2, F_mul3, and F_mul4 by performing multiplication operations on mantissas Fa1, Fb1, Fa2, Fb2, Fa3, Fb3, Fa4, and Fb4, and provides the first mantissa multiplication results to the fourth mantissa multiplication results F_mul1, F_mul2, F_mul3, and F_mul4 to alignment shifter 23. In one example embodiment, multiplier 21 may include an 11-bit multiplier, a 24-bit multiplier with extended bit width, or a 32-bit multiplier.

[0084] The symbol and exponent calculator 22 can generate exponent information EI and symbol information SI by determining the symbol based on the symbols Sa1, Sb1, Sa2, Sb2, Sa3, Sb3, Sa4 and Sb4 and the exponents Ea1, Eb1, Ea2, Eb2, Ea3, Eb3, Ea4 and Eb4 and performing exponent addition operations.

[0085] In one example embodiment, the sign and exponent calculator 22 can identify the maximum value from the exponent addition result and the value (or exponent value) of the exponent ACC1_E of the first accumulated data ACC1, and generate first shift information SH1 provided to the alignment shifter 23 based on the maximum value. The first shift information SH1 may include information about the shift direction and shift amount, and the first shift information SH1 corresponds to each of the first mantissa multiplication results F_mul1, F_mul2, F_mul3, and F_mul4 from the multiplier 21 and the mantissa ACC1_F of the first accumulated data ACC1. In detail, the sign and exponent calculator 22 can generate first shift information SH1 to perform a bit-by-bit alignment shift of each of the first mantissa multiplication results to the fourth mantissa multiplication results F_mul1, F_mul2, F_mul3 and F_mul4, and the mantissa ACC1_F of the first cumulative data ACC1, along the direction corresponding to the difference between each of the exponents ACC1_E of the first cumulative data ACC1 and the maximum value.

[0086] In one example embodiment, the alignment shifter 23 may perform alignment shifts of the first mantissa multiplication result to the fourth mantissa multiplication result F_mul1, F_mul2, F_mul3 and F_mul4 and the mantissa ACC1_F of the first cumulative data ACC1 based on the first shift information SH1, and provide the first aligned mantissa multiplication result to the fourth aligned mantissa multiplication result F_align1, F_align2, F_align3 and F_align4 and the aligned mantissa ACC1_F_align to the adder 24.

[0087] In one example embodiment, adder 24 may simultaneously receive the first aligned mantissa multiplication results to the fourth aligned mantissa multiplication results F_align1, F_align2, F_align3, and F_align4, as well as the aligned mantissa ACC1_F_align, to perform dot product and accumulation operations simultaneously. Adder 24 may include a first adder circuit 24_1 and a second adder circuit 24_2. The first adder circuit 24_1 may generate a first addition result data F_add1 by adding the higher bits of the first aligned mantissa multiplication results to the fourth aligned mantissa multiplication results F_align1, F_align2, F_align3, and F_align4, as well as the aligned mantissa ACC1_F_align, wherein the higher bits include the corresponding MSBs of the first aligned mantissa multiplication results to the fourth aligned mantissa multiplication results F_align1, F_align2, F_align3, and F_align4, as well as the aligned mantissa ACC1_F_align. The second adder circuit 24_2 generates the second addition result data F_add2 by adding the first aligned mantissa multiplication result to the fourth aligned mantissa multiplication results F_align1, F_align2, F_align3 and F_align4 and the lower bits of the aligned mantissa ACC1_F_align. The lower bits include the corresponding least significant bit (LSB) of the first aligned mantissa multiplication result to the fourth aligned mantissa multiplication results F_align1, F_align2, F_align3 and F_align4 and the aligned mantissa ACC1_F_align.

[0088] According to the example embodiment, one of the first adder circuit 24_1 and the second adder circuit 24_2 of adder 24 can be shared by an integer multiplier that performs integer multiplication operations, and is therefore used for floating-point calculations, integer calculations, etc. Therefore, the size of the neural network device including the floating-point operation circuit 20 can be effectively reduced.

[0089] In one example embodiment, the post-adder 25 may receive first addition result data F_add1 and second addition result data F_add2, and generate second cumulative data ACC2.

[0090] The leading 1 detection circuit 25_1 can perform a first leading 1 detection on a first number of higher bits of the third addition result data and a second leading 1 detection on a second number of lower bits of the third addition result data in parallel. The third addition result data is generated from the sum of the first addition result data F_add1 and the second addition result data F_add2. The higher bits include the MSB of the third addition result data, and the lower bits include the LSB of the third addition result data. In other words, the leading 1 detection circuit 25_1 can detect "1" while shifting the higher bits of the third addition result data along the direction from the MSB towards the lower bits during the first leading 1 detection. In this case, the maximum number of bits that can be shifted can be the same as the first number. The leading 1 detection circuit 25_1 can also detect "1" while shifting the lower bits of the third addition result data along the direction from the LSB towards the higher bits during the second leading 1 detection. In this case, the maximum number of bits that can be shifted can be the same as the second number. In one example embodiment, the first number may be different from the second number. For example, the first number may be less than the second number. The leader 1 detection circuit 25_1 can perform a shift of the third addition result data based on the results of the first leader 1 detection and the second leader 1 detection. Specifically, the leader 1 detection circuit 25_1 can perform a shift of the third addition result data bit by bit along the shift direction corresponding to the result of the first leader 1 detection or the second leader 1 detection that has detected a "1", where the bit number corresponds to the result of the first leader 1 detection or the second leader 1 detection.

[0091] The normalization and rounding circuit 25_2 can normalize the shifted third addition result data according to the position of "1" and round the shifted third addition result data according to the number of bits in the format of the data output from the floating-point arithmetic circuit 20.

[0092] The cumulative data generation circuit 25_3 can reflect the exponent value and sign in the normalized and rounded third addition result data based on the exponent information EI and the sign information SI to generate the second cumulative data ACC2, and store the second cumulative data ACC2 in the buffer 26.

[0093] Reference Figure 7BIn the first case, "Case 1", the floating-point arithmetic circuit 20 can receive first floating-point data IN_A and second floating-point data IN_B, which have the same type format for dot product operations. The corresponding sign bit fields Sa and Sb of the first floating-point data IN_A and second floating-point data IN_B can have the same width, the corresponding exponent bit fields Ea and Eb of the first floating-point data IN_A and second floating-point data IN_B can have the same width, and the corresponding mantissa bit fields Fa and Fb of the first floating-point data IN_A and second floating-point data IN_B can have the same width. For example, the first floating-point data IN_A and second floating-point data IN_B can have an FP16 type format.

[0094] In the second scenario, "Scenario 2", the floating-point arithmetic circuit 20 can receive first floating-point data IN_A and second floating-point data IN_B, which have different type formats for dot product operations. The corresponding sign bit fields Sa and Sb of the first floating-point data IN_A and second floating-point data IN_B can have the same width, the corresponding exponent bit fields Ea and Eb of the first floating-point data IN_A and second floating-point data IN_B can have different widths, and the corresponding mantissa bit fields Fa and Fb of the first floating-point data IN_A and second floating-point data IN_B can have different widths. For example, the first floating-point data IN_A and second floating-point data IN_B can have FP16 type format and BF16 type format, respectively.

[0095] In the third case, "Case 3", the floating-point arithmetic circuit 20 can receive first floating-point data IN_A and second floating-point data IN_B, which have the same type format for dot product operations. The corresponding sign bit fields Sa and Sb of the first floating-point data IN_A and second floating-point data IN_B can have the same width, the corresponding exponent bit fields Ea and Eb of the first floating-point data IN_A and second floating-point data IN_B can have the same width, and the corresponding mantissa bit fields Fa and Fb of the first floating-point data IN_A and second floating-point data IN_B can have the same width. For example, the first floating-point data IN_A and second floating-point data IN_B can have a BF16 type format.

[0096] Reference Figure 7A and Figure 7B The description of the floating-point arithmetic circuit 20 given is merely an example, and the embodiments are not limited thereto. The floating-point arithmetic circuit 20 can be implemented in various ways. The floating-point arithmetic circuit 20 can receive various quantities of floating-point data with various types of formats and perform dot product operations on various quantities of floating-point data with various types of formats.

[0097] Figure 8A and Figure 8B This is a diagram illustrating a bit-extended multiplier 21 that performs operations based on the type of floating-point data format.

[0098] Reference Figure 8A The multiplier 21 may include a bit extension circuit 21_1. In one example embodiment, the bit extension circuit 21_1 may extend at least one of the exponent bit field and the mantissa bit field included in the floating-point data, depending on the type of the floating-point data format. For example, the bit extension circuit 21_1 may receive floating-point data IN of a first type. type1 Output floating-point data IN as is type1 The exponent E type1 And IN floating-point data type1 The last digit F type1 The bit field is extended to the agreed first width. The agreed first width can be used with floating-point data of type 2 IN. type2 The last digit F type2 bit width, Figure 7A The multiplier 21 is related to at least one of the following: the bit width, the number of bits in the output data of the floating-point arithmetic circuit, and the type of the output data format.

[0099] Reference Figure 8B The bit extension circuit 21_1 can receive second type floating-point data IN. type2 IN floating-point data type2 The exponent E type2 The bit field is extended to the agreed second bit width, and the floating-point data IN is output as is. type2 The last digit F type2 The agreed second bit width can be used with floating-point data of type 1. type1 The exponent E type1 It is related to the bit width.

[0100] For example, when the first type is BF16 and the second type is FP16, according to the mantissa F of FP16 type2 The bit width, the first bit width can be conventionally defined as 11 bits (similar to the mantissa F of FP16). type2 The bit width corresponds to the sum of 10 bits and 1 bit. In other words, floating-point data IN with BF16 format... type1 The last digit F type1 It can be extended to correspond to floating-point data IN with FP16 format. type2 The last digit F type2 The bit width is then used to perform multiplication operations.

[0101] For example, according to the exponent E of BF16 type1The first bit width is 8 bits, and the second bit width can be predefined as 8 bits. In other words, floating-point data IN in FP16 format... type2 The exponent E type2 It can be expanded to correspond to floating-point data IN with BF16 format. type1 The exponent E type1 The bit width is then used to perform addition operations.

[0102] Figures 9 to 12 This is a diagram illustrating the operation of floating-point circuits 20a, 20b, 20c, and 20d according to an example embodiment.

[0103] Reference Figure 9 The floating-point operation circuit 20a may include a first multiplexer to a sixth multiplexer M11, M21, M31, M41, M51 and M61, an XOR circuit XOR, a maximum tree circuit MT, a first adder element AE11 and a multiplier element ME.

[0104] When the floating-point arithmetic circuit 20a operates in integer calculation mode, the first multiplexer M11 to the sixth multiplexer M61 can output "0" in response to the signal IS_FLOAT. Because the first multiplexer M11 to the sixth multiplexer M61 output "0", the elements in the floating-point arithmetic circuit 20a used for floating-point calculation can be deactivated. The following description is based on the fact that the floating-point arithmetic circuit 20a operates in floating-point calculation mode.

[0105] The floating-point arithmetic circuit 20a can receive first floating-point data to eighth floating-point data IN_A1, IN_B1, IN_A2, IN_B2, IN_A3, IN_B3, IN_A4, and IN_B4, as well as first accumulated data ACC1. The first multiplexer M11 can sequentially output the corresponding symbols Sa of the first floating-point data IN_A1, the third floating-point data IN_A2, the fifth floating-point data IN_A3, and the seventh floating-point data IN_A4 to the XOR circuit. The second multiplexer M21 can sequentially output the corresponding symbols Sb of the second floating-point data IN_B1, the fourth floating-point data IN_B2, the sixth floating-point data IN_B3, and the eighth floating-point data IN_B4 to the XOR circuit. The XOR circuit XOR performs an XOR operation on the symbols Sa and Sb, and generates symbol information SI.

[0106] The third multiplexer M31 can sequentially output the corresponding first exponents Ea of the first floating-point data IN_A1, the third floating-point data IN_A2, the fifth floating-point data IN_A3, and the seventh floating-point data IN_A4 to each of the maximum tree circuit MT and the first adder element AE11. The fourth multiplexer M41 can sequentially output the corresponding second exponents Eb of the second floating-point data IN_B1, the fourth floating-point data IN_B2, the sixth floating-point data IN_B3, and the eighth floating-point data IN_B4 to each of the maximum tree circuit MT and the first adder element AE11.

[0107] The maximum tree circuit MT can receive the third exponent ACC1_E of the first accumulated data ACC1, as well as the first exponent Ea and the second exponent Eb, and identify and output the maximum value Max_E used for alignment shift. In an example embodiment, the maximum tree circuit MT can perform addition operations on each corresponding pair of first exponent Ea and second exponent Eb, compare the results of multiple exponent additions with the third exponent ACC1_E, and output the largest exponent value as the maximum value Max_E.

[0108] The first adder element AE11 can output first-exponent addition results to fourth-exponent addition results E_add1, E_add2, E_add3, and E_add4 by performing addition operations on each corresponding pair of first-exponent Ea and second-exponent Eb (for example, the first-exponent Ea and second-exponent Eb of a pair of floating-point data IN_A1 and IN_B1, the first-exponent Ea and second-exponent Eb of a pair of floating-point data IN_A2 and IN_B2, the first-exponent Ea and second-exponent Eb of a pair of floating-point data IN_A3 and IN_B3, and the first-exponent Ea and second-exponent Eb of a pair of floating-point data IN_A4 and IN_B4).

[0109] The fifth multiplexer M51 can sequentially output the corresponding first mantissa Fa of the first floating-point data IN_A1, the third floating-point data IN_A2, the fifth floating-point data IN_A3, and the seventh floating-point data IN_A4 to the multiplier element ME. The sixth multiplexer M61 can sequentially output the corresponding second mantissa Fb of the second floating-point data IN_B1, the fourth floating-point data IN_B2, the sixth floating-point data IN_B3, and the eighth floating-point data IN_B4 to the multiplier element ME.

[0110] The multiplier element ME can output the first mantissa multiplication results to the fourth mantissa multiplication results F_mul1, F_mul2, F_mul3, and F_mul4 by performing multiplication operations on each pair of corresponding first mantissa Fa and second mantissa Fb (for example, the first mantissa Fa and second mantissa Fb of a pair of floating-point data IN_A1 and IN_B1, the first mantissa Fa and second mantissa Fb of a pair of floating-point data IN_A2 and IN_B2, the first mantissa Fa and second mantissa Fb of a pair of floating-point data IN_A3 and IN_B3, and the first mantissa Fa and second mantissa Fb of a pair of floating-point data IN_A4 and IN_B4).

[0111] Further reference Figure 10 The floating-point operation circuit 20b may include first inverters to fourth inverters IVT1, IVT2, IVT3 and IVT4, seventh multiplexers to tenth multiplexers M12, M22, M32 and M42, first shift circuits to fifth shift circuits SHT11, SHT21, SHT31, SHT41 and SHT51, and first subtractor elements to fifth subtractor elements SBE1, SBE2, SBE3, SBE4 and SBE5.

[0112] The seventh multiplexer M12 receives the first mantissa multiplication result F_mul1 and the first inverted mantissa multiplication result output from the first inverter IVT1. The eighth multiplexer M22 receives the second mantissa multiplication result F_mul2 and the second inverted mantissa multiplication result output from the second inverter IVT2. The ninth multiplexer M32 receives the third mantissa multiplication result F_mul3 and the third inverted mantissa multiplication result output from the third inverter IVT3. The tenth multiplexer M42 receives the fourth mantissa multiplication result F_mul4 and the fourth inverted mantissa multiplication result output from the fourth inverter IVT4.

[0113] Each of the seventh to tenth multiplexers M12 can output a non-inverted mantissa multiplication result or an inverted mantissa multiplication result in response to the corresponding symbol in the symbol information SI. For example, the seventh multiplexer M12 can output the first mantissa multiplication result F_mul1 when the first mantissa multiplication result F_mul1 is positive, and output the first inverted mantissa multiplication result when the first mantissa multiplication result F_mul1 is negative.

[0114] The first subtractor element SBE1 performs a subtraction operation on the maximum value Max_E and the first exponent addition result E_add1, and provides the subtraction result to the first shift circuit SHT11. The first shift circuit SHT11 outputs the first aligned mantissa multiplication result F_align1 by shifting the output of the seventh multiplexer M12 by bits along the direction corresponding to the difference between the maximum value Max_E and the first exponent addition result E_add1.

[0115] The second subtractor element SBE2 performs a subtraction operation on the maximum value Max_E and the result of the second exponent addition E_add2, and provides the subtraction result to the second shift circuit SHT21. The second shift circuit SHT21 outputs the second aligned mantissa multiplication result F_align2 by shifting the output of the eighth multiplexer M22 by bits along the direction corresponding to the difference between the maximum value Max_E and the result of the second exponent addition E_add2.

[0116] The third subtractor element SBE3 performs a subtraction operation on the maximum value Max_E and the third-exponent addition result E_add3, and provides the subtraction result to the third shift circuit SHT31. The third shift circuit SHT31 outputs the third-aligned mantissa multiplication result F_align3 by shifting the output of the ninth multiplexer M32 by bits along the direction corresponding to the difference between the maximum value Max_E and the third-exponent addition result E_add3.

[0117] The fourth subtractor element SBE4 performs a subtraction operation on the maximum value Max_E and the fourth exponent addition result E_add4, and provides the subtraction result to the fourth shift circuit SHT41. The fourth shift circuit SHT41 outputs the fourth aligned mantissa multiplication result F_align4 by shifting the output of the tenth multiplexer M42 by bits along the direction corresponding to the difference between the maximum value Max_E and the fourth exponent addition result E_add4.

[0118] The fifth subtractor element SBE5 performs a subtraction operation on the maximum value Max_E and the third exponent ACC1_E of the first accumulated data ACC1, and provides the subtraction result to the fifth shift circuit SHT51. The fifth shift circuit SHT51 outputs an aligned mantissa ACC1_F_align by shifting the mantissa ACC1_F of the first accumulated data ACC1 by bits along the direction corresponding to the difference between the maximum value Max_E and the third exponent ACC1_E.

[0119] In one example embodiment, the maximum shift amount of the first shift circuit SHT11 to the fifth shift circuit SHT51 can be limited so as not to affect the operational accuracy required by the neural network device, while reducing the size and power consumption of the shift circuits.

[0120] Further reference Figure 11 The floating-point arithmetic circuit 20c may include first sign extension circuits to eighth sign extension circuits SE1, SE2, SE3, SE4, SE5, SE6, SE7 and SE8, first zero extension circuits to eighth zero extension circuits ZE1, ZE2, ZE3, ZE4, ZE5, ZE6, ZE7 and ZE8, eleventh multiplexers to eighteenth multiplexers M13, M23, M33, M43, M53, M63, M73 and M83, and second adder elements to ninth adder elements AE12, AE22, AE32, AE42, AE52, AE62, AE72 and AE82. In the following text, for ease of description, the first aligned mantissa multiplication result F_align1 to the fourth aligned mantissa multiplication result F_align4 are referred to as the first mantissa multiplication result F_align1 to the fourth mantissa multiplication result F_align4.

[0121] The eleventh multiplexer M13 can receive the higher-order bits of the first extension of the first mantissa multiplication result F_align1 from the first sign extension circuit SE1, and the higher-order bits of the second extension of the first mantissa multiplication result F_align1 from the first zero extension circuit ZE1. The first sign extension circuit SE1 generates the higher-order bits of the first mantissa multiplication result F_align1 by copying the sign to the extension bits appended to the higher-order bits of the first mantissa multiplication result F_align1, such that the extension bits are filled with 1s. The first zero extension circuit ZE1 generates the higher-order bits of the second extension of the first mantissa multiplication result F_align1 by filling the extension bits appended to the higher-order bits of the first mantissa multiplication result F_align1 with 0s.

[0122] In this manner, each of the twelfth to fourteenth multiplexers M23 to M43 can receive the higher bit of the first extension and the higher bit of the second extension of a corresponding one of the second mantissa multiplication results F_align2 to the fourth mantissa multiplication results F_align4, wherein the higher bit of the first extension includes an extension bit filled with 1 by copying the sign to the extension bit using a corresponding one of the second sign extension circuits SE2 to the fourth sign extension circuits SE4, and the higher bit of the second extension includes an extension bit filled with 0 by a corresponding one of the second zero extension circuits ZE2 to the fourth zero extension circuits ZE4.

[0123] The fifteenth multiplexer M53 receives the lower-order bits of the first extension of the first mantissa multiplication result F_align1 from the fifth sign extension circuit SE5, and the lower-order bits of the second extension of the first mantissa multiplication result F_align1 from the fifth zero extension circuit ZE5. The fifth sign extension circuit SE5 generates the lower-order bits of the first mantissa multiplication result F_align1 by copying the sign to the extension bits appended to the lower-order bits of the first mantissa multiplication result F_align1, filling the extension bits with 1s. The fifth zero extension circuit ZE5 generates the lower-order bits of the second extension of the first mantissa multiplication result F_align1 by filling the extension bits appended to the lower-order bits of the first mantissa multiplication result F_align1 with 0s.

[0124] In this manner, each of the sixteenth multiplexer M63 to the eighteenth multiplexer M83 can receive the lower bit of the first extension and the lower bit of the second extension of a corresponding one of the second mantissa multiplication results F_align2 to the fourth mantissa multiplication results F_align4, wherein the lower bit of the first extension includes an extension bit filled with 1 by copying the sign to the extension bit using a corresponding one of the sixth sign extension circuits SE6 to the eighth sign extension circuits SE8, and the lower bit of the second extension includes an extension bit filled with 0 by a corresponding one of the sixth zero extension circuits ZE6 to the eighth zero extension circuits ZE8.

[0125] In one example embodiment, in floating-point mode (e.g., IS_FLOAT = 1), in response to the corresponding symbol in the symbol information SI, each of the eleventh multiplexer M13 to the fourteenth multiplexer M43 can output the higher bit of the first extension corresponding to a received symbol from the first symbol extension circuit SE1 to the fourth symbol extension circuit SE4. In floating-point mode (e.g., IS_FLOAT = 1), each of the fifteenth multiplexer M53 to the eighteenth multiplexer M83 can output the lower bit of the second extension corresponding to a received symbol from the fifth zero extension circuit ZE5 to the eighth zero extension circuit ZE8.

[0126] In some embodiments, in integer calculation mode (e.g., IS_FLOAT = 0), in response to the corresponding symbol in symbol information SI, each of the eleventh multiplexer M13 to the fourteenth multiplexer M43 can output the higher bit of a corresponding received first extension from the first symbol extension circuit SE1 to the fourth symbol extension circuit SE4, or the higher bit of a corresponding received second extension from the first zero extension circuit ZE1 to the fourth zero extension circuit ZE4. In integer calculation mode (e.g., IS_FLOAT = 0), in response to the corresponding symbol in symbol information SI, each of the fifteenth multiplexer M53 to the eighteenth multiplexer M83 can output the lower bit of a corresponding received first extension from the fifth symbol extension circuit SE5 to the eighth symbol extension circuit SE8, or the lower bit of a corresponding received second extension from the fifth zero extension circuit ZE5 to the eighth zero extension circuit ZE8. In integer calculation mode, the first aligned mantissa multiplication result F_align1 to the fourth aligned mantissa multiplication result F_align4 can be replaced with integer multiplication results.

[0127] The second adder element AE12 adds the output of the eleventh multiplexer M13 to the output of the twelfth multiplexer M23. The third adder element AE22 adds the output of the thirteenth multiplexer M33 to the output of the fourteenth multiplexer M43. The fourth adder element AE32 adds the output of the second adder element AE12 to the output of the third adder element AE22. The fifth adder element AE42 generates the first addition result data F_add1 by adding the output of the fourth adder element AE32 to the higher bits of the aligned mantissa ACC1_F_align. Although not shown, the higher bits of the aligned mantissa ACC1_F_align input to the fifth adder element AE42 correspond to sign-extended data.

[0128] The sixth adder element AE52 adds the output of the fifteenth multiplexer M53 to the output of the sixteenth multiplexer M63. The seventh adder element AE62 adds the output of the seventeenth multiplexer M73 to the output of the eighteenth multiplexer M83. The eighth adder element AE72 adds the output of the sixth adder element AE52 to the output of the seventh adder element AE62. The ninth adder element AE82 generates the second addition result data F_add2 by adding the output of the eighth adder element AE72 to the lower bits of the aligned mantissa ACC1_F_align. Although not shown, the lower bits of the aligned mantissa ACC1_F_align input to the ninth adder element AE82 may correspond to zero-extended data.

[0129] Reference Figure 12The floating-point operation circuit 20d may include nineteenth to twenty-first multiplexers M14, M24 and M34, tenth to twelfth adder elements AE13, AE23 and AE33, first leading 1 detector LOD_1 and second leading 1 detector LOD_2, sixth shift circuit SHT12 and seventh shift circuit SHT22, normalization and rounding circuit NR and cumulative data generation circuit CDG.

[0130] When the floating-point circuit 20d operates in integer calculation mode, the nineteenth multiplexer M14 and the twentieth multiplexer M24 can output "0" in response to the signal IS_FLOAT. Because the nineteenth multiplexer M14 and the twentieth multiplexer M24 output "0", the elements in the floating-point circuit 20d used for floating-point calculation can be deactivated. The following description is based on the assumption that the floating-point circuit 20d operates in floating-point calculation mode.

[0131] The nineteenth multiplexer M14 can output the first addition result data F_add1 to the tenth adder element AE13. The twentieth multiplexer M24 can output the second addition result data F_add2 to the tenth adder element AE13.

[0132] The tenth adder element AE13 generates a third addition result data by adding the first addition result data F_add1 to the second addition result data F_add2. A first leading 1 detector LOD_1 performs a first leading 1 detection on the "k" higher bits of the MSB containing the third addition result data, indicating that at least one bit of the sign has been removed from the third addition result data, where "k" is an integer at least 1. A second leading 1 detector LOD_2 performs a second leading 1 detection on the "n" lower bits of the LSB containing the third addition result data, indicating that at least one bit of the sign has been removed from the third addition result data, where "n" is an integer at least 1. The first leading 1 detector LOD_1 and the second leading 1 detector LOD_2 can be executed in parallel. For example, "k" can be less than "n". At least one bit representing the sign of the third addition result data can be provided to the cumulative data generation circuit CDG.

[0133] The sixth shift circuit SHT12 shifts the third addition result data output from the first preamble detector LOD_1 along a first direction. The seventh shift circuit SHT22 shifts the third addition result data output from the second preamble detector LOD_2 along a second direction. For example, the first direction can be opposite to the second direction. The first direction can be from higher bits to lower bits, and the second direction can be from lower bits to higher bits.

[0134] The eleventh adder element AE23 can receive a maximum value Max_E and shift information generated from the detection of the first leading 1 detector LOD_1 and the second leading 1 detector LOD_2, and add the maximum value Max_E to the shift information. The shift information can indicate the position of the detected "1" in the third addition result data, and can be generated by one of the first leading 1 detector LOD_1 and the second leading 1 detector LOD_2.

[0135] The twenty-first multiplexer M34 can provide the output to the normalization and rounding circuit NR, which corresponds to the leading 1 detector that has detected a "1" from the corresponding outputs of the sixth shift circuit SHT12 and the seventh shift circuit SHT22.

[0136] The normalization and rounding circuit NR can normalize and round the third addition result data that has been shifted by the sixth shift circuit SHT12 or the seventh shift circuit SHT22, and provide the exponent update information to the twelfth adder element AE33.

[0137] The twelfth adder element AE33 can add the exponent update information to the output of the eleventh adder element AE23 and provide the addition result to the cumulative data generation circuit CDG.

[0138] The cumulative data generation circuit CDG can use the output of the sign, normalization and rounding circuit NR of the third addition result data and the output of the twelfth adder element AE33 to generate the second cumulative data ACC2.

[0139] Figure 13 This is a block diagram of a computation circuit 1000 included in a neural network device according to an example embodiment.

[0140] Reference Figure 13 The arithmetic circuit 1000 may include a floating-point multiplier 1010, an integer multiplier 1020, an alignment shifter 1030, an adder 1040, a first post-adder 1050, a second post-adder 1060, and a buffer 1070. In addition to Figure 13 In addition to the components shown, the arithmetic circuit 1000 may also include multiple components for performing various neural network operations. Adder 1040 may include a first adder circuit 1041 and a second adder circuit 1042 that are logically or physically separated from each other. In one example embodiment, the arithmetic circuit 1000 may perform dot product and accumulation operations on floating-point data, and perform dot product operations on integer data.

[0141] In one example embodiment, the second adder circuit 1042 of adder 1040 may be shared by floating-point multiplier 1010 and integer multiplier 1020. When the arithmetic circuit 1000 is in integer calculation mode, floating-point multiplier 1010 may support integer multiplication operations.

[0142] In one example embodiment, when the arithmetic circuit 1000 is in floating-point calculation mode, the floating-point multiplier 1010 can perform floating-point calculations and provide the mantissa multiplication result to the first adder circuit 1041 and the second adder circuit 1042 via the alignment shifter 1030. When the arithmetic circuit 1000 is in integer calculation mode, the integer multiplier 1020 can perform integer calculations and provide the first integer multiplication result to the second adder circuit 1042, while the floating-point multiplier 1010 can perform integer calculations and directly provide the second integer multiplication result to the first adder circuit 1041 without going through the alignment shifter 1030.

[0143] In one example embodiment, when the arithmetic circuit 1000 is in floating-point calculation mode, adder 1040 can add the mantissa-aligned multiplication result of the floating-point data to the mantissa of the accumulated data, and provide the addition result data to a first post-adder 1050. The first post-adder 1050 can generate second accumulated data reflecting the updated exponent by performing leading 1 detection, normalization, and rounding on the addition result data, and store the second accumulated data in buffer 1070.

[0144] In one example embodiment, when the arithmetic circuit 1000 is in integer calculation mode, adder 1040 can add together the integer multiplication results of integer data and provide the addition result to second post-adder 1060. Second post-adder 1060 can generate second output data based on the addition result and store the second output data in buffer 1070.

[0145] In one example embodiment, the first post-adder 1050 and the second post-adder 1060 may be integrated with each other. In another example embodiment, the first post-adder 1050 and the second post-adder 1060 may be separate from each other.

[0146] Figure 14 This is a block diagram of an application processor 2000 according to an example embodiment.

[0147] Application processor 2000 may include CPU 2100, RAM 2200, GPU 2300, neural network device 2400, sensor interface 2500, display interface 2600, and memory interface 2700. Application processor 2000 may also include a communication module. The components of application processor 2000 (i.e., CPU 2100, RAM 2200, GPU 2300, neural network device 2400, sensor interface 2500, display interface 2600, and memory interface 2700) can exchange data with each other via bus 2800.

[0148] CPU 2100 provides overall control over the operation of application processor 2000. CPU 2100 may include a single core or multiple cores. CPU 2100 may process or execute programs and / or data stored in memory 2710. In one embodiment, CPU 2100 may control the functionality of neural network device 2400 by executing programs stored in memory 2710.

[0149] RAM 2200 may temporarily store programs, data, and / or instructions. In one embodiment, RAM 2200 may include DRAM or SRAM. RAM 2200 may temporarily store images input or output through an interface (e.g., sensor interface 2500 and display interface 2600) or generated by GPU 2300 or CPU 2100.

[0150] The application processor 2000 may also include ROM. ROM may store continuously used programs and / or data. ROM may include EPROM or EEPROM.

[0151] GPU 2300 can perform image processing on image data. For example, GPU 2300 can perform image processing on image data received through sensor interface 2500. In one embodiment, GPU 2300 can perform floating-point calculations.

[0152] Image data processed by GPU 2300 can be stored in memory 2710 or provided to display device 2610 via display interface 2600. Image data stored in memory 2710 can be provided to neural network device 2400.

[0153] The sensor interface 2500 can be connected to an interface for data (e.g., image data, audio data, etc.) input from the sensor 2510 connected to the application processor 2000.

[0154] The display interface 2600 can be connected to a data (e.g., image) interface output to the display device 2610. The display device 2610 can output images or data about images through a display (such as a liquid crystal display (LCD) or an active matrix organic light-emitting diode (AMOLED) display).

[0155] The memory interface 2700 can be connected to a data interface that receives data from or outputs to the memory 2710 external to the application processor 2000. In one embodiment, the memory 2710 may include volatile memory (such as DRAM or SRAM) or non-volatile memory (such as RRAM, PRAM, or NAND flash memory). The memory 2710 may include a memory card (such as a multimedia card (MMC), an embedded MMC (eMMC), an SD card, or a micro SD card).

[0156] The neural network device 2400 may include a floating-point arithmetic circuit 20, which performs the above-mentioned operations. Figures 1 to 13 The floating-point arithmetic circuit 20 is described above. It can perform dot product and accumulation operations on floating-point data with various formats, and reduces the shift width of the alignment shifter to a level that does not degrade the accuracy of the operation. Therefore, the size and power consumption of the floating-point arithmetic circuit 20 can be improved. The floating-point arithmetic circuit 20 can identify the maximum value from the exponent addition result of the floating-point data and the exponent value of the accumulated data in the dot product operation, and perform alignment shift and summation on the mantissa multiplication result of the floating-point data and the mantissa portion of the accumulated data based on the maximum value. Example embodiments of the floating-point arithmetic circuit 20 have been described above; therefore, example embodiments of the floating-point arithmetic circuit 20 are omitted below.

[0157] While the disclosure has been specifically shown and described with reference to exemplary embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the claims.

Claims

1. A neural network device, comprising: The floating-point arithmetic circuit is configured to generate the first accumulated data; and The buffer is configured to store the first accumulated data. The floating-point circuitry is also configured to perform dot product and accumulation operations through the following operations: Receives multiple floating-point data pairs and the first accumulated data stored in the buffer; The maximum value is identified from the exponent value of the first accumulated data and the multiple exponent addition results obtained by adding the exponents of the multiple floating-point data pairs respectively; Based on the maximum value, perform alignment shifting of the mantissa portion of the first accumulated data and the multiple mantissa multiplication results obtained by multiplying the mantissas of the multiple floating-point data pairs respectively; Perform the summation of the results of multiple aligned mantissa multiplications with the aligned mantissa portion of the first cumulative data. The detection of a first leading 1 for a first number of higher-order bits of the addition result data generated from the summation and the detection of a second leading 1 for a second number of lower-order bits of the addition result data are performed in parallel, wherein the higher-order bits include the most significant bits of the addition result data and the lower-order bits include the least significant bits of the addition result data; and The shift addition result data is based on the results of the first leading 1 detection and the second leading 1 detection. The first quantity is less than the second quantity.

2. The neural network device of claim 1, wherein, The floating-point circuitry is also configured to generate a second accumulated data through the following operations: Normalize and round the result of the shifted addition.

3. The neural network device of claim 2, wherein, The floating-point circuitry is also configured to store the second accumulated data in a buffer.

4. The neural network device of claim 1, wherein, The multiple data entries included in the plurality of floating-point data pairs have a format of a different type than the first cumulative data.

5. The neural network device according to claim 1, wherein, Multiple data entries included in the plurality of floating-point data pairs have floating-point 16 type format or floating-point 16 type format.

6. The neural network device according to claim 5, wherein, The first cumulative data has a floating-point 32 type format.

7. The neural network device according to claim 5, wherein, The floating-point arithmetic circuit is also configured to: extend the exponent field of the first data and the mantissa field of the second data among the plurality of data, wherein the first data has a floating-point 16 type format and the second data has a floating-point 16 type format.

8. The neural network device according to any one of claims 1 to 7, wherein, The floating-point arithmetic circuit includes: A first adder circuit is configured to add the higher bits of the results of the plurality of aligned mantissa multiplications to the higher bits of the aligned mantissa portion of the first accumulated data; and The second adder circuit is configured to add the lower bits of the results of the plurality of aligned mantissa multiplications to the lower bits of the aligned mantissa portion of the first accumulated data.

9. The neural network device according to claim 8, wherein, One of the first adder circuit and the second adder circuit is configured to be shared by an integer multiplier included in the neural network device, the integer multiplier being configured to perform integer multiplication operations.

10. The neural network device according to claim 8, wherein, The floating-point arithmetic circuit is also configured as follows: The second cumulative data is generated by adding the first addition result data output from the first adder circuit to the second addition result data output from the second adder circuit, and then normalizing and rounding the result of adding the first addition result data and the second addition result data. and Store the second accumulated data in the buffer.

11. The neural network device according to any one of claims 1 to 7, wherein, The floating-point circuitry includes an alignment shifter, which is configured as follows: The alignment shift of each of the plurality of mantissa multiplication results is performed along the direction corresponding to the first difference between each of the plurality of exponent addition results and the maximum value, by the number of bits corresponding to the first difference; and The mantissa portion of the first accumulated data is aligned and shifted in the direction corresponding to the second difference between the exponent value and the maximum value of the first accumulated data, by the number of bits corresponding to the second difference.

12. The neural network device according to claim 11, wherein, The shift width of the alignment shifter is determined based on the operational accuracy required by the application using data generated by the neural network device.

13. A method of operating a neural network device, the method comprising: The first accumulated data is generated through the floating-point arithmetic circuit; The first accumulated data is stored in a buffer; The floating-point arithmetic circuit receives multiple floating-point data pairs and the first accumulated data stored in the buffer. The maximum value is identified by the floating-point arithmetic circuit from the exponent value of the first accumulated data and the multiple exponent addition results obtained by adding the exponents of the multiple floating-point data pairs respectively. The floating-point arithmetic circuit performs alignment shifting of the mantissa portion of the first accumulated data and the mantissa multiplication results obtained by multiplying the mantissas of the plurality of floating-point data pairs respectively, based on the maximum value. The results of multiple aligned mantissa multiplications are added to the aligned mantissa portion of the first accumulated data using a floating-point arithmetic circuit. and The second accumulated data is generated by the floating-point arithmetic circuit based on the result of the addition. The step of generating the second accumulated data includes: performing a first leading 1 detection on the higher-order bits of a first number of addition result data generated by addition, wherein the higher-order bits include the most significant bit of the addition result data; performing a second leading 1 detection on the lower-order bits of a second number of addition result data, wherein the lower-order bits include the least significant bit of the addition result data; and shifting the addition result data based on the results of the first leading 1 detection and the second leading 1 detection. The first quantity is less than the second quantity. The first preamble 1 detection and the second preamble 1 detection are performed in parallel.

14. The method according to claim 13, wherein, The steps for performing an alignment shift include: Perform an alignment shift of each of the plurality of mantissa multiplication results along the direction corresponding to the first difference between each of the plurality of exponent addition results and the maximum value, by the number of bits corresponding to the first difference; and The mantissa portion of the first accumulated data is aligned and shifted in the direction corresponding to the second difference between the exponent value and the maximum value of the first accumulated data, by the number of bits corresponding to the second difference.

15. The method according to claim 13 or 14, wherein, The steps for generating the second cumulative data include: generating the second cumulative data by performing normalization and rounding on the result of the shift-addition result data.

16. An application processor including a neural network device, wherein, The neural network device includes: The floating-point arithmetic circuit is configured to generate the first accumulated data; and The buffer is configured to store the first accumulated data; The floating-point circuitry is also configured to perform dot product and accumulation operations by: Receives multiple floating-point data pairs and the first accumulated data stored in the buffer; Based on the exponent value of the first accumulated data and the multiple exponent addition results obtained by adding the exponents of the multiple floating-point data pairs respectively, the alignment shift of the mantissa portion of the first accumulated data and the multiple mantissa multiplication results obtained by multiplying the mantissas of the multiple floating-point data pairs respectively is performed. Perform the summation of the results of multiple aligned mantissa multiplications with the aligned mantissa portion of the first cumulative data. The detection of a first leading 1 for a first number of higher-order bits of the addition result data generated from the summation and the detection of a second leading 1 for a second number of lower-order bits of the addition result data are performed in parallel, wherein the higher-order bits include the most significant bits of the addition result data and the lower-order bits include the least significant bits of the addition result data; and The shift addition result data is based on the results of the first leading 1 detection and the second leading 1 detection. The first quantity is less than the second quantity.

Images