Methods and apparatus for executing neural networks

By quantizing the floating-point neural network parameters into a fixed-point format, and utilizing the statistical information of weights and activations and the Lp norm, the problems of high computational cost and high memory access frequency in low-performance devices are solved, thus achieving efficient neural network inference in low-performance devices.

CN114254746BActive Publication Date: 2026-11-17SAMSUNG ELECTRONICS CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110266381.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-09-23
Filing Date
2021-03-11
Publication Date
2026-11-17
Estimated Expiration
2041-03-11

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively process floating-point neural network parameters on low-performance devices, resulting in high computational demands and frequent memory accesses, hindering the smooth execution of neural network inference.

Method used

By quantizing the parameters of the neural network, the floating-point format is converted to the fixed-point format. The dynamic range is determined using the statistical information of the weights and activations, and quantization is performed based on the Lp norm to generate a neural network model in fixed-point format.

Benefits of technology

It reduces computational load and memory access frequency, making it suitable for neural network inference in low-performance devices while maintaining high accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114254746B_ABST
    Figure CN114254746B_ABST
Patent Text Reader

Abstract

A method and apparatus for executing a neural network are disclosed. A method for quantizing the parameters of a neural network includes: acquiring parameters in floating-point format used in a process inferred by the neural network; quantizing the weights to fixed-point format based on statistics of weights included in the parameters; determining the dynamic range of activations based on statistics of activations of one or more layers of the neural network configured in the parameters; and quantizing input data to fixed-point format based on statistics of input data of the neural network.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0123172, filed on September 23, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for quantizing the parameters of neural networks. Background Technology

[0003] To address the problem of classifying input patterns into specific groups, active research has been conducted to apply effective human pattern recognition methods to real computers. One area of ​​research relates to artificial neural networks (ANNs), which are modeled through mathematical expressions of the characteristics of human biological neurons. To solve the problem of classifying input patterns into specific groups, ANNs use algorithms that mimic human learning abilities. Based on these algorithms, ANNs can generate mappings between input and output patterns, and this ability to generate mappings is represented as the learning ability of the ANN. Furthermore, ANNs possess generalization capabilities, enabling them to generate relatively accurate outputs based on the learned results for input patterns that have not yet been used for learning. Summary of the Invention

[0004] This summary provides a simplified description of the concepts further described in the following detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method for quantizing the parameters of a neural network includes: obtaining parameters in floating-point format used in a process inferred by the neural network; quantizing the weights in fixed-point format based on statistics of weights included in the parameters; determining the dynamic range of activations based on statistics of activations of one or more layers of the neural network configured in the parameters; and quantizing input data in fixed-point format based on statistics of input data of the neural network.

[0006] The method may further include: inputting quantized input data into a neural network; performing an operation between the quantized input activation of the current layer of the neural network and the quantized weights of the current layer, and generating the output activation of the current layer; and quantizing the output activation of the current layer based on the dynamic range of the activation.

[0007] The steps of quantizing weights into a fixed-point format may include: calculating the Lp norm of the weights; determining the dynamic range of the weights based on the Lp norm of the weights; and quantizing the weights into a fixed-point format based on the dynamic range of the weights.

[0008] The steps for determining the dynamic range of activation may include: calculating a relational expression between the Lp norm of the layer's output activation and the Lp norm of the input activation based on the parameters; and inferring the Lp norm of the layer's output activation based on the relational expression and the Lp norm of the layer's output activation from the previous layer.

[0009] The steps to determine the dynamic range of activation may include: determining the dynamic range of the layer's output activation based on the Lp norm of the layer's output activation.

[0010] The steps for determining the dynamic range of activation may include: classifying the layers into a first layer that includes a normalized layer and a second layer that does not include a normalized layer; inferring the Lp norm of the output activation of the first layer based on the normalized parameters of the first layer included in the parameters; and inferring the Lp norm of the output activation of the second layer based on the parameters and the Lp norm of the output activation of the first layer.

[0011] The steps for inferring the Lp norm of the output activation of the second layer may include: calculating a relational expression between the Lp norm of the input activation and the Lp norm of the output activation of the second layer based on parameters; determining the Lp norm of the input activation of the second layer based on the Lp norm of the output activation of the first layer; and inferring the Lp norm of the output activation of the second layer based on the Lp norm of the input activation and the relational expression.

[0012] The steps for determining the dynamic range of activation may include: determining the dynamic range of the output activation of the first layer based on the Lp norm of the output activation of the first layer; and determining the dynamic range of the output activation of the second layer based on the Lp norm of the output activation of the second layer.

[0013] In another general aspect, a method for quantifying the parameters of a neural network includes: obtaining the parameters in floating-point format used in a process inferred by the neural network; calculating the Lp norm of the weights included in the parameters; inferring the Lp norm of the output activations of one or more layers of the configured neural network included in the parameters; and calculating the Lp norm of the input data of the neural network.

[0014] The method may further include: determining the dynamic range of the weights based on the Lp norm of the weights; quantizing the weights into a fixed-point format based on the dynamic range of the weights; determining the dynamic range of the output activation of the layer based on the Lp norm of the layer's output activation; determining the dynamic range of the input data based on the Lp norm of the input data; and quantizing the input data into a fixed-point format based on the dynamic range of the input data.

[0015] The method may further include: inputting quantized input data into a neural network; performing an operation between the quantized input activation of the current layer of the neural network and the quantized weights of the current layer, and generating the output activation of the current layer; and quantizing the output activation of the current layer based on the dynamic range of the activation.

[0016] The steps for calculating the Lp norm of the output activation may include: calculating a relational expression between the Lp norm of the layer's output activation and the Lp norm of the input activation based on the parameters; and inferring the Lp norm of the layer's output activation based on the relational expression and the Lp norm of the layer's previous output activation.

[0017] In another general aspect, a neural network device for quantizing parameters of a neural network includes: a processor configured to: acquire parameters in floating-point format used in a process inferred by the neural network; quantize the weights in fixed-point format based on statistics of weights included in the parameters; determine the dynamic range of activations based on statistics of activations of one or more layers of the neural network included in the parameters; and quantize input data in fixed-point format based on statistics of input data of the neural network.

[0018] The processor can be configured to: compute the Lp norm of the weights, determine the dynamic range of the weights based on the Lp norm of the weights, and quantize the weights into a fixed-point format based on the dynamic range of the weights.

[0019] The processor can be configured to: calculate the relational expression between the Lp norm of the output activation of the layer and the Lp norm of the input activation based on the parameters, and infer the Lp norm of the output activation of the layer based on the relational expression and the Lp norm of the output activation of the previous layer.

[0020] The processor can be configured to determine the dynamic range of the layer's output activation based on the Lp norm of the layer's output activation.

[0021] The processor can be configured to: classify layers into a first layer including a normalized layer and a second layer excluding a normalized layer; infer the Lp norm of the output activation of the first layer based on the normalized parameters of the first layer included in the parameters; and infer the Lp norm of the output activation of the second layer based on the parameters and the Lp norm of the output activation of the first layer.

[0022] The processor can be configured to: calculate the relationship expression between the Lp norm of the input activations and the Lp norm of the output activations of the second layer based on parameters; determine the Lp norm of the input activations of the second layer based on the Lp norm of the output activations of the first layer; and infer the Lp norm of the output activations of the second layer based on the Lp norm of the input activations and the relationship expression.

[0023] The processor can be configured to: determine the dynamic range of the output activation of the first layer based on the Lp norm of the output activation of the first layer, and determine the dynamic range of the output activation of the second layer based on the Lp norm of the output activation of the second layer.

[0024] In another general aspect, a neural network quantization device for quantizing parameters of a neural network includes: a processor configured to: acquire parameters in floating-point format used in a process inferred by the neural network; calculate the Lp norm of weights included in the parameters; infer the Lp norm of output activations of one or more layers of the configured neural network included in the parameters; and calculate the Lp norm of the input data of the neural network.

[0025] The processor can be configured to: calculate the relational expression between the Lp norm of the output activation of the layer and the Lp norm of the input activation based on the parameters, and infer the Lp norm of the output activation of the layer based on the relational expression and the Lp norm of the output activation of the previous layer.

[0026] In another general aspect, a neural processing unit (NPU) includes: a memory including the Lp norm of weights, the Lp norm of output activations of a layer, and the Lp norm of input data; and a neural network quantization device for determining the dynamic range of weights based on the Lp norm of the weights, quantizing the weights into a fixed-point format based on the dynamic range of the weights, determining the dynamic range of output activations of a layer based on the Lp norm of the output activations of the layer, determining the dynamic range of input data based on the Lp norm of the input data, and quantizing the input data into a fixed-point format based on the dynamic range of the input data.

[0027] The NPU may further include an arithmetic logic unit (ALU) configured to perform operations between the quantized input activations of the current layer of the neural network and the quantized weights of the current layer, and generate the output activations of the current layer, wherein the neural network quantization device may quantize the output activations of the current layer based on the dynamic range of the output activations.

[0028] In another general aspect, a neural network inference device includes: a memory including parameters in floating-point format used in the inference process performed by the neural network; a neural network quantization device that retrieves the parameters from the memory, quantizes the weights to a fixed-point format based on statistics of the weights included in the parameters, determines the dynamic range of the activations based on statistics of the activations of one or more layers of the neural network configured in the parameters, and quantizes the input data to a fixed-point format based on statistics of the input data of the neural network; and an NPU configured to perform inference by receiving the quantized weights, the dynamic range of the activations, and the quantized input data from the neural network quantization device.

[0029] The NPU can be configured to: perform operations between the quantized input activations of the current layer of the neural network and the quantized weights of the current layer, generate the output activations of the current layer, and quantize the output activations of the current layer based on the dynamic range of the activations.

[0030] In another general aspect, a processor-implemented method includes: acquiring parameters of a floating-point neural network model; calculating the Lp norm of weights included in the parameters; obtaining a relational expression between the Lp norm of the output activation and the Lp norm of the input activation of the floating-point neural network model based on the Lp norm of the weights; calculating a predicted value of the Lp norm of the output activation based on the Lp norm of the input activation; determining the dynamic range of the activation of the floating-point neural network model based on the predicted value; and quantizing the floating-point neural network model into a fixed-point neural network model based on the dynamic range.

[0031] Output activation can be the output activation of a convolutional layer that includes a normalized layer.

[0032] The Lp norm can include the L2 norm.

[0033] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating an example of quantizing floating-point parameters in a neural network into fixed-point parameters.

[0035] Figure 2 This is a diagram illustrating an example of an operation performed in a neural network.

[0036] Figure 3 This is a block diagram illustrating an example of the hardware configuration of a neural network inference device.

[0037] Figure 4 This is a block diagram illustrating another example of the hardware configuration of a neural network inference device.

[0038] Figure 5 This is a diagram illustrating an example of methods for quantizing weights and activation.

[0039] Figure 6 This is a diagram illustrating an example of a method for quantifying parameters based on the Lp norm of the parameters.

[0040] Figure 7 This is a diagram illustrating an example of quantizing floating-point numbers to fixed-point numbers in a neural network.

[0041] Figure 8 This is a flowchart illustrating an example of a quantization method.

[0042] Figure 9 This is a flowchart illustrating another example of a quantization method.

[0043] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0044] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0045] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0046] The structural or functional descriptions of the examples disclosed in this disclosure are intended only to describe the purpose of the examples, and the examples can be implemented in various forms. The examples are not intended to be limiting, but rather to indicate that various modifications, equivalents, and substitutions are also covered within the scope of the claims.

[0047] Although the terms "first" or "second" are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the conception of this disclosure, a "first" component may be referred to as a "second" component, or similarly, a "second" component may be referred to as a "first" component.

[0048] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" said other element, "connected to," or "bonded to" said other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0049] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that the term "comprising," when used in this specification, indicates the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0050] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in general dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0051] Examples can be implemented as various types of products (such as data centers, personal computers (PCs), laptops, tablets, smartphones, televisions, smart home devices, smart vehicles, kiosks, and wearable devices). The examples will be described in detail below with reference to the accompanying drawings, and the same reference numerals throughout the drawings denote the same elements.

[0052] Figure 1 This is a diagram illustrating an example of quantizing floating-point parameters in a neural network into fixed-point parameters.

[0053] Reference Figure 1 The neural network learning device 100 corresponds to a computing device with various processing functions (such as the function of generating neural networks, training or learning neural networks, and retraining neural networks). For example, the neural network learning device 100 can be implemented as various types of devices (such as PCs, server devices, mobile devices, etc.).

[0054] The neural network learning device 100 generates a trained neural network 110 by repeatedly training or learning from a given initial neural network. The initial neural network may have parameters in floating-point format (e.g., parameters with 32-bit floating-point precision) to ensure the processing accuracy of the neural network. Here, for example, the parameters may include various types of data input to or from the neural network (such as the neural network's input / output activations, weights, and biases). As the neural network is repeatedly trained, its floating-point parameters can be tuned to compute a more accurate output for a given input.

[0055] The neural network learning device 100 can transmit the trained neural network 110 to the neural network inference device 150. The neural network inference device 150 can be included in, for example, mobile devices and embedded devices. The neural network inference device 150 is dedicated hardware for driving neural networks and is implemented with relatively low power and low performance. Therefore, the neural network inference device 150 can be implemented to be more suitable for fixed-point operations than floating-point operations.

[0056] The neural network inference device 150 driving the quantized neural network 160 can be implemented in a separate device independent of the neural network learning device 100. However, this is merely an example, and the neural network inference device 150 can be implemented in the same device as the neural network learning device 100. In one example, for instance, the neural network can be executed by a processor (e.g., a neural processor) based on quantized fixed-point formatted parameters and quantized input data.

[0057] Meanwhile, compared to fixed-point, floating-point requires a relatively large amount of computation and a high memory access frequency. Specifically, it is known that most of the computation required to process neural networks is the multiply-accumulate (MAC) operation performed on various parameters. Therefore, neural network inference devices 150, including those in smartphones, tablet PCs, and wearable devices with relatively low processing performance, may not be able to handle neural networks with floating-point formatted parameters smoothly. Therefore, in order to drive neural networks within an acceptable loss of accuracy while sufficiently reducing the amount of computation in such devices, it is desirable to quantize the floating-point formatted parameters processed in the neural network. Here, parameter quantization means converting parameters with high-precision floating-point format to parameters with low-precision fixed-point format. Fixed-point formatted parameter conversion indicates the process of redefining functions implemented using decimals as functions related to integer arithmetic, and then integerizing all decimal arithmetic operations of the floating-point source code. If a floating-point variable is multiplied by an appropriate value to form an integer, integer arithmetic using integer operators can be performed. The resulting value can be divided by the value multiplied above to convert it back to a floating-point variable.

[0058] The neural network quantization device can perform quantization to convert the parameters of a trained neural network 110 into a pre-defined fixed-point format (e.g., N bits), where N is a positive integer. The neural network quantization device can operate within a neural network learning device 100 or a neural network inference device 150, or as a stand-alone third hardware accelerator. The neural network quantization device can perform quantization to convert the parameters of the trained neural network 110 into a pre-defined fixed-point format based on the processing performance of the neural network inference device 150 used for deploying the neural network, and transmit the quantized neural network 160 to the neural network inference device 150 used for deploying the neural network.

[0059] The neural network inference device 150, which will be deployed with the quantized neural network 160, may be included in, but is not limited to, autonomous vehicles, robots, smartphones, tablets, augmented reality (AR) devices, and Internet of Things (IoT) devices.

[0060] Figure 2 This is a diagram illustrating an example of an operation performed in a neural network.

[0061] Reference Figure 2 The neural network 200 has a structure including an input layer, hidden layers, and an output layer. The neural network 200 can perform operations based on received input data (e.g., I1 and I2) and can generate output data (e.g., O1 and O2) based on the results of the performed operations. For example, the input data may include at least one of image data, speech data, and text data.

[0062] As described above, neural network 200 can be an n-layer neural network or a deep neural network (DNN) comprising two or more hidden layers. Figure 2 In the example shown, neural network 200 is a DNN that includes an input layer (layer 1), two hidden layers (layer 2 and layer 3), and an output layer (layer 4). DNNs may include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks, and restricted Boltzmann machines.

[0063] When neural network 200 is implemented using a DNN architecture, it includes multiple hidden layers for efficient information processing, enabling it to handle more complex datasets than a neural network with a single hidden layer. Although neural network 200 is shown as having four layers, this is merely an example, and it may include more or fewer layers, or more or fewer channels. In other words, neural network 200 may include layers with... Figure 2 The diagram shows layers with different structures.

[0064] Each layer in a neural network 200 includes multiple channels. A channel corresponds to multiple artificial nodes, also referred to as neurons, processing elements (PEs), units, or similar terms. For example, as... Figure 2 As shown, layer 1 includes two channels (nodes), and each of layers 2 and 3 includes three channels (nodes, e.g., CH1, CH2, and CH3). However, this is just an example, and each layer included in neural network 200 may include a variety of numbers of channels (nodes).

[0065] The channels in the layers of the neural network 200 are interconnected to process data. For example, one channel can receive data from other channels and perform various operations on the data, and can output the results of the operations to other channels.

[0066] The input to each channel can be referred to as the input activation, and the output of each channel can be referred to as the output activation. In other words, activation is a parameter that is the output of one channel and simultaneously the input to one or more channels included in the next layer. Each channel can determine its own activation based on the activations, weights, and biases received from the channels included in the previous layer. Weights are parameters used to calculate the output activation in each channel and are values ​​assigned to the connections between channels.

[0067] Each channel can be processed by a computing unit or processing element that receives input and outputs activation. The input and output of each channel can be mapped. For example, if σ represents the activation function, This represents the weights from the k-th node included in layer (i-1) to the j-th node included in layer i. This represents the bias of the j-th node included in the i-th layer. Let j represent the activation of the j-th node in the i-th layer. The activation can be calculated according to Equation 1 below.

[0068] Equation 1:

[0069]

[0070] like Figure 2 As shown, the activation of the first channel (CH1) in the second layer (layer 2) is determined by... This indicates that... Furthermore, according to Equation 1, It can have The value of . However, Equation 1 above is merely an example to illustrate the activations, weights, and biases that can be used to process data in neural network 200, and Figure 2 Examples of neural networks are not limited to this. For example, activation can be a value obtained by allowing a weighted sum of activations received from the previous layer to pass through an activation function (such as a sigmoid function or a ReLU function).

[0071] Figure 3 This is a block diagram illustrating an example of the hardware configuration of a neural network inference device.

[0072] Reference Figure 3 The neural network inference device 300 may include a first memory 310, a quantization device 320, and a neural processing unit (NPU) 330.

[0073] The neural network inference device 300 may be included in, for example, embedded devices and / or mobile devices (such as smartphones, tablet PCs, and wearable devices) with relatively low processing performance. Since floating-point operations require a relatively large amount of computation and a high memory access frequency compared to fixed-point operations, the neural network inference device 300 included in mobile devices, embedded devices, etc., may not be able to smoothly process neural networks with parameters in floating-point format. Therefore, the neural network inference device 300 may perform inference using a neural network generated using parameters in a neural network trainer in a fixed-point format, wherein the floating-point format parameters processed in the neural network are quantized into fixed-point format parameters.

[0074] Therefore, instead of receiving a quantized neural network 160 from a separate neural network learning device 100, the neural network inference device 300 can use a quantization device 320 to receive floating-point format parameters and quantize the floating-point format parameters into fixed-point format parameters.

[0075] Without using a separate configuration file, the quantization device 320 can convert floating-point format parameters to quantized fixed-point format parameters based on the parameters' statistics. For example, the quantization device 320 can quantize a 32-bit floating-point parameter to a 16-bit or fewer fixed-point parameter.

[0076] The first memory 310 is hardware configured to store various types of data processed in the neural network inference device 300, and can store both data that has already been processed and data that will be processed in the neural network inference device 300. For example, the first memory 310 can store data associated with the trained neural network 110. Furthermore, the first memory 310 can store applications and drivers that will be executed by the neural network inference device 300. The first memory 310 can be, but is not limited to, dynamic random access memory (DRAM).

[0077] The quantization device 320 can acquire floating-point format parameters used in the inference process performed by the neural network, which are stored in the first memory 310. For example, the neural network inference device 300 can use a direct memory access (DMA) controller (not shown) to load the floating-point format parameters stored in the first memory 310 into the quantization device 320. The quantization device 320 can use statistical information about the parameters to determine the appropriate dynamic range for quantizing the parameters. The dynamic range can refer to the number of bits required to represent the parameters. For example, the quantization device 320 can quantize a 32-bit floating-point format parameter to an 8-bit dynamic range. (Refer to the following...) Figures 5 to 9 The process of quantizing floating-point format parameters into fixed-point format parameters is described in more detail.

[0078] The NPU 330 can perform inference using a neural network generated from parameters quantized in a fixed-point format using a neural network trainer. The Arithmetic Logic Unit (ALU) 332 processes convolution operations between quantized input activations and weights, and applies the convolution result to an activation function to generate an output activation. The second memory 331 stores quantized neural network data, as well as neural network-related datasets that will be processed during inference or have already been processed during inference. Furthermore, the second memory 331 can store intermediate values ​​computed during processing using the ALU 332 to perform bitwise operations and integer arithmetic.

[0079] In one example, a neural network inference device can be used for image recognition. The neural network inference device may include a memory, a neural network quantization device, and a neural processing unit (NPU). The memory may include parameters in floating-point format used in the inference process performed by the neural network. The neural network quantization device may be configured to: retrieve parameters from the memory; quantize weights to a fixed-point format based on statistics of weights included in the parameters; determine the dynamic range of activations based on statistics of activations of one or more layers of the neural network included in the parameters; and quantize input image data to a fixed-point format based on statistics of the input image data of the neural network. The neural processing unit may be configured to: perform inference by receiving the quantized weights, the dynamic range of activations, and the quantized input image data from the neural network quantization device. However, the above example is merely illustrative, and the invention is not limited thereto.

[0080] Figure 4 This is a block diagram illustrating another example of the hardware configuration of a neural network inference device.

[0081] Reference Figure 4 The neural network inference device 400 may include a first memory 410, an Lp norm computing device 420, and an NPU 430.

[0082] Unlike the neural network inference device 300, where parameter statistics calculation and quantization are performed in the quantization device 320, the neural network inference device 400 can perform parameter statistics calculation in the Lp norm calculation device 420 and perform parameter quantization in the quantization device 432 of the NPU 430.

[0083] The Lp norm calculation device 420 can acquire floating-point format parameters stored in the first memory 410 and calculate or infer statistical information about the parameters. For example, the Lp norm calculation device 420 can acquire the weights of a neural network stored in the first memory 410 and calculate the Lp norm of the weights. The Lp norm relates to a method of measuring the magnitude of a vector and may include L1 norm and L2 norm. The L2 norm can be calculated using the following Equation 2.

[0084] Equation 2:

[0085]

[0086] In equation 2, x1 to x n Let ||x||2 represent the elements of vector x, and let ||x||2 represent the L2 norm of vector x. However, in addition to the Lp norm, another metric for measuring the size or length of a vector can be used, as shown in the example.

[0087] The NPU 430 can receive the Lp norm of the parameters from the Lp norm computing device 420, quantize the floating-point format parameters into fixed-point format parameters based on the Lp norm of the parameters, and perform inference using a neural network generated using the quantized parameters in a neural network trainer.

[0088] The second memory 431 can store the Lp norm of the parameters received from the Lp norm calculation device 420. The quantization device 432 can determine the dynamic range of the parameters based on the Lp norm of the parameters received from the second memory 431, and quantize the floating-point format parameters into fixed-point format parameters. (Refer to the following...) Figures 5 to 9 The process of quantizing floating-point format parameters to fixed-point format parameters is described in more detail. The ALU 433 can handle the convolution operation between the quantized input activation and the weights, and apply the result of the convolution operation to the activation function to generate the output activation. Furthermore, the second memory 431 can store intermediate values ​​computed during the processing of bitwise operations and integer arithmetic operations performed using the ALU 433.

[0089] Figure 5 This is a diagram illustrating an example of methods for quantizing weights and activation.

[0090] Reference Figure 5 The quantization device can quantize weights and activations. The neural network inference device can perform inference using the weights of a trained neural network stored in a first memory. Therefore, the quantization device can pre-quantize weights offline.

[0091] Conversely, activations are determined after the input data for inference is fed into the neural network and the network is executed. Therefore, the quantization device does not need to pre-quantize the activations offline. Instead, the quantization device can determine the dynamic range of the activations offline. The quantization device can determine the dynamic range of the output activations based on the relationship between the input and output activations of the layers of the neural network.

[0092] Reference Figure 5 The neural network inference device can use quantized weights 540 and activations with defined dynamic ranges to output quantized output activations 590.

[0093] In operation 530, the quantization device can quantize weight 520 offline.

[0094] In operation 550, the neural network inference device can perform a convolution operation between quantized input activations 510 and quantized weights 540. If the parameters are quantized to 8 bits, the convolution operation can be an 8x8 operation. If the data is quantized to 4 bits, the convolution operation can be a 4x4 operation.

[0095] In operation 560, the neural network inference device can output an activation by performing an activation operation (e.g., a ReLU operation (or operation)) on the result of the convolution operation.

[0096] In operation 580, the neural network inference device can quantize the output activation based on a predetermined dynamic range of the output activation. For example, the neural network inference device can quantize the output activation 570, represented by 16 bits, into an output activation with a predetermined dynamic range of 8 bits.

[0097] Figure 6 This is a diagram illustrating an example of a method for quantifying parameters based on the Lp norm of the parameters.

[0098] Reference Figure 6 The quantization device can quantize floating-point format weights and activations 610 into fixed-point format weights and activations 650.

[0099] In operation 620, the quantization device can calculate the Lp norm of the weights and infer the Lp norm of the activations. Since the weights are predetermined once training is complete, the quantization device can calculate the Lp norm of the weights. After the input data for inference is fed into the neural network and the neural network is executed, the activations are determined. Therefore, the exact Lp norm of the activations cannot be calculated. Instead, the quantization device can infer the Lp norm of the output activations based on the relationship between the input and output activations of the layers of the neural network. (See reference...) Figure 7 The method for inferring the Lp norm of activation is described in more detail.

[0100] In operation 630, the quantization device can determine the dynamic range of the parameter based on the calculated or inferred Lp norm of the parameter. The quantization device can determine the dynamic range of the parameter based on the following equation 3.

[0101] Equation 3:

[0102] clip_val = k × ||a||2

[0103] In Equation 3, k represents a pre-defined constant (e.g., assuming a Laplace distribution). or ||a||2 represents the Lp norm of parameter a, and clip_val represents the dynamic range of parameter a.

[0104] In operation 640, the neural network inference device can output fixed-point weights and activations by quantizing floating-point weights and activations 650.

[0105] Figure 7 This is a diagram illustrating an example of quantizing floating-point numbers to fixed-point numbers in a neural network.

[0106] Reference Figure 7 The quantization device 320 can load floating-point format parameters used in the inference process performed by the neural network from the first memory 310. For example, the quantization device 320 can load the weights of the first convolutional layer 720 (e.g., Conv1), the weights of the second convolutional layer 740 (e.g., Conv2), the weights of the third convolutional layer 760 (e.g., Conv3), and data associated with the relationship between the Lp norm of the input activations and the Lp norm of the output activations of each of the layers 720, 730, 740, 750, 760, and 770.

[0107] The quantization device 320 can quantize the weights of the first convolutional layer 720, the weights of the second convolutional layer 740, and the weights of the third convolutional layer 760. For example, the quantization device 320 can calculate the Lp norm of the weights of the first convolutional layer 720, the second convolutional layer 740, and the third convolutional layer 760, represented in 32-bit form, and quantize each weight to 8 bits by determining the dynamic range of the corresponding weight. As described above, the quantization device 320 can perform weight quantization offline.

[0108] The quantization device 320 can predict the Lp norm of the output activation of the convolutional layer, including the normalized layer, according to the following equation 4.

[0109] Equation 4:

[0110]

[0111] In Equation 4, ||out||2 represents the Lp norm of the output activation, a represents the scaling parameter of the normalization layer, and b represents the offset parameter of the normalization layer.

[0112] For example, in Figure 7 In the first convolutional layer 720, a convolutional layer 721 (e.g., Conv), a normalization layer 722 (e.g., BN), and an activation layer 723 (e.g., ReLU) may be included. The second convolutional layer 740 may include a convolutional layer 741 and an activation layer 742, and the third convolutional layer 760 may include a convolutional layer 761 and a normalization layer 762. Since the first convolutional layer 720 and the third convolutional layer 760 each include a normalization layer 722 and a normalization layer 762, respectively, the quantization device 320 can predict the Lp norm of the output activation of the first convolutional layer 720 and the third convolutional layer 760 based on the normalization parameters of the normalization layers 722 and 762.

[0113] In the following text, the layer including the normalization layer is referred to as the first layer, and the layer not including the normalization layer is referred to as the second layer. As mentioned above, the Lp norm of the output activation can be inferred from Equation 4 of the first layer.

[0114] The quantization device 320 can infer the Lp norm of the output activation of the second layer based on the Lp norm of the output activation of the first layer and the relationship expression between the Lp norm of the input activation of the second layer and the Lp norm of the output activation.

[0115] The relationship between the Lp norm of input activation and the Lp norm of output activation can be determined based on the type of layer. For example, the relationship between the Lp norm of input activation and the Lp norm of output activation for fully connected layers, element-wise summation layers, concatenate layers, and convolutional layers can be expressed as Equations 5, 6, 7, and 8, respectively.

[0116] Equation 5:

[0117] |O|2≤|I|2*max{∑ nn}

[0118] Equation 6:

[0119] ||A||2=∑|A i ||2

[0120] Equation 7:

[0121]

[0122] Equation 8:

[0123]

[0124] In Equation 5, the L2 norm |O|2 of the output vector of the fully connected layer can be obtained from the maximum singular value of the weights, max{∑ nn The constraint is determined by the product of the L2 norm |I|2 of the input vector and the input vector.

[0125] In the case of element-wise summation, A = ∑A i Furthermore, in the case of element-wise summation, Equation 6 can be obtained under the assumption that the elements are independent.

[0126] In equation 7, N i The input tensor represents the number of parameters, ||A||² represents the L2 norm of the output tensor, and ∑N i ||A i ||2 represents the weighted sum of the L2 norms of the input tensors.

[0127] In equation 8, kw and k h The width and height of the kernel are represented by ||w||2, the L2 norm of the weights is represented by ||a||2, the L2 norm of the input activations is represented by ||O||2, and the L2 norm of the output activations is represented by ||O||2.

[0128] The quantization device 320 can load the relational expression for the second layer from the first memory 310. The Lp norm of the output activation of the first convolutional layer 720 can be the same as the Lp norm of the input activation of the element-wise summation layer 730. Therefore, the quantization device 320 can infer the Lp norm of the output activation of the element-wise summation layer 730 based on the Lp norm of the output activation of the first convolutional layer 720 and the relational expression (e.g., Equation 6) between the Lp norm of the input activation and the Lp norm of the output activation of the element-wise summation layer 730.

[0129] Similarly, the Lp norm of the output activation of the element-wise summation layer 730 can be the same as the Lp norm of the input activation of the second convolutional layer 740. Therefore, the quantization device 320 can infer the Lp norm of the output activation of the second convolutional layer 740 based on the relationship expression between the Lp norm of the output activation of the element-wise summation layer 730 and the Lp norm of the input activation of the second convolutional layer 740 and the Lp norm of the output activation (e.g., Equation 8).

[0130] Furthermore, since the pooling layer 750 does not cause a change in the Lp norm, the Lp norm of the output activation of the second convolutional layer 740 can be the same as the Lp norm of the input activation of the third convolutional layer 760. Therefore, the quantization device 320 can infer the Lp norm of the output activation of the third convolutional layer 760 based on the relationship expression between the Lp norm of the output activation of the second convolutional layer 740 and the Lp norm of the input activation and output activation of the third convolutional layer 760 (e.g., Equation 8).

[0131] Similarly, the Lp norm of the output activation of the third convolutional layer 760 can be the same as the Lp norm of the input activation of the fully connected layer 770. Therefore, the quantization device 320 can infer the Lp norm of the output activation of the fully connected layer 770 based on the Lp norm of the output activation of the third convolutional layer 760 and the relationship expression between the Lp norm of the input activation and the Lp norm of the output activation of the fully connected layer 770 (e.g., Equation 5).

[0132] If there is no first layer in the neural network, the quantization device 320 can measure the Lp norm of the input activations of the first-ordered layer. The quantization device 320 can use the relational expression and the Lp norm of the input activations of the first-ordered layer to infer the Lp norm of the output activations of subsequent layers.

[0133] The quantization device 320, which acquires the floating-point format weights and the Lp norm of the activations used in the inference process performed by the neural network, can determine the dynamic range of the weights and activations. Furthermore, the quantization device 320 can perform quantization on weights of data known to be in floating-point format. The dynamic range of the activations and the quantized weights can be stored in the second memory 331 of the NPU 330.

[0134] The quantization device 320 can receive input data 710 used for inference in the neural network and quantize the received input data 710. The ALU 332 of the NPU 330 can use the quantized input data, the activation dynamic range, and the quantized weights to perform neural network operations, thereby outputting output data 780.

[0135] In another example, the Lp norm calculation device 420 can perform the Lp norm calculation and inference operations of the quantization device 320, and repeated descriptions are omitted. The dynamic range of the activation and the dynamic range of the weights can be stored in the second memory 431 of the NPU 430. The quantization device 432 of the NPU 430 can perform the quantization of the weights and the input data 710. The ALU 433 of the NPU 430 can use the quantized input data, the dynamic range of the activation, and the quantized weights to perform neural network operations.

[0136] Figure 8 This is a flowchart illustrating an example of a quantization method.

[0137] Reference Figure 8 Operations 810 to 840 can be referenced. Figure 3 The quantization device 320 described is executed. The quantization device 320 may be implemented by one or more hardware modules, one or more software modules, or various combinations thereof.

[0138] In operation 810, the quantization device 320 acquires floating-point format parameters used in the inference process performed by the neural network. In operation 820, the quantization device 320 quantizes the weights into fixed-point format based on statistical information of the weights included in the parameters. In operation 830, the quantization device 320 determines the dynamic range of the activations based on statistical information of the activations of one or more layers of the neural network configured in the parameters. In operation 840, the quantization device 320 quantizes the input data into fixed-point format based on statistical information of the input data of the neural network. Furthermore, Figures 1 to 7 The description can be applied to quantification methods.

[0139] Figure 9 This is a flowchart illustrating another example of a quantization method.

[0140] Reference Figure 9 Operations 910 to 940 can be referenced. Figure 4The Lp norm calculation device 420 described is executed. The Lp norm calculation device 420 can be implemented by one or more hardware modules, one or more software modules, or various combinations thereof.

[0141] In operation 910, the Lp norm calculation device 420 acquires parameters in floating-point format used in the inference process performed by the neural network. In operation 920, the Lp norm calculation device 420 calculates the Lp norm of the weights included in the parameters. In operation 930, the Lp norm calculation device 420 infers the Lp norm of the output activations of one or more layers of the configured neural network included in the parameters. In operation 940, the Lp norm calculation device 420 calculates the Lp norm of the input data of the neural network. Furthermore, Figures 1 to 7 The description can be applied to quantification methods.

[0142] The devices, units, modules, apparatuses, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0143] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0144] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write such instructions based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0145] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable random access read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), card storage devices (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0146] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. A method for executing a neural network, the method comprising: Retrieve parameters in floating-point format from memory, which are used in the inference process performed by the neural network. Based on the statistical information of the weights included in the parameters, the weights are quantized into a fixed-point format; The dynamic range of activation is determined based on statistical information about the activation of one or more layers of the configured neural network included in the parameters. Based on the statistical information of the input data from the neural network, the input data is quantized into a fixed-point format; The neural network is executed by a neural processor based on quantized fixed-point formatted parameters and quantized fixed-point formatted input data. The steps for determining the dynamic range of activation include: The relationship expression between the Lp norm of the output activation of each of the one or more layers and the Lp norm of the input activation is loaded from memory, and the relationship expression for each layer is determined based on the type of each layer; The Lp norm of the output activation of the one or more layers is inferred based on the relational expression and the Lp norm of the output activation of the previous layer for each of the one or more layers; and The dynamic range of the output activation of the one or more layers is determined based on the Lp norm of the output activation of the one or more layers. The input data includes at least one of image data, voice data, and text data.

2. The method according to claim 1, wherein, The steps involved in executing a neural network include: Quantized input data is fed into the neural network; Performs the operation between the quantized input activations and the quantized weights of the current layer of the neural network, and generates the output activations of the current layer; and The output activation of the current layer is quantized based on the dynamic range of activation.

3. The method according to claim 1, wherein, The steps to quantize weights into a fixed-point format include: Calculate the Lp norm of the weights; The dynamic range of the weights is determined based on the Lp norm of the weights; and Based on the dynamic range of the weights, the weights are quantized into a fixed-point format.

4. The method according to any one of claims 1 to 3, wherein, The steps to determine the dynamic range of activation include: The one or more layers are classified into a first layer that includes a normalized layer and a second layer that does not include a normalized layer; The Lp norm of the output activation of the first layer is inferred based on the normalized parameters of the first layer included in the parameters; and The Lp norm of the output activation of the second layer is inferred based on the parameters and the Lp norm of the first layer's output activation.

5. The method according to claim 4, wherein, The steps to infer the Lp norm of the output activation of the second layer include: The expression relating the Lp norm of the input activation and the Lp norm of the output activation in the second layer is calculated based on the parameters. The Lp norm of the input activations of the second layer is determined based on the Lp norm of the output activations of the first layer; and The Lp norm of the output activation of the second layer is inferred based on the Lp norm of the input activation of the second layer and the relational expression.

6. The method according to claim 4, wherein, The steps to determine the dynamic range of activation include: The dynamic range of the output activation of the first layer is determined based on the Lp norm of the first layer's output activation; and The dynamic range of the output activation of the second layer is determined based on the Lp norm of the output activation of the second layer.

7. A non-transitory computer-readable recording medium comprising one or more computer programs, said one or more computer programs including instructions for performing the method according to any one of claims 1 to 6.

8. An apparatus for executing a neural network, the apparatus comprising: The memory stores the floating-point format parameters used in the inference process performed by the neural network. The processor is configured to fetch parameters in floating-point format from memory for use in the inference process performed by the neural network. Based on the statistical information of the weights included in the parameters, the weights are quantized into a fixed-point format. Based on statistical information about the activations of one or more layers of a configured neural network included in the parameters, the dynamic range of the activations is determined. Based on the statistical information of the input data from the neural network, the input data is quantized into a fixed-point format; and Execute a neural network based on quantized fixed-point formatted parameters and quantized fixed-point formatted input data. The processor is configured as follows: The relationship expression between the Lp norm of the output activation of each of the one or more layers and the Lp norm of the input activation is loaded from memory, and the relationship expression for each layer is determined based on the type of each layer; The Lp norm of the output activation of the one or more layers is inferred based on the relational expression and the Lp norm of the output activation of the previous layer for each of the one or more layers; and The dynamic range of the output activation of the one or more layers is determined based on the Lp norm of the output activation of the one or more layers. The input data includes at least one of image data, voice data, and text data.

9. The device according to claim 8, wherein, The processor is configured as follows: Calculate the Lp norm of the weights. The dynamic range of the weights is determined based on the Lp norm of the weights. The weight-based dynamic range quantizes the weights into a fixed-point format.

10. The device according to claim 8 or 9, wherein, The processor is configured as follows: The one or more layers are classified into a first layer that includes a normalized layer and a second layer that does not include a normalized layer. The Lp norm of the output activation of the first layer is inferred based on the normalized parameters of the first layer included in the parameters. The Lp norm of the output activation of the second layer is inferred based on the parameters and the Lp norm of the first layer's output activation.

11. The device according to claim 10, wherein, The processor is configured as follows: The relationship between the Lp norm of the input activation and the Lp norm of the output activation in the second layer is calculated based on the parameters. The Lp norm of the input activation of the second layer is determined based on the Lp norm of the output activation of the first layer. The Lp norm of the output activation of the second layer is inferred based on the Lp norm of the input activation of the second layer and the relational expression.

12. The device according to claim 10, wherein, The processor is configured as follows: The dynamic range of the output activation of the first layer is determined based on the Lp norm of the first layer's output activation. The dynamic range of the output activation of the second layer is determined based on the Lp norm of the output activation of the second layer.

13. A neural processing unit, comprising: The memory includes the Lp norm of the weights, the Lp norm of the layer's output activations, and the Lp norm of the input data; The neural network quantization device is configured as follows: The dynamic range of the weights is determined based on the Lp norm of the weights. The weights are quantized into a fixed-point format based on the dynamic range of the weights. The dynamic range of the layer's output activation is determined based on the Lp norm of the layer's output activation. The dynamic range of the input data is determined based on the Lp norm of the input data. The input data is quantized into a fixed-point format based on the dynamic range of the input data. and The processor is configured to execute a neural network based on quantized fixed-point formatted parameters and quantized fixed-point formatted input data. The neural network quantization device is configured as follows: The relationship expression between the Lp norm of the output activation and the Lp norm of the input activation of the layer is loaded from memory; the layer's relationship expression is determined based on the layer type; and The Lp norm of the layer's output activation is inferred based on the relational expression and the Lp norm of the output activation of the previous layer. The input data includes at least one of image data, voice data, and text data.

14. The neural processing unit according to claim 13, wherein, The processor includes: The arithmetic logic unit is configured to perform operations between the quantized input activations and the quantized weights of the current layer of the neural network, and to generate the output activations of the current layer. The neural network quantization device is configured to quantize the output activation of the current layer based on the dynamic range of the output activation.

15. A neural network inference device for image recognition, comprising: Memory, including floating-point format parameters used in the inference process performed by the neural network; A neural network quantization device is configured to: acquire parameters from a memory, quantize weights into a fixed-point format based on statistics of weights included in the parameters, determine the dynamic range of activations based on statistics of activations of one or more layers of a configured neural network included in the parameters, and quantize input image data into a fixed-point format based on statistics of input image data of the neural network. and The neural processing unit is configured to perform inference by receiving quantized weights, activation dynamic range, and quantized input image data from a neural network quantization device. The neural network quantization device is configured as follows: The relationship expression between the Lp norm of the output activation of each of the one or more layers and the Lp norm of the input activation is loaded from memory, and the relationship expression for each layer is determined based on the type of each layer; The Lp norm of the output activation of the one or more layers is inferred based on the relational expression and the Lp norm of the output activation of the previous layer for each of the one or more layers; and The dynamic range of the output activation of the one or more layers is determined based on the Lp norm of the output activation of the one or more layers.

16. The neural network inference device according to claim 15, wherein, The neural processing unit is configured to: perform operations between the quantized input activations of the current layer of the neural network and the quantized weights of the current layer, generate the output activations of the current layer, and quantize the output activations of the current layer based on the dynamic range of the activations.

17. A processor-implemented method comprising: Retrieve parameters from the floating-point neural network model from memory; Calculate the Lp norm of the weights included in the parameters; The dynamic range of weights is determined based on the Lp norm of the weights; The dynamic range based on the weights quantizes the weights into a fixed-point format; The relationship expression between the Lp norm of the output activation and the Lp norm of the input activation of the floating-point neural network model is loaded from memory, and the relationship expression is determined based on the type of the layer of the floating-point neural network model; The predicted value of the Lp norm of the output activation is calculated based on the relational expression and the Lp norm of the input activation. The dynamic range of activation of the floating-point neural network model is determined based on the predicted values; Quantize the floating-point neural network model into a fixed-point neural network model based on dynamic range; Based on the statistical information of the input data, the input data is quantized into a fixed-point format; and Execute a fixed-point neural network model based on quantized input data. The input data includes at least one of image data, voice data, and text data.

18. The method implemented by the processor according to claim 17, wherein, Output activation is the output activation of a convolutional layer that includes a normalization layer.

19. The method implemented by the processor according to claim 17, wherein, The Lp norm includes the L2 norm.

Citation Information

Patent Citations

  • Polyimide precursor resin composition

    KR1020200123172A

  • Artificial neural network adjustment method and device

    CN110413255A

  • Neural network quantification method and system

    CN111178514A