Systems and methods for quantizing neural networks

Abstract

A system for operating a neural network, comprising a processing unit, the processing unit being used to: receive input data; input the input data to a first layer of multiple layers of the neural network; in each of multiple iterations, calculate the output of a layer of the multiple layers by the following operations: divide the activation matrix of the layer received from the previous layer into multiple sub-matrices; calculate uniform quantization of each of the multiple sub-matrices to generate multiple quantized sub-matrices; combine the multiple quantized sub-matrices to generate a quantized activation matrix; calculate the matrix product of multiple weight values ​​of the layer and the quantized activation matrix to generate the output of the layer; and predict an output value in response to the input data and / or classify a finding detected in the input data based on the output of the last layer.

Description

Background Art

[0001] In some embodiments of the invention, the invention relates to a computerized system for executing a neural network, and more particularly, but not exclusively, to a computerized system for executing a neural network using quantized data.

[0002] The term "neural network" is often used to describe computer systems that are modeled after the human brain and nervous system. A neural network typically contains a large number of processing objects that run in parallel and are arranged and connected in layers (or hierarchies). The first layer receives raw input information (one or more input values) similar to the optic nerve in human vision processing or the auditory receptor neurons in human sound processing. Each successive layer receives output from the layer or layers before it, rather than from the raw input, similar to neurons farther away from the optic nerve or auditory receptor neurons receiving signals from neurons closer to the optic nerve or auditory receptor neurons. The last layer produces the output of the neural network. For example, in computer vision, the raw input information may be one or more images, and the output may be one or more classifications of features detected in the images. In a neural network system for regression, the output may be a predicted value calculated from one or more values ​​in the raw input information. The term "depth" in deep neural networks refers to the number of layers in such a neural network. A typical calculation in a neural network layer involves summing multiple products of input values ​​and associated weights and mapping the resulting sum to an output value. Mapping the resulting sum to an output value can be a nonlinear transformation of the value sometimes called an "activation function" and indicates whether the information represented by the resulting sum is relevant for a given input or should be ignored. The term "activation matrix" refers to the matrix of output values ​​sent as input to the next layer, each output value being the resulting sum of the mapping. Training a neural network refers to the process of adjusting multiple weight values ​​of multiple layers of a neural network based on an error term between the calculated output values ​​and the expected output values ​​for a set of input values, so that in the next iteration, the calculated output values ​​will be closer to the known expected output values ​​than in the previous iteration.

[0003] The latest breakthrough in machine learning is the use of large deep neural networks deployed on powerful general-purpose processing units. Typical deep neural networks include millions of parameters and may require millions of arithmetic operations, and for applications running on limited hardware, real-time processing using deep neural networks may require digital memory resources and power resources that exceed the capabilities of many devices, such as mobile devices, some embedded devices, and some custom hardware devices. In addition, the computational latency when executing such applications on limited hardware may exceed the acceptable latency threshold. Some mobile device applications need to utilize neural networks, for example, mobile devices using computer vision. Some applications can be designed so that the device communicates with a remote service, such as a cloud-implemented service, which uses a neural network for calculation. However, these architectures require network connections that may not always be available and may cause latency constraints. Multiple weight values ​​of a neural network are stored in memory. Reducing the amount of storage required to store each weight can allow a larger neural network to be assembled on the device than a neural network using weights that require greater storage. In addition, the neural network passes activation matrices between layers. Reducing the size of the activation matrix can increase processing speed and reduce communication bandwidth between layers. The size of the activation matrix can be reduced by reducing each of the multiple activation matrix values ​​of the activation matrix.

[0004] One way to reduce the computational and storage costs of a neural network is to quantize some of the multiple weight values ​​and activation matrices of the neural network. Quantization is the process of restricting the input from a large set of original values ​​to a set of discrete quantized values. The set of discrete quantized values ​​typically has fewer values ​​than the set of large original values.

[0005] In some neural networks, a plurality of weight values ​​and a plurality of activation matrix values ​​are real numbers (non-integers). Existing methods for reducing computational effort and memory load reduce the precision of the plurality of weight values ​​and the plurality of activation matrix values ​​with the goal of allowing more efficient memory representation and more efficient computation without significantly affecting the accuracy of the neural network output. Summary of the invention

[0006] An object of the present invention is to provide a system and method for operating a neural network using quantized data.

[0007] The foregoing and other objects are achieved by the features of the independent claims. Further implementations are apparent from the dependent claims, the detailed description and the drawings.

[0008] According to a first aspect of the present invention, a system for operating a neural network includes a processing unit for: receiving input data; inputting the input data into a first layer of multiple layers of the neural network; in each of multiple iterations, calculating the output of a layer in the multiple layers by the following operations: dividing the activation matrix of the layer received from the previous layer of the neural network into multiple sub-matrices; calculating the uniform quantization of each of the multiple sub-matrices to produce multiple quantized sub-matrices; combining the multiple quantized sub-matrices to produce a quantized activation matrix; calculating the matrix product of multiple weight values ​​of the layer and the quantized activation matrix to produce the output of the layer; predicting an output value in response to the input data and / or classifying a finding detected in the input data based on the output of the last layer of the multiple layers. Dividing the activation matrix and calculating the uniform quantization of each of the multiple sub-matrices can reduce the need to use calibration data for activation quantization, or can completely avoid such a need.

[0009] According to a second aspect of the present invention, a method for operating a neural network comprises: receiving input data; inputting the input data into a first layer of a plurality of layers of the neural network; in each of a plurality of iterations, calculating the output of a layer of the plurality of layers by the following operations: dividing the activation matrix of the layer received from the previous layer of the neural network into a plurality of sub-matrices; calculating a uniform quantization of each of the plurality of sub-matrices to generate a plurality of quantized sub-matrices; combining the plurality of quantized sub-matrices to generate a quantized activation matrix; calculating a matrix product of a plurality of weight values ​​of the layer and the quantized activation matrix to generate the output of the layer; predicting an output value in response to the input data and / or classifying at least one finding detected in the input data according to the output of the last layer of the plurality of layers. Dividing the activation matrix and calculating a uniform quantization of each of the plurality of sub-matrices can reduce the need to use calibration data for activation quantization, or can avoid such a need altogether.

[0010] In combination with the first aspect and the second aspect, in a first possible implementation of the first aspect and the second aspect of the present invention, each of the multiple sub-matrices includes some elements of the multiple matrix elements of the activation matrix, so that the first continuous element sequence in the row of the sub-matrix is ​​continuous in the activation row of the activation matrix, and the second continuous element sequence in the column of the sub-matrix is ​​continuous in the activation column of the activation matrix. Using multiple sub-matrices with continuous elements of the activation matrix can reduce the dynamic range variance of the quantized activation matrix, thereby improving the accuracy of the output of the neural network. The division can be done in particular in a way that allows a minimum number of parts. The division can be achieved, for example, in a linear way that does not destroy the linearity of matrix multiplication, which is similar to the reduction of batch size to some extent. The division can be performed even if the division destroys the linearity of the dot product of the tensor multiplication basis. Therefore, data calibration may not be required, and approximate quantization can give significantly higher accuracy.

[0011] In a second possible implementation of the first aspect and the second aspect of the present invention, the processing unit is further used to: calculate multiple saturated quantized weight values ​​of one layer of the multiple layers by calculating a quantization factor, so that the quantization factor maximizes the ratio between the expected square weight value of the multiple weight values ​​of the layer and the expected square noise value, and the expected square noise value reflects the difference between the multiple weight values ​​of the layer and the multiple saturated quantized weight values ​​of the layer; when calculating the matrix product of the multiple weight values ​​of the layer and the quantization activation matrix, the multiple saturated quantized weight values ​​are used as the multiple weight values ​​of the layer. Maximizing the ratio between the signal value and the quantization noise value can reduce the difference between the multiple approximate weight values ​​associated with the multiple weight values ​​and the multiple saturated quantized weight values, thereby improving the output accuracy of the neural network. Optionally, calculating the multiple saturated quantized weight values ​​also includes: dividing each of the multiple weight values ​​by the quantization factor to generate multiple quantized weight values; saturating the multiple quantized weight values ​​within a saturation range from the identified minimum quantization value to the identified maximum quantization value to generate the multiple saturated quantized weight values. Saturating the plurality of quantized weight values ​​may reduce the dynamic range of the plurality of saturated quantized weight values ​​compared to quantizing by dividing by the quantization factor, thereby improving the output accuracy of the neural network. Optionally, calculating the expected square noise value includes: calculating a noise value for each of the plurality of weight values ​​by: dividing the weight value by a possible quantization factor to produce a possible quantized weight value; saturating the possible quantized weight value within the saturation range to produce a possible saturated quantized weight value; multiplying the possible quantization factor by the possible saturated quantized weight value to produce a noise term; subtracting the noise term from the weight value to produce a noise value. Optionally, a non-convex one-dimensional line search method is used to calculate the quantization factor. Due to the high non-convexity of the problem of identifying a preferred quantization factor, using a non-convex one-dimensional line search to identify the preferred quantization factor may better identify a global minimum or a good local minimum than some conventional convex optimization methods, thereby improving the output accuracy of the neural network.

[0012] In a third possible implementation of the first aspect and the second aspect of the present invention, calculating the multiple saturated quantized weight values ​​includes: dividing the weight matrix into multiple weight sub-matrices; calculating multiple quantized weight sub-matrices by calculating a quantized sub-matrix for each weight sub-matrix using a sub-matrix quantization factor, so that the sub-matrix quantization factor maximizes the ratio between the expected square weight value of the weight sub-matrix and the expected square sub-matrix noise value; combining the multiple quantized weight sub-matrices to generate the multiple saturated quantized weight values. Quantizing the multiple weight values ​​in multiple parts can reduce the dynamic range variation in the multiple saturated quantized weight values, thereby improving the output accuracy of the neural network. Optionally, the activation matrix is ​​arranged in a plurality of activation matrix columns; each of the plurality of sub-matrices includes some of the activation matrix columns in the plurality of activation matrix columns; the weight matrix is ​​arranged in a plurality of weight matrix rows; each of the plurality of weight sub-matrices includes some of the weight matrix rows in the plurality of weight matrix rows; calculating the matrix product of the plurality of weight values ​​of the layer with the quantized activation matrix includes: generating a plurality of partial product matrices by calculating the product of each of the plurality of sub-matrices with each of the plurality of weight sub-matrices; combining the plurality of product matrices to generate the output of the layer. Multiplying the weight sub-matrices of the plurality of weights by the sub-matrices of the activation matrix can reduce the complexity of the calculation, thereby allowing the use of cheaper computing components, and thus can promote the reduction of the cost of the device or system that executes the neural network.

[0013] In a fourth possible implementation of the first aspect and the second aspect of the present invention, the system further comprises a digital communication network adapter connected to the processing unit. Optionally, the processing unit is used to receive the input data through the digital communication network adapter. Optionally, the processing unit is also used to output the predicted output value and / or the found classification through an output device connected to the processing unit. Optionally, the output device is selected from a group of output devices consisting of: an output digital communication network adapter and a display device.

[0014] Other systems, methods, features and advantages of the present invention will be or become apparent to those skilled in the art after studying the following drawings and detailed description. It is intended that all such other systems, methods, features and advantages be included in this description, be within the scope of the present invention, and be protected by the accompanying claims.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those known to those of ordinary skill in the art. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, exemplary methods and / or materials are described below. In the event of a conflict, the patent specification, including definitions, shall prevail. In addition, the materials, methods and examples are illustrative only and are not intended to be limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Some embodiments of the present invention are described herein with reference to the accompanying drawings only as examples. Now with reference to the accompanying drawings in detail, it should be emphasized that the items shown are examples for the purpose of illustrative discussion of the embodiments of the present invention. Thus, it will be apparent to those skilled in the art how to practice the embodiments of the present invention based on the accompanying drawings.

[0017] In the attached picture:

[0018] Figure 1 is a schematic block diagram of an exemplary system according to some embodiments of the present invention.

[0019] Figure 2 is a flow chart schematically illustrating an optional operation flow for operating a neural network according to some embodiments of the present invention.

[0020] Figure 3 is a schematic diagram of an exemplary partitioned activation matrix according to some embodiments of the present invention.

[0021] Figure 4 is a flow chart schematically illustrating an optional operation flow for quantizing a plurality of weight values ​​according to some embodiments of the present invention.

[0022] Figure 5 is a flow chart schematically illustrating an optional operation flow for calculating a plurality of saturated quantization weight values ​​according to some embodiments of the present invention.

[0023] Figure 6 is a flow chart schematically illustrating an optional operational flow for calculating an expected square noise value according to some embodiments of the present invention.

[0024] Figure 7 is a schematic diagram of an exemplary matrix multiplication using partitioning according to some embodiments of the present invention. DETAILED DESCRIPTION

[0025] In some embodiments of the invention, the invention relates to a computerized system for executing a neural network, and more particularly, but not exclusively, to a computerized system for executing a neural network using quantized data.

[0026] Some existing methods for quantizing neural networks use training data to train multiple weight values ​​and / or multiple activation matrix values ​​of the neural network with low precision, such as represented by 8-bit integers or represented in binary or ternary. However, training the neural network to such low precision requires access to training data, which may need to be manipulated to adapt to one or more calibration tools used to minimize the impact on the accuracy of the neural network output. When retraining a previously trained neural network, new data may need to be used, and these new data were not visible when the neural network was originally trained. These data may not be available or the cost of obtaining them is high, resulting in an increase in the development cost of the neural network. In addition, additional training may increase the development time of the neural network, resulting in an increase in the deployment cost of the neural network. In addition, when retraining or calibrating a previously trained neural network, there may be a risk of overfitting the new data used for retraining or calibration, that is, the previously trained neural network may memorize the new training data, but the ability to generalize to new situations may be degraded. In some environments, due to the cost of obtaining new data for retraining or calibration, the new data contains much less input data sets than the original data used to train the neural network, further increasing the risk of overfitting when retraining or calibrating with new data.

[0027] Some other existing methods of quantizing neural networks use multiple quantization values ​​for multiple weight values ​​and, in addition or alternatively, use multiple quantization values ​​for multiple activation matrix values. This approach may seriously affect the accuracy of the neural network output. It is necessary to identify multiple quantization values ​​to reduce the impact on the accuracy of the neural network output.

[0028] Possible measures of the impact on the accuracy of the neural network output are the distances between multiple weight values ​​represented by W and by replacing Each of the plurality of quantized weight values ​​represented by α is multiplied by the quantization factor represented by α to calculate a plurality of approximate weight values. When N represents the number of weight values ​​in W, k represents the integer bit precision of the identifier used for quantization, and I k express The values ​​of multiple quantization weights in allow for a dynamic range such that Middle I k N represents a set of allowed quantization weight values, which has a number of elements denoted by N, each of which is I k A member of.

[0029] When a quantization factor represented by α0 is given, a plurality of quantization weight values ​​can be calculated by dividing each of the plurality of weight values ​​W by α0 It is necessary to identify the value of α0, which separates W from The difference is minimized, thus reducing the impact on the accuracy of the neural network output.

[0030] Since identifying the minimum value of α0 is a highly non-convex problem, conventional convex optimization methods cannot obtain the global minimum of α0, or even a good local minimum of α0.

[0031] Some existing solutions propose a formula to calculate α0 so that

[0032]

[0033] therefore

[0034]

[0035] ||A|| F represents the Frobenius norm of the matrix represented by A. W and Possible measures of distance between are W and The norm of the difference between

[0036] In some embodiments of the present invention, the present invention proposes to quantize multiple weights of a neural network layer without training data using a quantization factor, wherein the quantization factor maximizes the ratio between a calculated signal value and a calculated noise value, wherein the calculated signal value reflects multiple weight values, and the calculated noise value reflects the difference between multiple weight values ​​and multiple quantized weight values ​​calculated by using the quantization factor. Optionally, the calculated signal value is the expected square weight value of the multiple weight values ​​of the layer. Optionally, the calculated noise value is the expected square noise value. Compared with using training data, quantizing multiple weight values ​​without using training data can reduce development costs. Compared with quantization using other unified mappings, using a quantization factor that maximizes the ratio between the calculated signal value and the calculated noise value can improve the accuracy of the neural network output.

[0037] Optionally, quantizing the plurality of weight values ​​further comprises saturating the plurality of quantized weight values ​​within an identified saturation range. Saturating the plurality of quantized weight values ​​can reduce the variance of the dynamic range of the plurality of weights of the layer after quantization, which can promote improved accuracy of the neural network output compared to quantizing but not saturating the plurality of weight values. Optionally, more than one layer of the neural network is quantized using a quantization factor of the corresponding layer, the quantization factor maximizing a ratio between a corresponding calculated signal value and a corresponding calculated noise value, the calculated signal value reflecting the plurality of weight values ​​of the corresponding layer, the calculated noise value reflecting a difference between the plurality of weight values ​​of the corresponding layer and the plurality of quantized weight values ​​of the corresponding layer calculated using the quantization factor of the corresponding layer.

[0038] In some embodiments of the present invention, multiple weights of a layer are arranged in a weight matrix. In these embodiments, the weight matrix is ​​optionally divided into multiple weight sub-matrices, and a quantization factor is optionally calculated for each weight sub-matrix. Optionally, multiple quantized weight sub-matrices are combined to produce multiple quantized weight values. Quantizing multiple weights of a layer in multiple sub-matrices can reduce the dynamic range variation of multiple quantized weight values ​​of the layer, which may help improve the accuracy of the neural network output compared to quantizing multiple weight values ​​without division.

[0039] Additionally, in some embodiments, the present invention provides for quantizing an activation matrix of a layer while executing a neural network by partitioning the activation matrix of the layer into a plurality of activation sub-matrices and uniformly quantizing each activation sub-matrix. When the activation matrix is ​​small relative to a large number of weight values ​​in a plurality of weight values, using uniform quantization when executing the neural network may improve the accuracy of the neural network output compared to performing certain other calibration methods. Quantizing the plurality of sub-matrices may reduce the variance of the dynamic range of the plurality of quantized activation matrix values ​​of the quantized activation matrix, thereby improving the accuracy of the neural network output compared to quantizing the activation matrix without partitioning.

[0040] Optionally, multiple weights of a layer are divided into multiple weight sub-matrices, and each weight sub-matrix is ​​uniformly quantized when executing the neural network.

[0041] Optionally, in some embodiments, a neural network quantized according to the present invention is used to estimate parameters of the input data. Estimating the parameters of the input data may include classifying at least one discovery detected in the input data, for example, in computer vision, the input data may include one or more images, and the estimated parameters may be one or more feature classifications detected in the one or more images. Optionally, estimating the parameters of the input data includes predicting output values ​​in response to the input data, for example in a neural network system for regression.

[0042] Optionally, in each of the multiple layers of the neural network quantized according to the present invention, multiple weight values ​​of the layer are quantized offline using multiple weight sub-matrices before the neural network is used to estimate the parameters of the input data. While the neural network is used to estimate the parameters of the input data, the activation matrix is ​​quantized online using multiple sub-matrices. Combining online and offline quantization can promote an increase in the throughput of the neural network due to the offline quantization of multiple weights, and promote an improvement in the accuracy of the neural network output due to the online quantization of the activation matrix.

[0043] Before explaining at least one embodiment of the present invention in detail, it should be understood that the application of the present invention is not necessarily limited to the structural details and arrangements of the components and / or methods described in the following description and / or drawings and / or examples. The present invention can have other embodiments or can be practiced or implemented in various ways.

[0044] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0045] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

[0046] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or to an external computer or external storage device via a network such as the Internet, a local area network, a wide area network, and / or a wireless network.

[0047] Computer readable program instructions can be executed completely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer and partly on a remote computer or completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer by any type of network including a local area network (local area network, LAN) or a wide area network (wide area network, WAN), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (field-programmable gate array, FPGA) or programmable logic arrays (programmable logic array, PLA) can be executed by utilizing the state information of computer readable program instructions to perform computer readable program instructions to personalize electronic circuits to perform aspects of the present invention.

[0048] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams and combinations of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0049] The flow chart and block diagram in the figure illustrate the structure, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each block in the flow chart or block diagram can represent the part of a module, section or instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the function indicated in the box may not occur in the order indicated in the figure. For example, in fact, the two blocks shown in succession can be executed almost simultaneously, or sometimes, the blocks can be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart can be performed by a system based on special hardware, which performs a specific function or action, or performs a combination of special hardware and computer instructions.

[0050] refer to Figure 1 , Figure 1 1 is a schematic block diagram of an exemplary system 100 according to some embodiments of the present invention. In these embodiments, the processing unit 101 is optionally connected to a digital communication network adapter 102 for receiving input data and additionally or alternatively for outputting predicted output values ​​and / or classifications found in the input data. The processing unit 101 can be configured as any type of programmable or non-programmable circuit that performs the following described operations. The processing unit may include hardware as well as software. For example, the processing unit may include one or more processors and a transient or non-transient memory carrying a program, which, when executed by one or more processors, causes the processing unit to perform corresponding operations. Optionally, the processing unit 101 is connected to more than one digital communication network adapter for receiving input data and additionally or alternatively for outputting predicted output values ​​and / or classifications found in the input data. Optionally, the digital communication network adapter 102 is connected to a local area network (LAN), such as an Ethernet or a wireless network. Optionally, the digital communication network adapter 102 is connected to a wide area network, such as the Internet.

[0051] Optionally, the processing unit 101 is connected to a display device 103 to output predicted output values ​​and / or classifications found in the input data. Some examples of display devices are computer screens and monitors. Optionally, the processing unit 101 is connected to a non-volatile digital memory 104. Some examples of non-volatile digital memory are hard drives, network attached storage, and storage networks. Optionally, the processing unit 101 stores the data set for the neural network on the non-volatile digital memory 104. Optionally, the processing unit 101 includes one or more high performance hardware components for computing matrix products.

[0052] To operate the neural network, in some embodiments of the invention, system 100 implements the following optional methods.

[0053] refer to Figure 2 , Figure 2 is a flow chart schematically representing an optional flow of operations 200 for operating a neural network according to some embodiments of the present invention. In these embodiments, in 201, the processing unit 101 optionally receives input data through the digital communication network adapter 104, and in 204, the processing unit optionally inputs the input data to a first layer of a plurality of layers of the neural network. Optionally, the processing unit 101 calculates an output of a layer of the plurality of layers in each of a plurality of iterations. To calculate the output of the layer, in 211, the processing unit 101 optionally divides an activation matrix of the layer received from a previous layer of the neural network into a plurality of sub-matrices. Optionally, each of the plurality of sub-matrices includes some of the plurality of matrix elements of the activation matrix such that a first sequence of consecutive elements in a row of the sub-matrix is ​​consecutive in an activation row of the activation matrix, and a second sequence of consecutive elements in a column of the sub-matrix is ​​consecutive in an activation column of the activation matrix.

[0054] refer to Figure 3 , Figure 3 300A is a schematic diagram of an exemplary partitioned activation matrix 300 according to some embodiments of the present invention. In this example, 300A is a sub-matrix of the partitioned activation matrix 300. In this example, matrix elements 301, 302, 303, and 304 are continuous in the rows of the activation matrix 300, and matrix elements 301, 311, and 321 are continuous in the columns of the activation matrix 300. In addition, in this example, matrix elements 301, 302, 303, and 304 are continuous in the rows of the sub-matrix 300A, and matrix elements 301, 311, and 321 are continuous in the columns of the sub-matrix 300A.

[0055] Reference again Figure 2 , in 213, the processing unit 101 optionally calculates a unified quantization of each of the plurality of sub-matrices to produce a plurality of quantized sub-matrices, and in 215, the processing unit 101 optionally combines the plurality of quantized sub-matrices to produce a quantized activation matrix. In 217, the processing unit 101 optionally calculates a matrix product of a plurality of weight values ​​of the layer and the quantized activation matrix to produce an output of the layer. Optionally, the output of the layer is an activation matrix for a next layer of the neural network. In 220, after optionally identifying a last layer of the plurality of layers of the neural network, the processor 101 optionally predicts an output value in response to the input data in 223 based on the output of the last layer of the plurality of layers. Alternatively or additionally, in 223, the processor 101 optionally classifies a finding in the input data based on the output of the last layer of the plurality of layers.

[0056] In some embodiments of the present invention, system 100 further implements the following optional method to quantize multiple weight values ​​of a layer.

[0057] refer to Figure 4 , Figure 4 is a flow chart schematically illustrating an optional process 400 for quantizing a plurality of weight values ​​represented by W according to some embodiments of the present invention. Optionally, the plurality of weight values ​​belong to a layer of a plurality of layers of a neural network. In these embodiments, in 401, processing unit 101 calculates the weight represented by W by calculating a quantization factor The processing unit 101 may optionally calculate α by calculating a scalar value of the quantization factor represented by α. So that α and Minimize W and , thus maximizing the ratio between the signal value and the expected noise value.

[0058] refer to Figure 5 , Figure 5 is a flow chart schematically representing an optional operation 500 process for calculating multiple saturated quantization weight values ​​according to some embodiments of the present invention. In these embodiments, in 501, the processing unit 101 calculates a quantization factor so that the quantization factor maximizes the ratio between the expected signal value and the expected noise value. Optionally, the quantization factor is calculated using a non-convex one-dimensional line search method. In 502, the processing unit 101 optionally divides each of the multiple weight values ​​by the quantization factor to generate a plurality of quantized weight values, and in 505, the processing unit 101 optionally saturates the multiple quantized weight values ​​within a saturation range from an identified minimum quantization value to an identified maximum quantization value to generate the multiple saturated quantized weight values. When the multiple weight values ​​are quantized to an identified bit precision represented by k, the saturation range may be

[0059] {-2 k-1 , 2 k-1 -1}.

[0060] For example, when the plurality of weight values ​​are quantized to an integer bit precision of 4, the saturation range may be {-8, 7}. The identified integer bit precision is optionally less than or equal to 8, such as 1, 2, 3, 4, 5, or 8. Optionally, the identified integer bit precision is greater than 8, such as 16 or 24.

[0061] Reference again Figure 4, optionally, the plurality of weight values ​​are arranged in a weight matrix. Optionally, the processing unit 101 divides the weight matrix into a plurality of weight sub-matrices, and optionally calculates a plurality of quantized weight sub-matrices by calculating a quantized sub-matrix for each weight sub-matrix using a sub-matrix quantization factor, such that the sub-matrix quantization factor maximizes a ratio between an expected square weight value of the weight sub-matrix and an expected square sub-matrix noise value. The processing unit optionally combines the plurality of quantized weight sub-matrices to generate a plurality of saturated quantized weight values.

[0062] In 410, processing unit 101 optionally uses multiple saturated quantized weight values ​​as multiple weight values ​​of the layer when calculating the matrix product of the multiple weight values ​​of the layer and the quantized activation matrix, such as in 217 above.

[0063] Optionally, the signal value is an expected squared weight value of a plurality of weight values ​​of the layer. Optionally, the expected noise value is an expected squared noise value of a plurality of weight values ​​of the layer. Figure 6 , Figure 6 6 is a flow chart schematically representing an optional flow of operations 600 for calculating square noise values ​​for corresponding weight values ​​according to some embodiments of the present invention. In these embodiments, in 601, the processing unit 101 divides the weight value by a possible quantization factor to generate a possible quantized weight value, and in 602, saturates the possible quantized weight value within a saturation range to generate a possible saturated quantized weight value. In 603, the processing unit 101 optionally multiplies the possible quantization factor by the possible saturated quantized weight value to generate a noise term, and in 604, the processing unit 101 optionally subtracts the noise term from the weight value to generate a noise value.

[0064] In some embodiments of the present invention, the processing unit 101 calculates the matrix product of the plurality of weight values ​​of the layer and the quantized activation matrix of the layer by multiplying the plurality of sub-matrices. Figure 7 , Figure 7 7 is a schematic diagram of an exemplary matrix multiplication 700 using partitioning according to some embodiments of the present invention. A plurality of weight values ​​of a layer may be arranged in a weight matrix 701 comprising a plurality of weight sub-matrices. Each of the plurality of weight sub-matrices may optionally be arranged in a plurality of weight matrix rows 711. A plurality of activation values ​​may be arranged in an activation matrix 702 comprising a plurality of sub-matrices. Each of the plurality of sub-matrices may optionally be arranged in a plurality of matrix columns 712. Optionally, the processing unit 101 generates a plurality of product matrices 720 by calculating the product of each sub-matrix in the plurality of sub-matrices 712 with each weight sub-matrix in the plurality of weight sub-matrices 711. Optionally, when the plurality of weight sub-matrices 711 are composed of W i The weight submatrix represented by α i The weight quantization factor represented by When the product of the quantized weight sub-matrices represented by And when the plurality of sub-matrices 712 are composed of X j The submatrix represented by β j The activation quantization factor represented by When the product of the uniform quantized submatrices represented by W i and X j The product of is equal to:

[0065]

[0066] Optionally, α i and β j is a real number. Optionally, calculate W i and X j The product includes the calculation of α i and β j The factor product of and , and the product matrix of the factor products and the quantization matrix product. Optionally, the processing unit 101 uses one or more high-performance hardware components to calculate the quantization matrix product. Optionally, calculating the product of a weight sub-matrix and a sub-matrix includes calculating the product of the quantization matrix product and the factor product calculated by multiplying the weight quantization factor by the activation quantization factor. Optionally, the factor product is a real number.

[0067] Processing unit 101 optionally combines multiple product matrices 720 to produce the output of the layer.

[0068] The description of various embodiments of the present invention is for illustrative purposes only, and these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles, practical applications, or technological advances of the present embodiments, or to enable other persons skilled in the art to understand the embodiments disclosed herein, compared to the technology available on the market.

[0069] It is expected that during the life of the patent expiring on this application, many related square wave noise values ​​will be developed, and the scope of the term "square wave noise value" is intended to include all such new technologies.

[0070] As used herein, the term "about" refers to ± 10%.

[0071] The terms "include", "comprising", "having" and variations thereof mean "including but not limited to". This term encompasses the terms "consisting of" and "consisting essentially of.

[0072] The phrase "consisting essentially of means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0073] As used herein, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including components thereof.

[0074] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any “exemplary” embodiment is not necessarily to be construed as preferred or advantageous over other embodiments, and / or does not exclude the incorporation of features of other embodiments.

[0075] The word “optionally” is used herein to mean “provided in some embodiments and not provided in other embodiments.” Any particular embodiment of the present invention may include multiple “optional” features, unless such features are mutually incompatible.

[0076] Throughout this application, various embodiments of the present invention can be presented in range format.It should be understood that the description of range format is only for convenience and simplicity, and should not be interpreted as a fixed limitation to the scope of the invention.Therefore, the description of the scope should be considered to have specifically disclosed all possible sub-ranges and individual numerical values ​​in the scope.For example, the description of the scope such as from 1 to 6 should be considered to have specifically disclosed sub-ranges, such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the single numerical value in the scope, such as, 1,2,3,4,5 and 6.No matter how the breadth of the scope, this is applicable.

[0077] When a numerical range is indicated herein, it is intended to include any number (fractional or integer) listed within the indicated range. The phrases "within the range of the first indicated number and the second indicated number" and "from the first indicated number to the second indicated number" are used interchangeably herein to include the first and second indicated numbers and all fractions and integers between the two.

[0078] It should be understood that certain features of the present invention described in the context of a separate embodiment for the sake of brevity may also be provided in combination in a single embodiment. Conversely, the various features of the present invention described in the context of a single embodiment for the sake of brevity may also be provided individually or in any suitable sub-combination or as any suitable other embodiment of the present invention. Certain features described in the context of various embodiments are not considered essential features of those embodiments, unless the embodiment described is invalid without these elements.

[0079] All publications, patents and patent specifications mentioned in this specification are hereby incorporated by reference into this specification, and likewise, each individual publication, patent or patent specification is specifically and individually incorporated herein. In addition, the citation or identification of any reference in this application shall not be construed as allowing such reference to take precedence over the present invention in the prior art. To the extent that section headings are used, they should not be construed as necessarily limiting.

Claims

1. A system for operating a neural network, characterized in that The invention comprises a processing unit, wherein the processing unit is configured to: receiving input data; the input data comprising one or more images; Inputting the input data into a first layer of a plurality of layers of a neural network; In each of a plurality of iterations, an output of a layer in the plurality of layers is calculated by: dividing an activation matrix of the layer received from a previous layer of the neural network into a plurality of sub-matrices; calculating a unified quantization of each of the plurality of sub-matrices to generate a plurality of quantized sub-matrices; combining the plurality of quantized sub-matrices to generate a quantized activation matrix; Calculating a matrix product of a plurality of weight values ​​of the layer and the quantized activation matrix to generate the output of the layer; predicting an output value in response to the input data and / or classifying a finding detected in the input data based on an output of a last layer of the plurality of layers; calculating a plurality of saturated quantized weight values ​​for a layer of the plurality of layers by calculating a quantization factor such that the quantization factor maximizes a ratio between an expected squared weight value for the plurality of weight values ​​for the layer and an expected squared noise value, the expected squared noise value reflecting a difference between the plurality of weight values ​​for the layer and the plurality of saturated quantized weight values ​​for the layer; When calculating the matrix product of the plurality of weight values ​​of the layer and the quantized activation matrix, using the plurality of saturated quantized weight values ​​as the plurality of weight values ​​of the layer; The processing unit is further configured to output the predicted output value and / or the discovered classification via an output device connected to the processing unit.

2. The system according to claim 1, characterized in that Each submatrix of the multiple submatrices includes some elements of the multiple matrix elements of the activation matrix, so that a first continuous sequence of elements in the rows of the submatrix is ​​continuous in the activation rows of the activation matrix, and a second continuous sequence of elements in the columns of the submatrix is ​​continuous in the activation columns of the activation matrix.

3. The system according to claim 1, characterized in that Calculating the multiple saturated quantization weight values ​​also includes: dividing each of the plurality of weight values ​​by the quantization factor to generate a plurality of quantized weight values; The plurality of quantization weight values ​​are saturated within a saturation range from an identified minimum quantization value to an identified maximum quantization value to produce the plurality of saturated quantization weight values.

4. The system according to claim 3, characterized in that Calculating the expected square noise value includes calculating a plurality of noise values ​​by calculating a noise value for each of the plurality of weight values ​​by: Dividing the weight value by a possible quantization factor to generate a possible quantized weight value; saturating the possible quantization weight values ​​within the saturation range to generate possible saturated quantization weight values; multiplying the possible quantization factor by the possible saturation quantization weight value to generate a noise term; The noise term is subtracted from the weight value to produce a noise value.

5. The system according to any one of claims 1, 3 and 4, characterized in that The quantization factors are calculated using a non-convex one-dimensional line search method.

6. The system according to any one of claims 1, 3 and 4, characterized in that The plurality of weight values ​​are arranged in a weight matrix; The calculation of the multiple saturated quantization weight values ​​comprises: Dividing the weight matrix into a plurality of weight sub-matrices; calculating a plurality of quantized weight sub-matrices by calculating a quantized sub-matrix for each weight sub-matrix using a sub-matrix quantization factor such that the sub-matrix quantization factor maximizes a ratio between expected square weight values ​​of the weight sub-matrix and an expected square sub-matrix noise value; The plurality of quantization weight sub-matrices are combined to generate the plurality of saturated quantization weight values.

7. The system according to claim 6, characterized in that wherein the activation matrix is ​​arranged in a plurality of activation matrix columns; wherein each of the plurality of sub-matrices includes some of the plurality of activation matrix columns; wherein the weight matrix is ​​arranged in a plurality of weight matrix rows; wherein each of the plurality of weight sub-matrices includes some of the plurality of weight matrix rows; Wherein calculating the matrix product of the plurality of weight values ​​of the layer and the quantized activation matrix comprises: generating a plurality of product matrices by calculating the product of each submatrix of the plurality of submatrices and each weight submatrix of the plurality of weight submatrices; The plurality of product matrices are combined to produce the output of the layer.

8. The system according to any one of claims 1 to 4, characterized in that: Also included is a digital communications network adapter connected to the processing unit.

9. The system according to claim 8, characterized in that The processing unit is configured to receive the input data through the digital communication network adapter.

10. The system according to any one of claims 1 to 4, characterized in that: The output device is selected from the group consisting of: an output digital communications network adapter and a display device.

11. A method for operating a neural network, characterized in that include: receiving input data; the input data comprising one or more images; Inputting the input data into a first layer of a plurality of layers of a neural network; In each of a plurality of iterations, an output of a layer in the plurality of layers is calculated by: dividing an activation matrix of the layer received from a previous layer of the neural network into a plurality of sub-matrices; calculating a unified quantization of each of the plurality of sub-matrices to generate a plurality of quantized sub-matrices; combining the plurality of quantized sub-matrices to generate a quantized activation matrix; Calculating a matrix product of a plurality of weight values ​​of the layer and the quantized activation matrix to generate the output of the layer; predicting an output value in response to the input data and / or classifying at least one finding detected in the input data based on an output of a last layer of the plurality of layers; calculating a plurality of saturated quantized weight values ​​for a layer of the plurality of layers by calculating a quantization factor such that the quantization factor maximizes a ratio between an expected squared weight value for the plurality of weight values ​​for the layer and an expected squared noise value, the expected squared noise value reflecting a difference between the plurality of weight values ​​for the layer and the plurality of saturated quantized weight values ​​for the layer; When calculating the matrix product of the plurality of weight values ​​of the layer and the quantized activation matrix, using the plurality of saturated quantized weight values ​​as the plurality of weight values ​​of the layer; The predicted output value and / or the discovered classification is output.

12. The method according to claim 11, characterized in that Each submatrix of the multiple submatrices includes some elements of the multiple matrix elements of the activation matrix, so that a first continuous sequence of elements in the rows of the submatrix is ​​continuous in the activation rows of the activation matrix, and a second continuous sequence of elements in the columns of the submatrix is ​​continuous in the activation columns of the activation matrix.

13. The method according to claim 11, characterized in that The quantization factors are calculated using a non-convex one-dimensional line search method.

14. The method according to claim 11, characterized in that The plurality of weight values ​​are arranged in a weight matrix; The calculation of the multiple saturated quantization weight values ​​comprises: Dividing the weight matrix into a plurality of weight sub-matrices; calculating a plurality of quantized weight sub-matrices by calculating a quantized sub-matrix for each weight sub-matrix using a sub-matrix quantization factor such that the sub-matrix quantization factor maximizes a ratio between expected square weight values ​​of the weight sub-matrix and an expected square sub-matrix noise value; The plurality of quantization weight sub-matrices are combined to generate the plurality of saturated quantization weight values.

Images