A Reasoning Method Based on Deep Learning Model

Abstract

The present invention discloses a reasoning method based on a deep learning model. The deep learning model includes a plurality of model layers connected in sequence, each of which has its weight matrix. The method includes the steps of: receiving an input matrix to be reasoned and calculated by the deep learning model so as to quantize each input matrix unit in the input matrix into an integer value having a first bit number; performing reasoning using each model layer, wherein the first layer of the plurality of model layers connected in sequence receives the quantized input matrix, performs matrix operations on the input matrix using its weight matrix to generate an output matrix, requantizes each matrix unit in the output matrix into an integer value having a first bit number, and sends the output matrix to the next layer model for further operation; and obtaining a reasoning result based on the output matrix generated by the last layer of the plurality of model layers connected in sequence.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to the field of deep learning and quantizing a deep learning model to accelerate reasoning. Background Art

[0002] With the rapid development of artificial intelligence in recent years, artificial intelligence, especially deep learning algorithms, has been widely used in fields such as visual images, speech recognition and natural language processing.

[0003] With the popularization of edge intelligence, more and more artificial intelligence computing needs to be performed on edge devices to bring new interactive experiences. However, compared with servers, edge devices have lower computing power and less memory. Therefore, how to optimize deep learning algorithms that consume a lot of computing power and storage so that they can be deployed on edge devices is a hot issue in the field of artificial intelligence.

[0004] One method to optimize deep learning models is to quantize the parameters sampled in the model, that is, to quantize the parameters such as floating point numbers into integer format. Compared with processing floating point operations, the processor in the end device, whether it is a GPU or other processor, can process integer operations more quickly, so the deep learning model deployed on the end device can be accelerated by parameter quantization, thereby speeding up the inference process.

[0005] Therefore, as mentioned above, a new deep learning model quantization scheme is needed that can more efficiently utilize register resources and more efficiently perform reasoning based on deep learning models. Summary of the invention

[0006] To this end, the present invention provides a reasoning method and a computing system based on a deep learning model, in an effort to solve or at least alleviate at least one of the above problems.

[0007] According to one aspect of the present invention, a method for reasoning based on a deep learning model is provided. The deep learning model includes a plurality of model layers connected in sequence, each of which has its weight matrix. Each weight matrix unit in the weight matrix is ​​quantized to an integer value having a first bit number and stored with a second bit number greater than the first bit number. The method includes the steps of: receiving an input matrix to be inferred by the deep learning model so as to quantize each input matrix unit in the input matrix to an integer value having a first bit number; performing reasoning using each model layer, wherein the first layer of the plurality of model layers connected in sequence receives the quantized input matrix, performs matrix operations on the input matrix using its weight matrix to generate an output matrix, requantizes each matrix unit in the output matrix to an integer value having a first bit number, and sends the output matrix to the next layer of the model for further operation; and obtaining the reasoning result based on the output matrix generated by the last layer of the plurality of model layers connected in sequence. The matrix operation includes multiplication and accumulation operations on the input matrix units of the input matrix and the weight matrix units of the weight matrix, and the matrix units of the output matrix store the results of the multiplication and accumulation operations with a third bit number, the third bit number is twice the second bit number, and the first bit number is set so that when each model layer performs multiplication and accumulation operations on the corresponding matrix units of the weight matrix and the input matrix with the second bit number, the multiplication and accumulation result is less than the maximum value corresponding to the third bit number.

[0008] Optionally, in the inference method according to the present invention, the method for performing a multiplication-accumulation operation includes the steps of: loading a predetermined number of input matrix unit values ​​in a first register, wherein the first register has a fourth bit length, and the fourth bit length is an integer multiple of the third bit length, and the predetermined number is the ratio of the fourth bit to the second bit; loading a predetermined number of weight matrix unit values ​​in a second register, and the second register has the same length as the first register; and performing a multiplication-accumulation operation so as to multiply the input matrix unit value in the first register and the weight matrix unit value at the corresponding position of the second register to obtain a multiplication result of a third bit number, and adding the multiplication result and the value at the corresponding position in the third register with a length of the third bit number, and updating the value at the corresponding position of the third register to the addition result.

[0009] Optionally, in the inference method according to the present invention, the length of the third register is twice the length of the first register.

[0010] Optionally, in the inference method according to the present invention, the length of the third register is the same as that of the first register, and the multiplication-accumulation result corresponding to the cell value of the low-order area of ​​the first register is stored in the third register, and the step of performing the multiplication-accumulation operation includes: multiplying the input matrix cell value in the high-order area of ​​the first register and the weight matrix cell value at the corresponding position of the second register to obtain a multiplication result of a third bit number, and adding the multiplication result and the value at the corresponding position in the fourth register with a length of the third bit number, and updating the value at the corresponding position of the fourth register to the addition result, wherein the length of the fourth register is the same as that of the first register.

[0011] Optionally, in the reasoning method according to the present invention, each model layer also includes one or more arithmetic operations, each model layer includes a mapping table established for each arithmetic operation, the arithmetic mapping table establishes a mapping relationship between the quantized operands and the quantized operation results, wherein the operand has a first bit or a second bit, and the operation result has a second bit; and the step of performing reasoning using each model layer includes: when performing an arithmetic operation, querying the mapping table corresponding to the arithmetic operation to obtain the operation result corresponding to the operand.

[0012] Optionally, in the inference method according to the present invention, the operands of the arithmetic operation include results of multiplication-accumulation operations.

[0013] Optionally, in the inference method according to the present invention, the algorithm operation includes an activation function operation.

[0014] Optionally, in the inference method according to the present invention, each model layer also includes a bias value corresponding to the weight matrix unit in the weight matrix of each layer, and each model layer includes a constant value of the third bit obtained by quantizing the bias value; and the step of performing inference using each model layer includes: before requantizing the output matrix of each model layer, summing the output matrix units of each output matrix and the corresponding constant values, and then requantizing the summed result.

[0015] Optionally, in the inference method according to the present invention, the first bit number is one, two or three less than the second bit number.

[0016] Optionally, in the reasoning method according to the present invention, the first bit number is 6 and the second bit number is 8.

[0017] Optionally, in the inference method according to the present invention, the first bit number is 5 or 7, and the second bit number is 8.

[0018] Optionally, in the inference method according to the present invention, the length of the first and second registers is 64 bits.

[0019] Optionally, in the inference method according to the present invention, the length of the first and second registers is 128 bits.

[0020] Optionally, in the inference method according to the present invention, the deep learning model includes a convolutional neural network, and the matrix operation for performing inference in each model layer includes a convolution operation of an input matrix unit and a weight matrix unit.

[0021] According to another aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for being executed by the at least one processor, and the program instructions include instructions for executing any of the methods described above.

[0022] According to another aspect of the present invention, a storage medium is provided, wherein program instructions are stored in the storage medium. When the program instructions are read out from the storage medium and executed in a computing device, the method according to the present invention is executed.

[0023] According to the solution of the present invention, before using the deep learning model, the parameters in each model layer in the model are first integer quantized, and when quantizing the parameters, the possibility of overflow when the parameters are subsequently used for multiplication and accumulation operations is fully considered. The parameters are not quantized to the number of bits used in integer data operations (for example, 8 bits), but are quantized to a number of bits smaller than this number of bits, for example, 6 bits. In this way, during the subsequent reasoning using the quantized deep learning model, when performing multiplication and accumulation operations, using 16 bits to store the results of the multiplication and accumulation operations will not cause overflow, so there is no need to use 32 bits to store the multiplication and accumulation results, thereby reducing the space occupied by registers and improving the utilization rate of registers in the inference device.

[0024] In addition, according to the solution of the present invention, when performing integer multiplication operations during the reasoning process, there is no need to first expand the integer parameters to a higher bit position (for example, 16 bits), and 8-bit multiplication operations can be performed directly, which speeds up the multiplication operation.

[0025] In addition, according to the solution of the present invention, when the solution of the present invention is used in a convolutional neural network, especially when performing convolution calculations, since the same operation as before can be completed with almost half of the registers, a larger microkernel can be used, and more input unit values ​​and weight values ​​can be loaded into the L2 cache at one time for processing, thereby reducing the frequency of loading data from the memory and improving the reasoning execution speed.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0028] Figure 1A A schematic diagram showing an example computer system 9100 according to some embodiments of the present invention;

[0029] Figure 1B A schematic diagram of a deep neural network as a machine learning model 9120 according to some embodiments of the present invention is shown;

[0030] Figure 2A A schematic diagram of a computing device 200 according to an embodiment of the present invention is shown;

[0031] Figure 2B The implementation of an application including artificial intelligence in a computing device 200 is shown in the form of a software stack;

[0032] Figure 2C 1 shows an implementation of executing instructions related to a deep learning model in a processor 9204 of a computing device 200 according to one embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a process of a reasoning method based on a deep learning model according to an embodiment of the present invention is shown;

[0034] Figure 4 A schematic diagram showing a flow chart of executing a multiply-accumulate instruction in a processor according to an embodiment of the present invention; and

[0035] Figure 5 A processing example in inference processing using a convolutional neural network according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0036] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0037] Figure 1A A block diagram of an example computing system 9100 is depicted, according to an example embodiment of the present disclosure. System 9100 includes a user computing device 9110, a server computing system 9130, and a training computing system 9150 communicatively coupled via a network 9180.

[0038] The user computing device 9110 may be any type of computing device, including but not limited to, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (a smart phone or tablet), a game console or controller, a wearable computing device, an embedded computing device, an edge computing device, or any other type of computing device. The user computing device 9110 may be deployed at the user site as a terminal smart device and interact with the user to process user input.

[0039] The user computing device 9110 may store or include one or more machine learning models 9120. The machine learning model 9120 may be designed to perform various tasks, such as image classification, target detection, speech recognition, machine translation, content filtering, and the like. The machine learning model 9120 may be a machine learning model such as a neural network (e.g., a deep neural network) or other types of machine learning models including nonlinear models and / or linear models. Examples of machine learning models 9120 include, but are not limited to, various types of deep neural networks (DNNs), such as feedforward neural networks, recurrent neural networks (RNNs, e.g., long short-term memory recurrent neural networks (LSTMs), transformer neural networks (Transformers) including or not including attention mechanisms), convolutional neural networks (CNNs), or other forms of neural networks. The machine learning model 9120 may include one machine learning model, or may be a combination of multiple machine learning models.

[0040] Figure 1B 9120 according to some embodiments. A neural network has a layered architecture, with each network layer having one or more processing nodes (called neurons or filters) for processing. In a deep neural network, the output of the previous layer after processing is the input of the next layer, where the first layer in the architecture receives the network input for processing, and the output of the last layer is provided as the network output. Figure 1BAs shown, the machine learning model 9120 includes network layers 9124, 9126, etc., wherein network layer 9124 receives network input and network layer 9126 provides network output.

[0041] In a deep neural network, the main processing operations within the network are interwoven linear and nonlinear transformations. These processes are distributed across various processing nodes. Figure 1B Also shown is an enlarged view of a node 9121 in the model 9120. The node 9121 receives multiple input values ​​a1, a2, a3, etc., and processes the input values ​​based on corresponding processing parameters (such as weights w1, w2, w3, etc.) to generate an output z. The node 171 can be designed to process the input using an activation function, which can be expressed as:

[0042] z= σ(w T α+b) (1)

[0043] where α∈R N represents the input vector of node 9121 (including elements a1, a2, a3, etc.); w∈R N represents the weight vector (including elements w1, w2, w3, etc.) in the processing parameters used by node 9121, each weight is used to weight the corresponding input; N represents the number of input values; b∈R N Represents the bias vector (including elements b1, b2, b3, etc.) in the processing parameters used by node 9121, each bias is used to bias the corresponding input and weighted result; σ() represents the activation function used by node 9121, and the activation function can be a linear function or a nonlinear function. Common activation functions in neural networks include sigmoid function, ReLu function, tanh function, maxout function, etc. The output of node 9121 can also be called an activation value. Depending on the network design, the output (i.e., activation value) of each network layer can be provided to one, multiple or all nodes of the next layer as input.

[0044] Each network layer in the machine learning model 9120 may include one or more nodes 9121. When the processing in the machine learning model 9120 is viewed in terms of network layers, the processing of each network layer may be similarly expressed in the form of formula (1) or formula (2), where a represents the input vector of the network layer and w represents the weight of the network layer.

[0045] It should be understood that Figure 1B The architecture of the machine learning model shown and the number of network layers and processing nodes therein are all schematic. In different applications, the machine learning model can be designed to have other architectures as needed.

[0046] Continue to refer Figure 1AIn some implementations, the user computing device 9110 may receive the machine learning model 9120 from the server computing system 130 via the network 9180, store it in the memory of the user computing device, and use or implement it by an application in the user computing device.

[0047] In other implementations, the user computing device 9110 may call a machine learning module 9140 stored and implemented in the server computing system 9130. For example, the machine learning model 9140 may be implemented by the server computing system 9130 as part of a web service, so that the user computing device 9110 may call the machine learning model 9140 implemented as a web service, for example, over the network 9180 and according to a client-server relationship. Thus, the machine learning module that can be used at the user computing device 102 includes the machine learning model 9120 stored and implemented at the user computing device 9110 and / or the machine learning model 9140 stored and implemented at the server computing system 9130.

[0048] The user computing device 9110 may also include one or more user input components 9122 that receive user input. For example, the user input component 9122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, a camera, or other devices through which a user can provide user input.

[0049] The server computing system 9130 may include one or more server computing devices. Where the server computing system 9130 includes multiple server computing devices, the server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0050] As described above, the server computing system 9130 may store or include one or more machine learning models 9140. Similar to the machine learning model 9120, the machine learning model 9140 may be designed to perform various tasks, such as image classification, target detection, speech recognition, machine translation, content filtering, and the like. The model 9140 may include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recursive neural networks, and convolutional neural networks.

[0051] The user computing device 9110 and / or the server computing system 9130 may train the models 9120 and / or 9140 via interaction with a training computing system 9150 communicatively coupled via a network 9180. The training computing system 9150 may be separate from the server computing system 9130, or may be part of the server computing system 9130.

[0052] Similar to server computing system 9130, training computing system 9150 may include or otherwise be implemented by one or more server computing devices.

[0053] The training computing system 9150 may include a model trainer 9160 that trains the machine learning models 9120 and / or 9140 stored at the user computing device 9110 and / or the server computing system 9130 using various training or learning techniques such as, for example, back propagation of errors. In some implementations, performing back propagation of errors may include performing truncated backpropagation through time. The model trainer 9160 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0054] Specifically, the model trainer 9160 can train the machine learning model 9120 and / or 9140 based on the set of training data 9162. The training data 9162 can include multiple different training data sets, each of which is, for example, respectively helpful in training the machine learning model 9120 and / or 9140 to perform multiple different tasks. For example, the training data set includes a data set that helps the machine learning model 9120 and / or 9140 perform object detection, object recognition, object segmentation, image classification, and / or other tasks.

[0055] In some implementations, if the user has explicitly consented, the training examples may be provided by the user computing device 9110. Thus, in such implementations, the model 9120 provided to the user computing device 9110 may be trained by the training computing system 9150 on user-specific data received from the user computing device 9110. In some cases, this process may be referred to as personalizing the model.

[0056] In addition, in some implementations, the model trainer 9160 may modify the machine learning model 9140 in the server computing system 9130 to obtain a machine learning model 9120 suitable for use in the user computing device 9110. These modifications include, for example, reducing the number of various parameters in the model, storing parameter values ​​with less precision, etc., so that the trained machine learning models 9120 and / or 9140 are suitable for running in view of the different processing performances of the server computing system 9130 and the user computing device 9110.

[0057] The model trainer 9160 includes computer logic for providing the desired functionality. The model trainer 9160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, the model trainer 9160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 9160 includes a collection of one or more computer executable instructions stored in a tangible computer-readable storage medium such as RAM, a hard disk, or an optical or magnetic medium. In some implementations, the model trainer 9160 can be copied and / or distributed across multiple different devices.

[0058] The network 9180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications through the network 9180 can be carried via any type of wired and / or wireless connection, using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML, and JSON), and / or protection schemes (e.g., VPN, HTTPS, SSL).

[0059] Figure 2A An example computing system that can be used to implement the present invention is shown. The present invention can also be implemented using other computing systems. For example, in some implementations, the user computing device 9110 can include a model trainer 9160 and a training data set 9162. In such an implementation, the model 9120 can be trained and used locally on the user computing device 9110. In some such implementations, the user computing device 9110 can implement the model trainer 9160 to personalize the model 120 based on user-specific data.

[0060] Figure 1A The user computing device 9110, the server computing system 9130, and the training computing system 9150 in the illustrated example computing system 9100 may all be implemented by a computing device 9200 as described below. Figure 2AA schematic diagram of a computing device 9200 according to one embodiment of the present invention is shown.

[0061] like Figure 2A As shown, in a basic configuration 9202, the computing device 9200 typically includes a system memory 9206 and one or more processors 9204. A memory bus 9208 may be used for communication between the processor 9204 and the system memory 9206.

[0062] Depending on the desired configuration, the processor 9204 can be any type of processing, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), a graphics processor (GPU), a neural network processor (NPU), or any combination thereof. The processor 9204 may include one or more levels of cache such as a first-level cache 9210 and a second-level cache 9212, a processor core 9214, and registers 9216. The example processor core 9214 may include an arithmetic logic unit (ALU), a floating point unit (FPU), or any combination thereof. An example memory controller 9218 may be used with the processor 9204, or in some implementations, the memory controller 9218 may be an internal part of the processor 9204.

[0063] Depending on the desired configuration, the system memory 9206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 9206 may include an operating system 9220, one or more applications 9222, and data 9224. In some embodiments, one or more processors 9204 execute program instructions in the application and process data 9224 to implement the functions of the application 9222.

[0064] The computing device 9200 may also include an interface bus 9240. The interface bus 9240 enables communication from various interface devices (e.g., output devices 9242, peripheral interfaces 9244, and communication devices 9246) to the basic configuration 9202 via the bus / interface controller 9230. Example output devices 9242 include a graphics processing unit 9248 and an audio processing unit 9250. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 9252. Example peripheral interfaces 9244 may include a serial interface controller 9254 and a parallel interface controller 9256, which can be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, video input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 9258. An example communication device 9246 may include a network controller 9260 , which may be arranged to facilitate communications with one or more other computing devices 262 via one or more communication ports 9264 over a network communication link (eg, over the network 9180 ).

[0065] The computing device 9200 may also include a storage interface bus 9234. The storage interface bus 9234 enables communication from a storage device 9232 (e.g., a removable storage 9236 and a non-removable storage 9238) via the bus / interface controller 9230 to the basic configuration 9202. At least a portion of the operating system 9220, the application 9222, and the data 9224 may be stored on the removable storage 9236 and / or the non-removable storage 9238, and when the computing device 9200 is powered on or the application 9222 is to be executed, it is loaded into the system memory 9206 via the storage interface bus 9234 and executed by one or more processors 9204.

[0066] In some implementations, when utilizing the computing device 9200 to implement the server computing system 9130 and / or the training computing system 9150 , the computing device 9200 may not include the output device 9242 and the peripheral interface 9244 so that the computing device 9200 is dedicated to reasoning and training of the machine learning model 9140 .

[0067] The application 9222 is executed on the operating system 9220, that is, the operating system 9220 provides various interfaces for operating hardware devices (e.g., storage device 9232, output device 9242, peripheral interface 9244, and communication device), and also provides an environment for application context management (e.g., storage space management and allocation, interrupt processing, process management, etc.). The application 9222 uses the interface and environment provided by the operating system 9220 to control the computing device 9200 to perform corresponding functions. In some implementations, some applications 9222 also provide interfaces. In this way, other applications 9222 can call these interfaces to implement functions.

[0068] Figure 2B The implementation of application 9222 in computing device 9200 is shown in the form of a software stack. Figure 2B As shown, the application using the machine learning model 9120 / 9140 for reasoning is called a machine learning application 9602. As described above, the machine learning application 9602 can implement any type of machine intelligence, including but not limited to: image recognition, mapping and positioning, autonomous navigation, speech synthesis, medical imaging or language translation.

[0069] The machine learning framework 9604 can provide a library of machine learning operation units. The machine learning operation unit is the basic operation that the machine learning algorithm usually performs. When the machine learning model 9120 / 9140 is designed and run based on the machine learning framework 9604, the operation unit provided by the machine learning framework 9604 can be used to perform the necessary calculations. Exemplary operation units include tensor convolution, activation function and pooling, which are calculation operations performed when training convolutional neural networks (CNN). The machine learning framework 9604 can also provide operation units for implementing basic linear algebra subroutines performed by many machine learning algorithms, such as matrix and vector operations. Using the machine learning framework 9604 can significantly simplify the development process of the machine learning model and improve its execution efficiency. For example, in the absence of the machine learning framework 9604, the developer of the machine learning model needs to create and optimize the main calculation logic associated with the machine learning algorithm from scratch, and then re-optimize the calculation logic when a new parallel processor is developed, which requires a lot of time and effort. The machine learning framework 9604 known on the market includes, for example, Google's tensorflow and Facebook's pytorch. The present invention is not limited to the specific machine learning framework 9604. Any machine learning framework that is convenient for implementing a machine learning model is within the protection scope of the present invention.

[0070] The machine learning framework 9604 can process the input data received from the machine learning application 9602 and generate appropriate outputs to the computing framework 9606. The computing framework 9606 can abstract the underlying instructions provided to the underlying hardware driver 9608 so that the machine learning framework 9604 can utilize the hardware acceleration function provided by the hardware 9610 (e.g., the processor 9204 in 2A) without being very familiar with the architecture of the hardware 9610. In addition, the computing framework 9606 can implement hardware acceleration for the machine learning framework 9604 across multiple types and generations of hardware 9610. For example, currently known computing frameworks 9606 include CUDA of Nvidia Corporation, etc. The present invention is not limited to a specific computing framework 9606, and any computing framework that can abstract the instructions of the hardware driver 9608 and utilize the hardware acceleration function of the hardware 9610 is within the protection scope of the present invention.

[0071] According to one embodiment, the underlying hardware driver 9608 may be included in the operating system 9220, and the computing framework 9606 and the machine learning framework 9604 may be implemented as separate applications or incorporated into each application 9222. All such configurations are exemplary and are within the scope of protection of the present invention.

[0072] Figure 2CAn implementation of executing instructions in a neural network in a processor 9204 is shown. The processor 9204 includes one or more processor cores 9214. Each processor core includes an FPU 9278A and an ALU 9278B. In the embodiments described herein, the processor core 9214 can execute instructions of various data types. The FPU performs operations related to floating point numbers, and the ALU can perform operations related to integer numbers. Operands can be stored in the first input register, the second input register, and the third input register 9271a-c. These operands can be used as inputs to perform operations defined by an opcode (e.g., opcode 9279). For example, opcode 9279 can specify a multiplication and addition operation. When opcode 9279 indicates a multiplication and addition operation of a floating point number, the first, second, and third operands stored in the first to third registers are floating point numbers. The FPU 9278A can multiply the floating point primes stored in the first and second registers, add the product of the multiplication to the third operand, and update the value in the third operand. Similarly, when the opcode 9279 indicates an integer multiplication and addition operation, the first, second and third operands stored in the first to third registers are integers. The ALU 9278B may perform an integer multiplication and addition operation, wherein the integers stored in the first and second registers are multiplied, and the product of the multiplication is added to the third operand and the value in the third operand is updated. According to other implementations, the execution results of the FPU and ALU may also be stored in the output register 9272 according to the definition of the opcode. In various embodiments, different register sizes may be used, including 32-bit, 64-bit, 128-bit, 256-bit and 512-bit registers.

[0073] The techniques discussed here refer to processor cores, processors, servers, databases, software applications and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows various possible configurations, combinations and divisions of tasks and functionality between and among components. For example, the processing discussed here can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0074] Figure 3 A flowchart of a reasoning method based on a deep learning model according to an embodiment of the present invention is shown. Figure 3 The reasoning method shown is applicable to the Figure 1A The user computing device 9110 is executed and is particularly suitable for execution on an application in the user computing device 9110. A deep learning model is used in the application for reasoning processing. As described above with reference to Figures 1-2, the deep learning model has been trained and quantized in advance.

[0075] As above reference Figure 1B As described above, the deep learning model includes multiple model layers, and the processing in each layer can be identified by the above formula (1), that is, the model layer includes matrix operations w T α, in addition, further function operations on the matrix operation results may also be included, such as σ() in formula (1).

[0076] The quantization process can be expressed by the following formula (2):

[0077] r=S(qZ) (2)

[0078] Among them, r is the floating point number to be quantized, q is the quantized integer (i.e., fixed point number), S is the scaling factor, and Z is the zero bias value.

[0079] Set each matrix cell in matrix w to Each matrix unit in α is Substituting the above quantization formula (2) into formula (1), after a series of calculations, we get:

[0080]

[0081] in

[0082] In formula (3), we input the sum into q 1 Unrelated operations are merged (such operations can be performed offline in advance because they are irrelevant to the input), and we get:

[0083]

[0084] in

[0085]

[0086] Further settings Then the above formula (4) can be converted to:

[0087]

[0088] According to the above quantization formula, for each model layer, when performing inference processing, the calculation is mainly concentrated on In this matrix multiplication and accumulation operation. When quantizing the weight matrix w, considering the number of register bits used for the matrix multiplication and accumulation operation results, the number of register bits used to store the multiplication and accumulation results should not cause overflow. If quantized q1 and q2 have the same bit positions, and the number of bits used to store the multiplication and accumulation results is twice that of q1 or q2, it is possible that the number of bits of the multiplication and accumulation results exceeds twice that of q1 or q2 due to the excessive values ​​of q1 and q2, resulting in overflow.

[0089] For example, in one example, the length of q1 and q2 is 8 bits, and the multiplication-accumulation result is stored in 16 bits. In this case, the range of q1 and q2 is [-128, 127], and the range that can be stored in the 16-bit multiplication-accumulation result is [-32768, 32767]. At this time, as shown in the following public representation:

[0090] (-128)*(-128)+(-128)*(-128)=32768>32767

[0091] Therefore, the result of multiplying these two numbers and adding them continuously is likely to overflow.

[0092] Based on the above analysis, in one embodiment of the present invention, when each weight value in the weight matrix w is quantized to q2 or q2z, it will be quantized to a number of bits less than the number of bits used for multiplication and accumulation operations and stored at this number of bits.

[0093] Specifically, in one embodiment, the range of q2 value is [0,63], that is, the weight value is quantized into a 6-bit value and stored in 8 bits. In this way, when the q2 value is subsequently read in an 8-bit manner and a multiplication-accumulation operation is performed, an overflow will not occur when a 16-bit register length is used to store the multiplication-accumulation operation.

[0094] The above example shows that the number of bits of the quantization result is two less than the number of bits for storage and operation, but the present invention is not limited thereto. In practice, the number of bits of the quantization result may be one or three less than the number of bits for storage and operation. As long as a register having a bit length twice the number of bits for storage and operation is subsequently used to store the multiplication and accumulation operation result, such a number of quantization bits is within the protection scope of the present invention.

[0095] For example, when the quantization result is stored in 8 bits and a matrix operation is subsequently performed, the quantization scheme may also adopt 5-bit and 7-bit quantization methods without departing from the protection scope of the present invention.

[0096] The above describes the way to quantize the weight matrices in each model layer. Optionally, the quantization scheme also includes quantizing values ​​that are independent of the input matrix of each layer, for example, for:

[0097]

[0098] The B value is related to the corresponding weight value and bias value of each layer, but has nothing to do with the quantized value of the input matrix. Therefore, for each layer model, the quantized B value can be calculated in advance. According to one embodiment, the B value has the same bit length as the register length for storing the multiplication and accumulation result. For example, when the multiplication and accumulation operation of the matrix unit is performed with 8 bits and the register stores the multiplication and accumulation result with 16 bits, the B value can be quantized to a fixed-point number with a length of 16 bits.

[0099] Optionally, the B value may be quantized to the same bit length as the length of the weight unit value after quantization (eg, 8 bits). As long as the precision of the B value does not affect the inference accuracy, the B value may be stored in any length.

[0100] In each model layer, in addition to matrix operations, some functions or arithmetic operations may be included, which either operate on the input matrix cell values ​​or on the intermediate results obtained by other operations, such as the results obtained by matrix operations. Such function operations include, for example, the above reference Figure 1B The activation function σ() operation described above

[0101] Optionally, such function operations also need to be quantized.

[0102] Taking the sigmoid function as an example, the function can be expressed as:

[0103] r 5 =sigmoid(r 4 ) (6)

[0104] According to the quantization formula (2), the quantized function can be expressed as:

[0105]

[0106] In the above quantization formula (7), except for q4 which is the input value, the variables on the right side of other formulas are all known to the model, that is, q5 is only related to q4. Given a q4, there will be a corresponding q5 value. Therefore, considering that q4 is a quantized fixed-point number, for example, when q4 is 6 bits in length, the value of q4 is one of [0,63], or when q4 is 8 bits, the value is one of [0,255]. Therefore, according to one embodiment of the present invention, a mapping table of q5 and q4 can be established in advance according to the above formula (7). When the q4 value to be processed is obtained, the corresponding q5 value is obtained by querying the mapping table, thereby reducing the amount of calculation in the reasoning process.

[0107] According to one embodiment of the present invention, a corresponding mapping table can be established in advance for each function operation in each model layer of the deep learning model, so that when the deep learning model is used for reasoning processing later, the mapping table can be directly called to complete the corresponding processing.

[0108] After the deep learning model quantized according to the above description has been stored in the user computing device 9110, in step S310, the input matrix of the inference calculation is received, and each matrix unit in the input matrix is ​​quantized. The scheme for quantizing the input matrix is ​​the same as the scheme for quantizing the deep learning model, that is, each matrix unit value is quantized to the same bit as the weight unit value of each model layer, and is stored in the bit (one to three bits longer than the quantization bit) used for subsequent matrix operations (e.g., multiplication and accumulation operations).

[0109] Then, in step S320, reasoning is performed using each model layer in the quantized deep learning model. For each model layer, the output matrix from the previous model layer is received as an input matrix, and the input matrix is ​​subjected to matrix operation using the weight matrix in the model layer to generate an output matrix, and the output matrix is ​​further quantized into a fixed-point number having the same bit position as the weight unit value of the next model layer, and is sent to the next layer as the input matrix of the next layer for processing.

[0110] The first model layer in the model 9120 receives the input matrix quantized in step S310 as input, and in step S330, the output matrix of the last model layer is used to obtain the final reasoning result. For example, according to one embodiment, a softmax layer can be added after the last model layer to take the result with the highest output value as the final reasoning result. It should be noted that the present invention is not limited to the method of obtaining the reasoning result according to the output of the last model layer, and all such methods are within the protection scope of the present invention.

[0111] In step S320, in each model layer, the processing of the input matrix includes matrix operations on the input matrix using the weight matrix, for example, such matrix operations include multiplication and accumulation operations. As described above for the quantization of the weight matrix, when performing the multiplication and accumulation operation, the bits used to store the weight matrix unit values ​​and the input matrix unit values ​​are used for multiplication, and the multiplication and accumulation results are saved with twice the length of the bit number. For example, the quantized weight matrix unit values ​​and the input matrix unit values ​​are 6-bit fixed-point numbers, but are stored in 8 bits, and multiplication is performed in 8 bits, and the multiplication and accumulation results are stored in 16 bits so as not to cause the multiplication and accumulation results to overflow. In other words, the multiplication and accumulation results of the 6-bit fixed-point number will not exceed the maximum value that can be represented by 16 bits, so the multiplication and accumulation results will not overflow.

[0112] Subsequently, before the output matrix is ​​sent as an input matrix of the next layer model, the output matrix may be quantized again to be a fixed-point number with the same number of bits as the weight unit in the model.

[0113] Optionally, as described above for the quantization of each model layer, each model layer also establishes a corresponding mapping table for the arithmetic or function operation in each layer. In step S320, in addition to performing matrix operations, function operations, such as activation function operations, etc., need to be performed on the input matrix unit values ​​or intermediate results such as matrix operations. At this time, in step S320, when arithmetic or function operations need to be performed during the inference processing of each model layer, the mapping table corresponding to each arithmetic or function operation established in advance can be queried to obtain the output fixed-point number corresponding to the input fixed-point number. As described above, the length of the output fixed-point number can be the same as the input fixed-point number, thereby speeding up the inference process.

[0114] In addition, optionally, when performing inference processing in each model layer, it is also necessary to calculate some values ​​that are independent of the input matrix and related to specific parameters in the layer, such as weight matrix, bias, etc. As described above for the quantization of each model layer, these values ​​have been calculated in advance when quantizing each model layer, and these values ​​are stored as constant fixed-point numbers (having the same bits as the quantized weight value, or the same bits as the matrix operation (multiplication and accumulation) result). According to one embodiment, in step S320, when some constant values ​​that are independent of the input matrix unit value are needed during the inference processing, these constant values ​​can be obtained from the quantization values ​​stored in the model layer and used directly, thereby speeding up the inference process.

[0115] According to one embodiment of the present invention, in step S320, the output matrix obtained by the reasoning operation of each model layer needs to be quantized again and sent to the next model layer as its input. The timing of quantizing the output matrix can be changed according to the operation efficiency. For example, the length of the multiplication and accumulation result (for example, 16 bits) can be used to perform function operations and arithmetic operations with constant values ​​(16-bit mode), and then the final result can be quantized (6 bits), or the matrix operation result can be quantized (quantized to 6 or 8 bits) first, and then the function operation can be performed. The present invention is not limited to the timing of quantization. As long as no overflow occurs during the reasoning process, all quantization opportunities are within the protection scope of the present invention.

[0116] Reference above Figure 2C The implementation of the instruction executed in the processor 9204 has been described. According to another embodiment of the present invention, when registers in the processor 9204 are used to execute instructions related to matrix operations, such as multiply-accumulate operations, since the overflow problem that may be caused by the multiply-accumulate operation has been considered when the model is quantized in the present invention, when the processor 9204 executes the instruction, there is no need to further consider the overflow problem. Therefore, according to the solution of the present invention, the use efficiency of the registers can be improved. Specifically, Figure 4 A flow 400 of executing a multiply-accumulate instruction in a processor according to one embodiment of the present invention is shown.

[0117] like Figure 4 As shown, in step S410, a predetermined number of input matrix unit values ​​are loaded into the first register. The length of the first register is usually an integer multiple of the bit length of the multiplication-accumulation result, so multiple input matrix unit values ​​can be loaded into the first register at one time, that is, the number that can be loaded into the first register is a multiple of the length of the first register and the input matrix unit length. For example, if the input matrix unit value is 8 bits long, and the first register is 64 bits long, that is, 4 times the 16 bits used for the multiplication-accumulation operation, then the first register can load 8 input matrix unit values ​​at one time.

[0118] Then, in step S420, the same number of weight matrix unit values ​​as in step S410 are loaded into the second register. Since the weight matrix unit has been quantized in the same scheme as the input matrix unit, the two have the same length, so as long as the length of the second register is the same as the first register, the same number of matrix units can be loaded.

[0119] Next, in step S430, a multiplication-accumulation operation is performed to multiply the input matrix unit value loaded in the first register and the weight matrix unit value at the corresponding position of the second register to obtain a multiplication result having a length twice that of the input matrix unit, that is, a length equal to the length of the multiplication-accumulation result, and the multiplication result is added to the value at the corresponding position in the third register, whose length is also the length of the multiplication-accumulation result, and the value at the corresponding position of the third register is updated to the addition result, thereby realizing the multiplication-accumulation operation.

[0120] According to the solution of the present invention, the input matrix unit value and the weight matrix unit value are first quantized to a length shorter than the bit used in the storage and matrix multiplication, and then the matrix operation (for example, multiplication and accumulation operation) is performed using the standard bit. At this time, if the multiplication and accumulation result is stored with twice the length of the fixed-point number, no overflow will occur. Therefore, more multiplication and accumulation results can be stored in the third register, or more operands can be loaded in the first and second registers to perform matrix multiplication and accumulation operations.

[0121] According to one embodiment, the weight unit and the input unit are quantized to a length of 6 bits and stored in 8 bits. The first and second registers are loaded with 8 bits for the weight unit and the input unit, respectively, and the multiplication operation is performed to represent the multiplication result in 16 bits, and the result is accumulated with the previous value of the 16 bits at the corresponding position in the third register. The accumulated result can still be stored in the register in 16 bits without causing overflow. In this way, when the first and second registers are 64 bits and 8 weights and input units are loaded at the same time, if the third register is 128 bits, 8 multiplication and accumulation results can be stored in the third register.

[0122] In the prior art, considering the impact of overflow, even if the weights and inputs are loaded into the first and second registers as 8-bit fixed-point numbers, they are expanded to 16 bits before the multiplication operation, and the multiplication and accumulation results are saved in the third register with a length of 32 bits.

[0123] When the register resources in a computing device, especially the processor 9204, are limited, the solution according to the present invention can increase the number of weights and input units that can be operated by an instruction, thereby improving the efficiency of register use and increasing the reasoning speed.

[0124] In addition, the third register is described above as being twice the length of the first register. However, it should be noted that the third register may have the same length as the first and second registers. In this case, in order to store all the multiplication-accumulation results, a fourth register having the same length as the first to third registers may be introduced. The total length of the third and fourth registers is twice the length of the first and second registers, so that all the multiplication-accumulation results can be stored. For example, the multiplication-accumulation results of the unit values ​​of the low-order regions of the first and second registers may be stored in the third register, while the multiplication-accumulation results of the unit values ​​of the high-order regions of the first and second registers may be stored in the fourth register.

[0125] For a specific example, when the third and fourth registers are both 64-bit registers, the multiplication and accumulation results of the 4 weight units and the input unit stored in the [0:31] bits of the first and second registers can be stored in the third register, and the multiplication and accumulation results of the 4 weight units and the input unit stored in the [32:63] bits can be stored in the fourth register.

[0126] In addition, the present invention is not limited to the length of each register. As in the example given above, the length of each register is 64 bits or 128 bits, but the length of the register can be shorter, such as 32 bits, or longer, such as 256 bits or 512 bits.

[0127] Figure 5 An example of post-quantization inference processing of a certain network layer in a convolutional neural network model according to an embodiment of the present invention is shown.

[0128] like Figure 5 As shown, in a convolutional neural network layer 500, a convolution process 510 and an activation function operation 520 are included. For example, for an input matrix of a 32-bit floating point number, quantization (530) can be performed as described in step S310 to quantize it into a 6-bit fixed point number, and convolution (510) is performed with a weight matrix that has been quantized into a 6-bit fixed point number in advance. Figure 3 and 4 As described above, although the input value and the weight value are quantized to 6-bit fixed-point numbers, they are stored as 8-bit fixed-point numbers, and thus a 16-bit convolution result is obtained through convolution (510), and the convolution result and the bias value (which has been quantized to a 16-bit constant fixed-point number) are summed (540) and requantized to a 6-bit fixed-point number. Subsequently, an activation function operation (520) is performed, for example, using ReLU to perform an activation function operation. According to one implementation, when performing an activation function operation, a table query is performed using a mapping table corresponding to the activation function to obtain an output corresponding to the input 6-bit fixed-point number (which can also be a 6-bit fixed-point number), and the output is used as the input of the next convolution layer.

[0129] For convolutional neural networks, in order to speed up convolution calculations, the appropriate microkernel size is selected taking into account the number of register resources in the processor so that multiple weight matrices and multiple units in the input matrix can be read from the storage unit at one time into the register for parallel processing. Figure 3-5 As described above, when the scheme according to the present invention is used for a convolutional neural network, since the influence of overflow is taken into consideration when quantizing and a quantization scheme that does not overflow is designed, for the same register resources, more operation units can be loaded into the register according to the scheme according to the present invention. For example, if a microkernel size of 4*8 is selected in the existing scheme, a microkernel size of 4*16 can be selected according to the scheme of the present invention, thereby reducing the number of times data is read from the storage device during the reasoning process and improving the execution speed of the reasoning.

[0130] The quantization scheme according to the present invention can be used in various machine learning applications that use deep learning models for reasoning. According to one embodiment, this quantization scheme can be used in machine learning applications that process speech to speed up speech recognition and speech generation processing. According to another embodiment, this quantization scheme can also be used in the field of natural language processing to speed up semantic analysis, language understanding and generation processing. According to another embodiment, this quantization scheme can also be used in image processing to speed up image segmentation, image recognition, image style change and image generation processing.

[0131] In addition, the quantization scheme according to the present invention can be used in various terminal devices (such as mobile terminals, AIoT terminals, and other computing devices with lower processing performance) to improve processing efficiency while maintaining calculation accuracy as much as possible, and can also be used in cloud devices to further speed up processing performance.

[0132] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or parts of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0133] In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

[0134] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0135] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present invention. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to implement the content of the present invention described herein, and the above description of specific languages ​​is for disclosing an embodiment of the present invention.

[0136] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0137] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0138] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0139] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0140] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0141] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0142] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0143] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative and not restrictive with respect to the scope of the present invention, and the scope of the present invention is defined by the appended claims.

Claims

1. An inference method based on a deep learning model, the deep learning model comprising a plurality of model layers connected in sequence, each model layer having its weight matrix, each weight matrix unit in the weight matrix being quantized to an integer value having a first bit number and stored in a second bit number, the second bit number being greater than the first bit number, the method comprising the steps of: Receiving an input matrix to be computed for inference by the deep learning model, so as to quantize each input matrix unit in the input matrix into an integer value having a first number of bits; Performing reasoning using each model layer, wherein a first layer of the plurality of sequentially connected model layers receives a quantized input matrix, performs a matrix operation on the input matrix using its weight matrix to generate an output matrix, requantizes each matrix unit in the output matrix to an integer value having a first bit number, and sends the output matrix to a next layer model for further operation, wherein the further operation includes receiving an output matrix from a previous model layer as an input matrix, performing a matrix operation on the input matrix using its weight matrix to generate an output matrix, requantizing each matrix unit in the output matrix to an integer value having a first bit number, and sending the output matrix to a next layer model, until a last layer model generates an output matrix; as well as Obtaining an inference result based on an output matrix generated by a last layer of the plurality of sequentially connected model layers; wherein the matrix operation comprises a multiplication-accumulation operation on the input matrix unit of the input matrix and the weight matrix unit of the weight matrix, and the matrix unit of the output matrix stores the result of the multiplication-accumulation operation with a third bit number, the third bit number is twice the second bit number, and the first bit number is set so that when each model layer performs a multiplication-accumulation operation on the corresponding matrix unit of the weight matrix and the input matrix with the second bit number, the multiplication-accumulation result is less than the maximum value corresponding to the third bit number; The method for performing a multiplication-accumulation operation comprises the steps of: Loading a predetermined number of input matrix cell values ​​in a first register, the first register having a fourth bit length, the fourth bit length being an integer multiple of the third bit length, the predetermined number being a ratio of the fourth bit to the second bit; loading a predetermined number of weight matrix unit values ​​in a second register, the second register having the same length as the first register; as well as A multiplication-accumulation operation is performed to multiply the input matrix unit value in the first register and the weight matrix unit value at the corresponding position of the second register to obtain a multiplication result of a third bit number, and the multiplication result and the value at the corresponding position in the third register with a length of the third bit number are added, and the value at the corresponding position of the third register is updated to the addition result. 2 . The inference method according to claim 1 , wherein the length of the third register is twice the length of the first register.

3. The inference method according to claim 1, wherein the length of the third register is the same as that of the first register, and the multiplication-accumulation result corresponding to the unit value of the low-order area of ​​the first register is stored in the third register, and the step of performing the multiplication-accumulation operation comprises: The input matrix unit value in the high-order area of ​​the first register and the weight matrix unit value at the corresponding position of the second register are multiplied to obtain a multiplication result of a third bit number, and the multiplication result and the value at the corresponding position in the fourth register with a length of the third bit number are added, and the value at the corresponding position of the fourth register is updated to the addition result, wherein the length of the fourth register is the same as that of the first register.

4. The inference method according to any one of claims 1 to 3, wherein each model layer further comprises one or more arithmetic operations, each model layer comprises a mapping table established for each arithmetic operation, the arithmetic mapping table establishes a mapping relationship between a quantized operand and a quantized operation result, wherein the operand has a first bit position or a second bit position, and the operation result has a second bit position; and The step of performing reasoning using each model layer includes: When performing an arithmetic operation, a mapping table corresponding to the arithmetic operation is queried to obtain an operation result corresponding to the operand. 5 . The inference method of claim 4 , wherein the operand of the arithmetic operation comprises a result of the multiply-accumulate operation. The inference method of claim 4 , wherein the arithmetic operation comprises an activation function operation.

7. The inference method of claim 1, wherein each model layer further comprises a bias value corresponding to a weight matrix unit in a weight matrix of each layer, and each model layer comprises a constant value of a third bit number obtained by quantizing the bias value; and The step of performing reasoning using each model layer includes: Before requantizing the output matrix of each model layer, the output matrix unit of each output matrix and the corresponding constant value are summed, and then the summation result is requantized.

8. The inference method as claimed in claim 1, wherein the first bit number is one, two or three less than the second bit number.

9. The inference method as claimed in claim 8, wherein the first bit number is 6 and the second bit number is 8. 10 . The inference method according to claim 8 , wherein the first bit number is 5 or 7, and the second bit number is 8.

11. The inference method according to claim 1, wherein the length of the first register and the second register is 64 bits.

12. The inference method according to claim 1, wherein the length of the first register and the second register is 128 bits.

13. The reasoning method of claim 1, wherein the deep learning model comprises a convolutional neural network, and the matrix operations for performing reasoning in each model layer comprise convolution operations of input matrix units and weight matrix units.

14. A computing device comprising: at least one processor; and A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 13.

15. A storage medium storing program instructions, wherein the program instructions are suitable for being executed by a computing device so as to perform the method according to any one of claims 1 to 13.

Images