Information processing device, information processing method, and program

Abstract

To provide an information processing device capable of providing a model for efficiently operating a hardware operation unit that performs inference processing using a neural network.SOLUTION: An information processing device according to the present disclosure includes: an inference operation unit including a product-sum operation unit and an activation function operation unit; and control means for performing operations of a learned model that is a model having a plurality of layers including activation function operations and product-sum operations using coupling coefficients and input signals, by using the inference operation unit. Here, when the numbers of product-sum operations required by a first layer and a second layer of the model are different, the control means changes the coupling coefficients of the layers of the model so that a difference between an operation amount for the first layer using the product-sum operation unit and an operation amount for the second layer using the product-sum operation unit becomes small, depending on a configuration of the product-sum operation unit included in the inference operation unit.SELECTED DRAWING: Figure 3B

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, research on learning and inference using deep learning and its application to the real world have progressed, and large-scale neural networks have come to be used. Since inference processing using large-scale neural networks requires large computing power, it can be difficult to install hardware resources corresponding to the size of the neural network in devices with limited hardware resources, such as edge devices.

[0003] In such a situation, a method has been proposed for adjusting a model, which is the result of learning of a neural network, to a format suitable for hardware (Patent Document 1, Patent Document 2). The technology proposed in Patent Document 1 converts the weight parameters of a neural network model into a format suitable for hardware during learning. Then, the weight parameters are updated based on the error propagated by the backpropagation unit using an arithmetic expression having a regularization term (i.e., an evaluation term whose cost increases the further it deviates from the binary value) that causes the weight parameters to converge to a binary value when learning the model. The technology proposed in Patent Document 2 determines the numerical format of the weights, for example, the position of the decimal point in fixed-point notation, based on the distribution of weights in the weight layer when converting the weight parameters of the neural network model into a format suitable for hardware. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-040414 A [Patent Document 2] JP 2018-010618 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, the influence of inference errors caused by quantization errors when quantizing the weights of a model can be reduced. However, improving the computing efficiency of the hardware inference calculator is not taken into consideration, and the computing efficiency of the hardware inference calculator may decrease depending on the model structure.

[0006] Furthermore, Patent Document 2 discloses determining the numerical format of weights based on the distribution of weights, but does not consider increasing the computational efficiency of a hardware inference calculator. In the technology of Patent Document 2, when a model structure is adopted in which the amount of information (e.g., the number of bits) expressing weights differs for each layer, a difference may occur between the maximum and minimum amounts of computation for weights for each layer. In other words, the computational efficiency of the hardware inference calculator may decrease.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to realize a technology that can provide a model that efficiently operates a hardware computing unit that performs inference processing using a neural network. [Means for solving the problem]

[0008] In order to solve this problem, for example, an information processing device of the present invention has the following configuration: That is, the information processing device has an inference calculator including a multiply-accumulate calculator and an activation function calculator, and a control means for executing calculations of the trained model using the inference calculator, the model having a plurality of layers including a multiply-accumulate calculation using a coupling coefficient and an input signal and an activation function calculation, and the control means is characterized in that, when the number of product-accumulate calculations required for the first layer and the second layer of the model differs, the control means changes the coupling coefficients of the layers of the model according to the configuration of the product-accumulate calculator included in the inference calculator so as to reduce the difference between the amount of calculation for the first layer using the product-accumulate calculator and the amount of calculation for the second layer using the product-accumulate calculator. Effect of the Invention

[0009] According to the present invention, it is possible to provide a model that efficiently operates a hardware computing unit that performs inference processing using a neural network. [Brief description of the drawings]

[0010] [Figure 1A] Diagram illustrating the structure of a neural network for identifying object classes. [Figure 1B] A diagram illustrating the detailed structure of a neural network for identifying object classes. [Figure 2A] FIG. 1 is a block diagram showing an example of the functional arrangement of a personal computer as an example of an information processing apparatus according to a first embodiment; [Figure 2B] FIG. 1 is a diagram showing a schematic structure of a model exemplified in the first embodiment; [Figure 3A] 1 is a flowchart showing the operation of an information processing device including a model adjustment process according to the first embodiment. [Figure 3B] 1 is a flowchart showing a series of operations in a model adjustment process according to the first embodiment. [Figure 4] FIG. 1 is a diagram showing an example of the configuration of one multiply-add calculator according to the first embodiment; [Diagram 5] FIG. 1 is a diagram showing an example of the configuration of a hardware inference unit including a multiply-and-accumulate unit according to the first embodiment; [Figure 6] FIG. 1 is a diagram showing a matrix operation for signal propagation calculation according to the first embodiment; [Figure 7] FIG. 1 is a diagram for explaining the operation of a multiply-and-accumulate calculator according to the first embodiment; [Figure 8] FIG. 1 is a diagram showing a detailed configuration example of a multiply-and-accumulate calculator according to the first embodiment; [Figure 9] FIG. 1 is a diagram for explaining a signal flow in a multiply-add calculator according to the first embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Embodiment 1) Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] Hereinafter, a technique for optimizing a model (also simply called a model) configured with a neural network for a configuration of a hardware arithmetic unit will be described. In this embodiment, when optimizing the model, the model is adjusted so as to increase the operating efficiency of the hardware. In the following description, a case where a neural network is used as an example of the model will be described.

[0013] First, referring to FIG. 1A, the configuration of a Convolutional Neural Network (CNN) as an example of a neural network model will be described. The CNN shown in FIG. 1A outputs the class of the object of the input 2D image data (i.e., it functions as an object classifier). The left end is the input and processing proceeds to the right. The CNN has multiple layers, each of which is a set of two layers called a feature detection layer (S layer) and a feature integration layer (C layer). The S layer is also called a convolution layer or a convolution layer, and the C layer is also called a pooling layer.

[0014] In CNN, the S layer is configured to detect the next feature based on the features detected in the previous layer, and the C layer is configured to integrate the features detected in the S layer and send them to the next layer as the detection result of that layer.

[0015] The S layer contains feature detection cell surfaces, and each feature detection cell surface detects a different feature. The C layer is made up of feature integration cell surfaces, and pools the detection results from the previous feature detection cell surfaces. Below, unless there is a need to distinguish between them, the feature detection cell surfaces and feature integration cell surfaces may be collectively referred to as feature surfaces. Note that the output layer, which is the final layer, does not use the C layer and is made up of only the S layer.

[0016] The details of the feature detection process on the feature detection cell surface and the feature integration process on the feature integration cell surface will be explained with reference to Fig. 1B. The feature detection cell surface is composed of multiple feature detection neurons, which are connected to the C layer of the previous layer in a predetermined structure (i.e., connected to specific neurons in the C layer of the previous layer). The feature integration cell surface is composed of multiple feature integration neurons, which are connected to the S layer of the same layer in a predetermined structure (i.e., connected to specific neurons in the S layer).

[0017] In the M-th cell plane of the S layer of the Lth hierarchy shown in Fig. 1B, the output value of the feature detection neuron at position (ξ, ζ) is expressed as y M LS (ξ,ζ), in the Mth cell plane of the Lth layer C, the output value of the feature integration neuron at position (ξ,ζ) is y M LC Let (ξ,ζ) be the coupling coefficient of each neuron. M LS (n, u, v), w M LC If the output values ​​are (u, v), then each output value can be expressed as Equation 1 and Equation 2.

[0018]

number

number

[0019] f in Equation 1 is an activation function, which can be any sigmoid function such as a logistic function or a hyperbolic tangent function, and can be realized, for example, by a tanh function. M LS (ξ,ζ) is the internal state of the feature detection neuron at position (ξ,ζ) on the Mth cell plane of layer S of the Lth hierarchy.

[0020] Equation 2 does not use an activation function and simply takes a linear sum. When no activation function is used as in Equation 2, the internal state of the neuron u M LC (ξ,ζ) and output value y M LC (ξ,ζ) are equal. Also, y shown in Equation 1 n L-1C (ξ+u,ζ+v), y shown in Equation 2 M LS (ξ+u, ζ+v) are called the output values ​​of the feature detection neuron and the feature integration neuron, respectively.

[0021] The (ξ, ζ, u, v, n) in Equation 1 and Equation 2 will be explained. The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, y M LS If (ξ,ζ) has a high output value, it means that there is a high possibility that the feature to be detected in the Mth cell plane in the Sth layer of the Lth hierarchical level exists at pixel position (ξ,ζ) of the input image. In addition, n in equation 2 means the nth cell plane in the Cth layer of the L-1th hierarchical level, and is called the target feature number. Basically, a product-sum operation is performed for all cell planes that exist in the Cth layer of the L-1th hierarchical level.

[0022] (u,v) is the relative position coordinate of the coupling coefficient, and the product-sum operation is performed in a finite range (u,v) according to the size of the feature to be detected. Such a finite range (u,v) is called the receptive field. The size of the receptive field is called the receptive field size below and is expressed as the number of horizontal pixels x the number of vertical pixels in the coupling range.

[0023] In addition, in equation 1, L=1, that is, in the first S layer, y n L-1C (ξ+u,ζ+v) is the input image y in_image(ξ+u,ζ+v), or the input location map y in_posi_map (ξ+u,ζ+v). Incidentally, the distribution of neurons and pixels is discrete, and the connection feature numbers are also discrete, so ξ,ζ,u,v,n are not continuous variables but take discrete values. Here, ξ,ζ are non-negative integers, n is a natural number, and u,v are integers, all of which are finite.

[0024] w in number 1 M LS (n,u,v) is the distribution of coupling coefficients for detecting a specific feature, and by adjusting the distribution of coupling coefficients to an appropriate value, it becomes possible to detect the specific feature of the subject in the image. Adjusting this distribution of coupling coefficients is called learning. In the learning stage of constructing a CNN, various data for learning are input to the CNN, and y M LS The coupling coefficients are repeatedly and gradually adjusted so that (ξ,ζ) becomes the appropriate output value. In this way, at the final stage of learning, the coupling coefficients are adjusted (i.e. optimized) to the appropriate value.

[0025] Next, w in number 2 M LS (u, v) uses a two-dimensional Gaussian function and can be expressed as the following equation 3.

number

[0026] Here again, (u, v) is a finite range, so just as in the explanation of feature detection neurons, this finite range is called the receptive field, and the size of the range is called the receptive field size. Here, this receptive field size can be set to an appropriate value depending on the size of the Mth feature in the Sth layer of the Lth hierarchy. σ in Equation 3 is the feature size factor, and can be set to an appropriate constant depending on the receptive field size. Specifically, it is best to set it so that the outermost value of the receptive field can be considered to be almost 0.

[0027] By performing the above-mentioned calculations at each layer, the object class is identified at the final layer, the S layer (for example, a numerical value indicating whether or not the object belongs to a class is output).

[0028] Next, a specific learning method using the above-mentioned model will be described. In this embodiment, the coupling coefficients are adjusted by supervised learning. In supervised learning, various patterns of learning data (for example, images of a subject) are input to the model to actually find the output value of the neuron. Then, the coupling coefficient w is calculated from the relationship between the output value of the model and the teacher signal (the desired output value that the neuron should output). M LS (n, u, v) are repeatedly corrected. In the learning of this embodiment, the last feature detection layer uses the least squares method, and the intermediate feature detection layer corrects the coupling coefficients using the backpropagation method. As a method for correcting coupling coefficients using the least squares method, backpropagation method, etc., for example, the method of S. Haykin, "Neural Networks A Comprehensive Foundation 2nd Edition", Prentice Hall, pp. 156-255, July 1998 can be used.

[0029] In this embodiment, when learning in advance, many specific patterns to be detected and patterns not to be detected are prepared as learning data patterns, and when learning is performed additionally, the data patterns are selected from the buffer by the above-mentioned method. Each pattern is a set of an image and a teacher signal.

[0030] For example, when using the tanh function as the activation function, when a specific pattern to be detected is presented, a teacher signal is given to the neurons in the area where the specific pattern exists on the feature detection cell surface of the final layer so that their output becomes 1. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to the neurons in the area of ​​that pattern so that their output becomes -1.

[0031] The neural network is constructed in this way. In the inference stage, where an unknown image is input to the trained neural network, the connection coefficient w M LS A calculation is performed using (n, u, v). Then, if the neuron output on the feature detection cell plane of the final layer is equal to or greater than a predetermined value, it is determined that the object class corresponding to the neuron is present in the input image.

[0032] (Configuration of information processing device) Next, a configuration example of a PC (personal computer) as an example of an information processing device will be described with reference to Fig. 2A. In the following description, the information processing device will be described as a PC, but the information processing device is not limited to this. The information processing device may be any device such as a digital camera, a video camera, a mobile terminal, or a game machine.

[0033] The system bus 201 is a data bus for exchanging information between each block in the PC 200. The CPU 202 is a central processing unit that controls each block of the PC 200 and controls input / output with the outside. The CPU 202 controls the overall operation of the PC 200 and the operation of each component by loading a computer program stored in the ROM 203 or HDD 205 into the RAM 204 and executing it.

[0034] ROM 203 is a read only memory, and is constituted by a non-volatile semiconductor memory such as an EEPROM, and stores, for example, a computer program that CPU 202 uses to control PC 200. RAM 204 is a random access memory, and is a volatile semiconductor memory such as a DRAM. RAM 204 is used by CPU 202 and each block in PC 200 as a place to temporarily store information as needed. This information includes, for example, information related to program execution.

[0035] The HDD 205 is a hard disk drive. It is used as an information storage area as long-term memory by each block in the PC 200 as necessary. Information stored in the HDD 205 includes a computer program for the CPU 202 to control the PC 200, learning data, values ​​of the coupling coefficients after learning, and the like. In this embodiment, the HDD is described as an example, but a semiconductor memory such as an SSD may be used as long as it can be used as storage.

[0036] The GPU 206 is a graphics processing unit, and performs, for example, model adjustment processing described below and inference processing using the product-adder 207 described below. The GPU 206 can perform efficient calculations by processing more information in parallel. For this reason, the GPU 206 is effective when performing repeated calculations multiple times using a learning model such as machine learning. Note that the model adjustment processing and inference processing using the hardware inference unit 207 may be performed by the CPU 202 and the GPU 206 in cooperation with each other.

[0037] The hardware inference unit 207 includes a multiply-accumulate unit and an activation function unit, which will be described later, and is an inference unit for executing model processing by hardware. The configuration and operation example of the multiply-accumulate unit will be described later.

[0038] The input unit 208 includes input devices such as a keyboard and a mouse, and accepts operation instructions from a user who uses the PC 200. The display unit 209 controls the screen display of, for example, a monitor or a display device equivalent thereto. The display unit 209 may include the display device itself, such as a monitor. The screen display includes a display of the operating state in response to an operation instruction from the user who uses the PC 200, a display of an operation menu, a display of various applications, and the like.

[0039] 2B shows a schematic diagram of the overall model of the neural network used in this embodiment. For ease of explanation, in the following explanation, neurons are called nodes, signals connecting neurons are called edges, and connection coefficients representing the strength of connections between neurons are called edge weights.

[0040] (A series of operations including model learning and model adjustment processes) Next, a series of operations including model learning and model adjustment processing executed in the PC 200 will be described with reference to Fig. 3A. Note that in the following description, for example, a case where this processing is realized by the GPU 206 (or the CPU 202) executing a program stored in the HDD 205 will be described as an example. However, this is not limiting, and the execution entity may be appropriately different, such as the GPU 206 performing model learning and the CPU 202 performing model adjustment processing.

[0041] In S300, the GPU 206 executes a model learning process. As described above, the GPU 206 repeatedly inputs various images of the learning data to the model, adjusts the coupling coefficients using the backpropagation method or the like, and converges to an appropriate value (such as minimizing the least square error). The model is put into a learned state by this model learning process.

[0042] In S310, the GPU 206 executes a model adjustment process. In the model adjustment process, edge weights calculated by model learning and configuration information of the hardware inference unit 207 are used. Details of the model adjustment process will be described later with reference to FIG. 3B. For example, when the GPU 206 ends the model adjustment process, it ends this series of processes.

[0043] (A series of operations for model adjustment processing according to hardware configuration) Furthermore, a series of operations of the model adjustment process according to the hardware configuration will be described with reference to Fig. 3B. Note that the series of operations of the model adjustment process is realized by, for example, the GPU 206 (or the CPU 202) executing a program stored in the HDD 205.

[0044] In S311, the GPU 206 first determines whether or not to perform pruning processing on the model shown in FIG. 2B. The pruning processing is processing for changing the values ​​of some of the edges to zero. If the GPU 206 determines to perform pruning processing according to the configuration of the hardware inference unit 207, etc., the process proceeds to S101, and if not, the process proceeds to S312. For example, when the numbers of edges (the number of multiply-add operations) of the two layers are different, the GPU 206 determines that pruning is necessary to make the numbers of multiply-add operations of both layers uniform or to reduce the difference. Note that the determination of whether or not to perform pruning processing is not limited to this, and other determinations may be made. For example, when the PC 200 is set to prioritize the quantization processing over the pruning processing, the PC 200 may determine not to perform pruning processing and perform the quantization processing (NO in S311).

[0045] In S311, the GPU 206 performs pruning processing. In this embodiment, when the weight value of an edge connecting nodes is small, the GPU 206 determines that the weight of the edge has little effect on the inference result, and deletes the edge from the model (i.e., sets the weight to zero). By deleting edges from the model through pruning processing, the amount of calculation during inference can be reduced, and a model that efficiently operates the hardware inference unit 207 can be generated, as described below.

[0046] In this embodiment, the GPU 206 determines the edge to be pruned according to the weight of the edge itself and the configuration of the hardware inference device. The hardware inference device 207 in this embodiment includes, for example, 20 multiply-accumulate operators. The multiply-accumulate operators are operators that can perform a multiply-accumulate operation in one clock cycle, as shown in FIG. 4. The model realized by the hardware inference device 207 has, for example, an input layer, one hidden layer, and an output layer, as shown in FIG. 2B. The input layer has four nodes, the hidden layer has five nodes, and the output layer has two nodes. The hidden layer node includes an arbitrary activation function. In this example, the edge group connecting the input layer and the hidden layer may be called the first layer, and the edge group connecting the hidden layer and the output layer may be called the second layer.

[0047] The input a of the hidden layer node shown in Fig. 2B j (1) To calculate (j=1), we use the multiply-accumulate calculator shown in Figure 4 to calculate the input x i (1) (i=1,2,3,4) and edge weight w j,i (1) (i=1, 2, 3, 4, j=1) is multiplied and accumulated. Alternatively, the input to node 220 may be calculated by arranging the multiply-accumulators shown in FIG. 4 in parallel and adding them at the end, as shown in FIG.

[0048] Output x of node 220 i (2) is the output of the multiply-accumulate unit a j (1) is obtained by passing it through the activation function calculator. The activation function calculator calculates sigmoid functions, tanh functions, etc. The input a of the hidden layer j (1) To calculate, follow the matrix formula in Figure 6(a) and j (1) For each a, four multiplication and accumulation operations are performed. j (1) A total of 20 product-sum operations are required to calculate (j=1,2,3,4,5). Note that the bias term that is generally included in neural networks is omitted for simplicity.

[0049] Next, the input a of the output layer j (2) To calculate, a is calculated according to the matrix equation in Fig. 6(b). j (2) Five multiplication and accumulation operations are performed for each a j (2) A total of 10 product-sum operations are required to calculate (j=1,2). Here, we assume that the input to the input layer is input continuously and sequentially. This corresponds to, for example, the sequential input of images to a neural network model that performs object recognition.

[0050] As described above, the hardware inference unit 207 has 20 multiply-accumulate units, and can perform 20 multiply-accumulate operations in one cycle. Therefore, to calculate the first layer of the neural network, 20÷20=1 cycle is required. On the other hand, to calculate the second layer, 10÷20=0.5 cycle is required, and therefore 1 cycle is required. When calculating the second layer, 10 multiply-accumulate operations are required, and therefore only 10 of the 20 multiply-accumulate units are used. As is clear from FIG. 7(a), 20 multiply-accumulate units are in operation during the period from time t0 to time t0+1, while 10 multiply-accumulate units are not in operation during the period from time t0+1 to time t0+2. In other words, the hardware inference unit 207 is not able to demonstrate its original performance when calculating the processing of the second layer.

[0051] For example, as shown in Fig. 7(b), if the calculations for two images are assigned to different multiply-and-accumulate units, the hardware inference unit can be kept running. However, this increases the latency, which is the time it takes for an image to be input to the model and output, from two cycles to three cycles.

[0052] In contrast, in this embodiment, the number of edges in the first layer is reduced from 20 to 10 by pruning (so that the number of edges in the first layer is equal to the number of edges in the second layer). As a result, as shown in FIG. 7(c), 20 multiply-accumulators are always in operation, and the processing latency for one image is maintained at 2 cycles. In other words, the operating rate of the hardware inference unit is increased, and the performance can be fully demonstrated. In this embodiment, the GPU 206 deletes 10 edges with small weight values ​​from among the edges in the first layer in the pruning process. Alternatively, the configuration may be such that edges with small weights are selected and deleted with a high probability.

[0053] In the above embodiment, the number of multiply-accumulators operating for the first layer and the number of multiply-accumulators operating for the second layer (i.e., the amount of calculation using the multiply-accumulators) are made equal to each other. However, pruning may be performed so that the difference between the number of multiply-accumulators operating for the first layer and the number of multiply-accumulators operating for the second layer is reduced. In the above embodiment, the number of multiply-accumulators operating for the first layer is reduced, but the number of multiply-accumulators operating for the first layer and the number of multiply-accumulators operating for the second layer may be changed. That is, when the number of multiply-accumulators operating for the first layer is greater than the number of multiply-accumulators operating for the second layer, the number of edges pruned for the first layer may be greater than the number of edges pruned for the second layer. This also increases the operating rate of the hardware inference unit when executing each layer.

[0054] In S313, the GPU 206 performs model adjustment by quantization. Specifically, the GPU 206 inputs the image input signal and the edge weights (combination coefficients) to each multiply-accumulate calculator while controlling the calculation mode for each multiply-accumulate calculator in order to control the bit width of the edge weights. As shown in FIG. 3B, the quantization process may be performed after the pruning process in S312. However, in the following explanation of the quantization process, a processing example will be described using a case where quantization is performed without pruning (NO in S311) as an example. That is, at the stage of executing the process of S313, the neural network shown in FIG. 2B has 20 edges in the first layer and 10 edges in the second layer.

[0055] In this embodiment, the bit width of the input signal S is 4 bits, and the bit width of the edge weight W before quantization is also 4 bits. The bit width of the edge weight W becomes 4 bits or 2 bits by quantization. As shown in FIG. 8, the multiply-accumulate calculator of this embodiment has a configuration that can perform a multiply-accumulate operation whether the bit width of the edge weight W is 4 bits or 2 bits. When such a multiply-accumulate calculator is used, the amount of calculation for each layer is proportional to the sum of the bit widths of the edge weights of each layer of the model, so that the amount of calculation for a specific layer can be adjusted by quantization.

[0056] Two types of multiply-and-accumulate methods with a bit width of 4 bits and 2 bits are described below. For simplicity of explanation, the input signal S, edge weight W, edge weight W1, and edge weight W2 are unsigned signals. The relationship between the edge weights W, W1, and W2 is expressed as in Equation 4. [Number 4] W=W2·4+W1

[0057] In Fig. 9(a), the signal flow when performing 4-bit x 4-bit multiplication is indicated by a bold line. When performing 4-bit x 4-bit multiplication, the GPU 206 sets the calculation mode M (signal 1060) to 0. Here, the multiplication value to be calculated is S·W, which can be expressed as Equation 5 by transforming the formula using Equation 4. [Number 5] S W = S W2 4 + S W1

[0058] The procedure for calculating the formula shown in Equation 5 by the product-sum calculator of this embodiment will be described below. Bit division 1070 divides edge weight W (signal 1004) into signals W1 (signal 1006) and W2 (signal 1008). Input signal S (signal 1002) is integrated by signal 1006 and integrator 1072, and signal 1008 and integrator 1074. At this time, signal 1010 is the product of signal 1002 and signal 1006, and is a 6-bit signal. Signal 1012 is the product of signal 1002 and signal 1008, and is a 6-bit signal. Signal 1016 is the sum of signal 1010 and signal 1014, and is a 7-bit signal.

[0059] Registers 1020 and 1022 are elements that store the sum-of-products values ​​immediately before the repeated sum-of-products operation, and each has 6 bits. The sum-of-products values ​​held in registers 1020 and 1022 are reset to zero at the start of the repeated sum-of-products operation. The outputs of register 1020 and register 1022 are concatenated by bit concatenation 1080 to form a 12-bit signal 1017 as shown in number 6. [Number 6] [signal 1017] = [register 1022] · 64 + [register 1020]

[0060] The adder 1081 adds the signal 1017 and the signal 1016 to output a 13-bit signal 1018. When the outputs of the register 1020 and the register 1022 are zero, the signal 1018 is equal to the product of the signal S and the signal W. The limiter 1082 limits the signal 1018 to a 12-bit signal. The bit divider 1085 divides the signal 1024 into 6-bit signals 1026 and 1028. The signal 1026 is composed of the lower bits of the signal 1018, and the signal 1028 is composed of the upper 6 bits of the signal 1018. The signal 1026 passes through the selector 1084 and is stored in the register 1020 as the signal 1032. The signal 1028 passes through the selector 1086 and is stored in the register 1022 as the signal 1034.

[0061] After the above-mentioned process is repeated the number of times for the multiply-and-accumulate operation, the multiply-and-accumulate calculator outputs the final multiply-and-accumulate value as a signal 1017. A limiter 1086 limits the signal 1017 to a 4-bit signal 1036, which is then output as a signal 1062 via a selector 1088.

[0062] Next, in Fig. 9(b), the thick lines show the signal flow when the multiply-accumulate unit performs two 4-bit x 2-bit sums. In this case, the GPU 206 sets the calculation mode M (signal 1060) to 1. The bit division 1070 divides the edge weight W (signal 1004) into signals W1 (signal 1006) and W2 (signal 1008). Here, the sum values ​​to be calculated are S·W1 and S·W2.

[0063] Signal W1 (signal 1006) and signal W2 (signal 1008) represent the weights of two different edges, and the multiply-accumulate calculator of this embodiment calculates S·W1 and S·W2 simultaneously. The processing procedure in this case will be described below.

[0064] The input signal S (signal 1002) is integrated with the signal 1006 by the integrator 1072, and is integrated with the signal 1008 by the integrator 1074. The signal 1010 is S·W1. The adder 1095 adds the signal 1010 and the value of the register 1020, and outputs a 7-bit signal 1040. The signal 1012 is S·W2. The adder 1096 adds the signal 1012 and the value of the register 1022, and outputs a 7-bit signal 1042. The signal 1040 passes through the selector 1084 and is stored in the register 1020 as the signal 1032. The signal 1042 passes through the selector 1086 and is stored in the register 1022 as the signal 1034.

[0065] After the above-mentioned process is repeated the number of times for the multiply-and-accumulate operation, the final multiply-and-accumulate value is output as the value of registers 1020 and 1022. Limiter 1097 limits the value of register 1020 to a 4-bit signal 1044 and outputs it. Limiter 1098 limits the value of register 1022 to a 4-bit signal 1046 and outputs it. Bit concatenation 1099 bit-concatenates signals 1044 and 1046, and selector 1094 outputs bit-concatenated signal 1048 as signal 1062. At this time, the relationship between signals 1044, 1046, and 1048 is expressed as follows: [Number 7] [signal 1048] = [signal 1046] 4 + [signal 1044]

[0066] 8, the multiply-and-accumulate operation can be performed by selecting either one 4-bit x 4-bit multiply-and-accumulate operation or two 4-bit x 2-bit multiply-and-accumulate operations. Therefore, when the edge weight is set to 2 bits by quantization, two multiply-and-accumulate operations can be performed in one cycle.

[0067] For this reason, in S313, the GPU 206 controls the operation mode M of the multiply-accumulate calculator so that, for example, the edge weights of the first layer are quantized to 2 bits and the edge weights of the second layer are quantized to 4 bits. In this way, the first layer and the second layer can complete the operations in the same cycle time. In other words, the time during which the multiply-accumulate calculator is not operating, as shown in FIG. 8, can be reduced. The GPU 206 ends this series of operations when it completes the execution of the hardware inference calculator on the input image while adjusting the model.

[0068] In this embodiment, the multiply-add calculator has been described as being configured to selectively perform one 4-bit x 4-bit multiply-add calculation or two 4-bit x 2-bit multiply-add calculations. However, the multiply-add calculator may have other configurations. For example, the input signal S and edge weights may be 8 bits, and the edge weight W may be configured with four different edge weights W4, W3, W2, and W1, as shown in Equation 6 below. In this case, a hardware inference unit may be configured to simultaneously calculate S·W1, S·W2, S·W3, and S·W4. [Number 8] W=W4·16+W3·8+W2·4+W1 Alternatively, the weight edge W may be configured as edge weights W2 and W1 with different bit widths as shown in Equation 9, instead of making the weights W2 and W1 of two different edges have the same bit width. In this case, a hardware inference unit may be configured so that S·W1 and S·W2 can be calculated simultaneously. [Number 9] W=W2·16+W1

[0069] In the above example, the edge weights in the first layer are quantized, but the edge weights in the first and second layers may be quantized separately. In this case, if the number of multiplication and accumulation operations required by the first layer is greater than the number of multiplication and accumulation operations required by the second layer, the bit width of the coupling coefficients may be quantized so that the bit width of the edge weights in the first layer is smaller than the bit width of the edge weights in the second layer.

[0070] As described above, in this embodiment, the weight edge of the model is pruned, the weight edge is quantized, and the number of times the multiply-accumulator is used is adjusted. When the number of multiply-accumulator operations required for the two layers of the model is different, the coupling coefficient of the model is changed so that the difference in the amount of operations for the two layers using the multiply-accumulator is reduced according to the configuration of the multiply-accumulator included in the hardware inference device. The configuration of the multiply-accumulator is the number of multiply-accumulators included in the hardware inference device and the bit width of the multiply-accumulator. In this way, it is possible to increase the operating rate of the hardware inference device and to fully utilize the performance of the hardware inference device. In other words, it is possible to provide a model that efficiently operates a hardware calculator that performs inference processing using a neural network.

[0071] (Embodiment 2) Next, a second embodiment will be described. In the second embodiment, model learning is performed while taking into consideration model adjustment. Note that this embodiment differs from the first embodiment in that processing while taking into consideration model adjustment is performed while learning the model, but the configuration of the PC 200 and other operations are the same or substantially the same as those of the first embodiment. Therefore, the same components are given the same reference numerals and duplicated explanations will be omitted.

[0072] Model learning in this embodiment uses the backpropagation method described in embodiment 1. In the backpropagation method in model learning, the value of the coupling coefficient in the next learning is determined by subtracting from the current coupling coefficient the value obtained by partially differentiating the evaluation function with respect to the coupling coefficient and multiplying the result by the learning rate η, as shown in equation 10. In this embodiment, if the updated edge weight W is smaller than a predetermined threshold ε, the edge weight W is further calculated and adjusted to approach zero, as shown in equation 11.

number

number

[0073] By setting the value of the edge weighting coefficient W close to zero, the impact on inference error when pruning edges (i.e., setting the value of the coupling coefficient to zero) is minimized in the model adjustment process for the hardware inference device.

[0074] Furthermore, in this embodiment, edges to be dropped out are selected based on the configuration of the hardware inference device. Dropout is a technique in which some edges are selected in a learning cycle, and learning proceeds with the selected edges deleted. By deleting some edges, learning proceeds in a pseudo-different model, and an effect similar to that of ensemble learning using multiple models can be expected.

[0075] The edge weight of a dropped out edge is not updated in that learning cycle. In this embodiment, when selecting an edge to be dropped out, the GPU 206 preferentially selects an edge whose weight is close to zero. By selecting an edge whose weight is close to zero, it is possible to reduce the effect on inference error when pruning is performed in model adjustment. Alternatively, in consideration of the concern that the effect of dropout may be reduced by repeatedly selecting an edge whose weight is close to zero for each learning cycle, it is possible to configure the edge whose weight is small to be selected with a high probability.

[0076] Alternatively, when selecting edges to be dropped out, edges that will have a small quantization error when adjusting the model for the hardware inference device may be selected. For example, consider a case where the hardware inference device handles the coupling coefficient in a fixed-point format of 0 to 1 with 4 unsigned bits, 1 bit integer part, and 3 bits decimal part. With this numerical representation, it is possible to express from 0 to 1 in increments of 0.125. Here, a numerical value with a small difference in absolute value from a number that is divisible by 0.125, such as 0.250 or 0.375, has a small quantization error in this numerical representation. Edges with a small quantization error have a small effect on the inference error even when model adjustment is performed by quantization. As described above, the weight of the dropped out edge is not updated in that learning cycle, so learning proceeds with the quantization error remaining small. On the other hand, the value of the edges that are not dropped out is updated by learning. Depending on the amount of update, the quantization error of the coupling coefficient may become a small value, and such a coupling coefficient is dropped out in the next learning cycle.

[0077] Therefore, by preferentially selecting edges with small quantization errors as targets for dropout, the weight values ​​of the model edges are adjusted so that they include many with small quantization errors as learning progresses. Furthermore, when the model is adjusted for a hardware inference device in the model adjustment process, the inference error can be reduced even if the edge weights are quantized.

[0078] As described above, in this embodiment, in consideration of inference errors that occur when further adjusting a model to be operable by a hardware inference unit, model adjustment is performed based on the configuration of the hardware inference unit during model learning. That is, when the updated edge weight W value is smaller than a predetermined threshold, the edge weight is adjusted to approach zero. Alternatively, edges are dropped out during model learning, and edges with weights close to zero are preferentially dropped out. In this way, inference errors can be reduced when the model adjustment process is executed.

[0079] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0080] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0081] 202...CPU, 206...GPU, 207...hardware inference unit, 220...node

Claims

1. an inference calculator including a multiply-and-accumulate calculator and an activation function calculator; A control means for executing a calculation of the trained model having a plurality of layers including a product-sum calculation using a coupling coefficient and an input signal and a calculation of an activation function, using the inference calculator; the control means, when the number of product-sum operations required for a first layer and a second layer of the model differs, changes the coupling coefficients of the layers of the model in accordance with a configuration of a product-sum operation unit included in the inference operation unit so that a difference between the amount of operations for the first layer using the product-sum operation unit and the amount of operations for the second layer using the product-sum operation unit is reduced.

2. 2. The information processing device according to claim 1, wherein the configuration of said multiply-accumulate calculator of said inference calculator includes the number of said multiply-accumulate calculators or the bit width of a signal of said multiply-accumulate calculator.

3. 3 . The information processing apparatus according to claim 1 , wherein the control means changes values ​​of some of the coupling coefficients to zero when changing the coupling coefficients of the layers of the model. 4 .

4. 4. The information processing device according to claim 3, characterized in that, when the number of product-sum operations required by the first layer is greater than the number of product-sum operations required by the second layer, the control means causes the number of coupling coefficients whose values ​​are changed to zero for the first layer to be greater than the number of coupling coefficients whose values ​​are changed to zero for the second layer.

5. 3 . The information processing apparatus according to claim 1 , wherein the control means quantizes a bit width of the coupling coefficients when changing the coupling coefficients of the layers of the model. 4 .

6. 3. The information processing device according to claim 1, wherein the control means quantizes the bit width of the coupling coefficients of the first layer so that the bit width of the coupling coefficients of the first layer is smaller than the bit width of the coupling coefficients of the second layer when the number of product-sum operations required by the first layer is greater than the number of product-sum operations required by the second layer.

7. Further comprising a learning means for learning the model that has not been trained and calculating a coupling coefficient of the model, 7. The information processing device according to claim 1, wherein the learning means updates the coupling coefficients of the model by backpropagation of error, and when updating the coupling coefficients by backpropagation of error, controls the update amount of the coupling coefficients so that coupling coefficients that are smaller than a predetermined threshold value among the coupling coefficients are brought closer to zero.

8. Further comprising a learning means for learning the model that has not been trained and calculating a coupling coefficient of the model, 7. The information processing device according to claim 1, wherein the learning means updates the coupling coefficients of the model by backpropagation, and drops out one or more coupling coefficients when updating the coupling coefficients by backpropagation.

9. 9. The information processing apparatus according to claim 8, wherein, when dropping out the one or more coupling coefficients, the learning means gives priority to dropping out coupling coefficients closer to zero among the coupling coefficients.

10. 9. The information processing apparatus according to claim 8, wherein, when dropping out the one or more coupling coefficients, the learning means gives priority to dropping out coupling coefficients that have smaller quantization errors caused by quantization.

11. 7. The information processing apparatus according to claim 5, wherein the amount of calculation for each layer using said multiply-add calculator is proportional to the sum of the bit widths of the coupling coefficients of each layer of said model.

12. An information processing method executed by an information processing device having an inference calculator including a multiply-add calculator and an activation function calculator, A control step of executing, using the inference calculator, a calculation of the trained model having a plurality of layers including a product-sum calculation using a coupling coefficient and an input signal and a calculation of an activation function, an information processing method, characterized in that in the control step, when the number of product-sum operations required for the first layer and the second layer of the model differs, the coupling coefficients of the layers of the model are changed in accordance with the configuration of a product-sum operation unit included in the inference operation unit so that a difference between the amount of operations for the first layer using the product-sum operation unit and the amount of operations for the second layer using the product-sum operation unit is reduced.

13. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 11.

Images