Deduction processor

Abstract

To reduce the circuit scale when performing a product-sum operation on feature data of two bits or more using a one-bit weighting factor.SOLUTION: In a neural network having multiple layers, the result of a product-sum operation between data of multiple feature maps and a one-bit weighting factor is output, and a correction value for correcting the result of the product-sum operation, which is independent of the data of the feature maps, is added to the result of the product-sum operation.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to processing of a neural network with 1-bit weight coefficients. [Background technology]

[0002] Deep neural networks (DNNs) such as convolutional neural networks (CNN) and vision transformers (ViT) are attracting attention as high-performance artificial intelligence (AI) algorithms. Neural networks have multiple layers connected hierarchically. Each layer has multiple feature maps. The product-and-accumulate (MAC) result is calculated using the learned filter coefficients (weighting coefficients) and the pixels (feature data) of the feature maps. The product-and-accumulate operation includes multiple multiplications and cumulative additions. In the case of CNN, the product-and-accumulate operation is often realized by convolution processing. The feature map in the current layer is calculated using the feature map in the previous layer and the weighting coefficients corresponding to the previous layer. To calculate one feature map in the current layer, information on multiple feature maps in the previous layer is required. After the product-and-accumulate operation, the product-and-accumulate operation result is used to perform activation processing, quantization, pooling, and other processing based on the neural network structure information, and the feature map of the current layer is calculated.

[0003] Since neural networks perform a large number of product-sum operations, when they are applied to embedded systems such as mobile terminals and in-vehicle devices, it is necessary to perform the product-sum operations efficiently and shorten the overall processing time. As a method for achieving this, neural networks in which the bit width of the weight coefficients is binary (binarized) are attracting attention.

[0004] Patent Document 1 proposes a configuration for performing product-sum calculations of binary weighting coefficients using a circuit for calculating exclusive OR (XOR), a subtraction circuit, and a bit shift circuit.

[0005] Patent Document 2 proposes an algorithm for calculating a product-sum operation using exclusive negative OR (XNOR) with binary weighting coefficients and binary input data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-060968 A [Patent Document 2] US10311342 publication Summary of the Invention [Problem to be solved by the invention]

[0007] Compared to conventional networks in which feature data and weighting coefficients are stored as real numbers, a network structure in which feature data or weighting coefficients are quantized can be processed with less computational cost, but it is necessary to design the computing units to match the quantization bit width and the possible values ​​of the weighting coefficients.

[0008] The technologies described in Patent Documents 1 and 2 can process neural networks with binarized weighting coefficients. Since the calculations of neural networks with weighting coefficients of ±1 can be performed using exclusive OR (XOR) or exclusive NOR (XNOR), the calculation costs can be reduced.

[0009] The method described in Patent Document 1 uses an exclusive OR (XOR) calculation circuit, a subtraction circuit, and a bit shift circuit to perform product-sum calculations of binary weighting coefficients. The value output from the bit counter needs to be corrected by the subtraction circuit. If the subtraction circuit is performed with the same degree of parallelism as the product-sum calculation process, the circuit size may increase.

[0010] The method described in Patent Document 2 is called XNOR-Net, and can use exclusive negative OR (XNOR) to calculate product-sum operations using binary weight coefficients and binary input data. In the method described in Patent Document 2, both the input data and weight coefficients are 1 bit. XNOR-Net, whose input data and weight coefficients are both 1 bit, cannot be directly applied to a network in which the bit width of the input data is increased to improve recognition accuracy by inference. In the method described in Patent Document 2, the circuit size is reduced by making both the input data and weight coefficients 1 bit, but the amount of information in the network is smaller than when the bit width of the input data is 2 bits or more, so there is a possibility that the recognition accuracy will be lower.

[0011] Furthermore, in order to improve the accuracy of algorithms such as ViT, weighting coefficients with values ​​of 0 and +1 are necessary in addition to weighting coefficients with values ​​of ±1. The techniques described in Patent Documents 1 and 2 can process neural networks with weighting coefficients with values ​​of ±1, but cannot process neural networks with weighting coefficients with values ​​of 0 and +1.

[0012] The present invention has been made in consideration of the above problems, and has an object to reduce the circuit scale when performing product-sum calculations on feature data of 2 bits or more using a 1-bit weighting coefficient. [Means for solving the problem]

[0013] The inference processing device according to the present invention has the following configuration: In a neural network having multiple layers, the device has a product-sum calculation means for outputting a product-sum calculation result between multiple feature map data and a 1-bit weighting coefficient, and a first adding means for adding a correction value for correcting the product-sum calculation result, the correction value being independent of the feature map data. Effect of the Invention

[0014] The present invention can reduce the circuit scale when performing product-sum calculations on feature data of 2 bits or more using a weighting coefficient of 1 bit in neural network processing. [Brief description of the drawings]

[0015] [Figure 1] Neural network processing flow chart [Diagram 2] Example of the structure of the network to be processed [Diagram 3] A block diagram showing an example of the configuration of an inference processing device. [Figure 4] Block diagram of inference unit 305 [Diagram 5] Block diagram of the product-sum calculation unit 406 [Figure 6] An example showing the relationship between networks and convolution processing [Figure 7] Example showing the relationship between the correction unit and the multiply-and-accumulate unit [Figure 8] Neural network processing flow chart [Figure 9] Block diagram of inference unit 305 [Figure 10] An example showing the relationship between networks and convolution processing [Figure 11] Table showing the relationship between modes and multiplication [Figure 12] Multiplier Example [Figure 13] Multiplier Example [Figure 14] Example of feature data with a bit width of 8 bits DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] [First embodiment] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0017] <Example of inference processing device configuration> FIG. 3 is a block diagram showing an example of the configuration of an inference processing device according to the present invention.

[0018] The data storage unit 302 is a section for storing image data. It is usually configured as a hard disk, a flexible disk, a CD-ROM, a CD-R or DVD, a memory card, a CF card, a smart media, an SD card, a memory stick, an xD picture card, a USB memory, or the like. In addition to image data, the data storage unit 302 can also store programs and other data. Alternatively, a part of the RAM 308 described later may be used as the data storage unit 302. Alternatively, the data storage unit 302 may be configured virtually such that a storage device of a device connected via a communication unit 303 described later is used via the communication unit 303.

[0019] The display unit 304 is a device that displays images before and after image processing, or displays images such as a GUI, and is generally a CRT or liquid crystal display, etc. Alternatively, it may be an external display device connected via a cable, etc.

[0020] The input unit 301 is a device for inputting instructions and data from a user, and includes a keyboard, a pointing device, buttons, and the like.

[0021] Alternatively, the display unit 304 and the input unit 301 may be the same device, as in a known touch screen device. In that case, input via the touch screen is treated as input to the input unit 301.

[0022] CPU 306 controls the operation of the entire device. ROM 307 and RAM 308 provide CPU 306 with programs, data, work areas, and the like required for the processing. When a program required for the processing described below is stored in data storage unit 302 or in ROM 307, it is first loaded into RAM 308 and then executed. Alternatively, when the device receives a program via communication unit 303, it is first recorded in data storage unit 302 and then loaded into RAM 308, or it is directly loaded from communication unit 303 into RAM 308 and then executed.

[0023] The image processing unit 309 receives a command from the CPU 306 , reads out the image data written in the data storage unit 302 , adjusts the range of the pixel values, and writes the result back into the RAM 308 .

[0024] The inference unit 305 performs inference processing (steps S101 to S114) including product-sum operations using the image processing results stored in the RAM 308 according to the flowchart of Figure 1 described below, and outputs the processed results to the data storage unit 302 (or RAM 308).

[0025] Based on the result of the neural network processing, image processing or image recognition for the moving image (in the case of multiple frames) is performed by the CPU 306. The result of the image processing or image recognition performed by the CPU 306 is stored in the RAM 308.

[0026] 3, the configuration has only one CPU (CPU 306), but multiple CPUs may be provided. Also, the configuration is not limited to a CPU, and may include a GPU (Graphics Processing Unit) or an NPU (Neural Processing Unit).

[0027] The communication unit 303 is an I / F for communication between devices. Although the input unit 301, data storage unit 302, and display unit 304 are all shown in Fig. 3 as being included in a single device, these units may be connected via a communication path using a known communication method, and may have such a configuration as a whole.

[0028] The system configuration of the device includes various other components in addition to those described above, but as they are not the focus of the present invention, a description of them will be omitted.

[0029] <Network to be processed> An example of the neural network structure to be processed is shown in Figure 2. The arrows represent product-sum operations, quantization, activation processing, etc.

[0030] The network structure includes information about each layer (connections between layers, filter structure, feature map size, bit width, number, weighting coefficients, etc.). This network has four layers (layers 1-4), each of which contains four feature maps, with each feature map containing multiple feature data. The filters and multiple feature data are structured hierarchically. The number of feature maps in a layer is not limited to four, and any number can be used. For example, layer 1 can be an image in RGB format with three maps.

[0031] Figure 2 shows the bit width of the weighting coefficients for each layer of the network to be processed. All weighting coefficients are 1 bit, and there are two possible values ​​for each bit. Two of the three weighting coefficients are set according to the mode (variable m). When the mode is 0, the weighting coefficient value is -1 or +1. When the mode is 1, the weighting coefficient value is 0 or +1. By providing two modes, the number of weighting coefficient values ​​processed by the network increases to three, -1, 0, and +1, improving the expressiveness of information and the accuracy of recognition by inference. If there are only two weighting coefficient values, ±1, the weighting coefficient that would be learned as 0 when there are three weighting coefficient values ​​is approximated to either ±1, which is one of the factors that reduces the accuracy of inference.

[0032] The calculations of layers 1-4 will be described. In layer 1 (m=0), a product-sum operation result is calculated using multiple feature maps 201 and a weighting coefficient with a value of -1 or +1 based on formula 1 described later, to generate multiple feature maps 202 in layer 2. In layer 2 (m=1), a product-sum operation result is calculated using multiple feature maps 202 and a weighting coefficient with a value of 0 or +1 based on formula 1, to generate multiple feature maps 203 in layer 3. In layer 3 (m=0), a product-sum operation result is calculated using multiple feature maps 203 and a weighting coefficient with a value of -1 or +1 based on formula 1, to generate multiple feature maps 204 in layer 4.

[0033] Figure 6 shows an example of a network and convolutional processing. Feature data is extracted from the same position of four feature maps 601 in layer 1, and product-sum processing is performed to calculate the result of activation processing. The result is a feature at the same position in layer 2, which becomes the feature data of image 602. The network processing is not limited to convolutional processing, and may include matrix operations and other product-sum processing.

[0034] <Flowchart of neural network processing in the present invention> 4 shows the configuration of the inference unit 305. The inference unit 305 is made up of a control unit 401, a memory 402, a memory bus 403, a product-sum calculation processing unit 406, an addition unit 407, an activation pooling processing unit 408, and a correction unit 410.

[0035] The product-sum calculation processing unit 406 is composed of a plurality of bit calculation units (1-bit logical operation circuits) 405. The product-sum calculation processing unit 406 calculates the multiplication result and the cumulative addition result of the multiplication results, and outputs the product-sum calculation result. In order to reduce the circuit scale for the multiplication process, in this embodiment, the multiplication process is substituted by a 1-bit logical operation circuit rather than a multiplier that performs multi-bit operations. However, since the operation using the weighting coefficient -1 is processed by logical operations of 0 and 1, correction is required in post-processing using a correction value. Details will be described later.

[0036] The product-sum calculation unit 406 calculates a product-sum calculation result from the weighting coefficients and feature data based on Equation 1, which will be described later. The weighting coefficients input to the multiplication unit 406 have a bit width of 1 bit. The activation pooling unit 408 calculates the activation pooling processing result based on the product-sum calculation result.

[0037] Each step of the flowchart shown in Fig. 1 will be described based on the configuration of inference unit 305 shown in Fig. 3. Steps S101 to S114 are control operations of the CPU or sequencer included in control unit 401.

[0038] In step S 101 , the control unit 401 reads out pixels (feature data) of a plurality of input feature maps or pixels of an input image, weighting coefficients, and network structure information 409 from the RAM 308 , and stores them in the memory 402 via the memory bus 403 .

[0039] In step S102, the control unit 401 starts a loop through the layers and processes the first layer (layer 1). Layer 1 becomes the layer to be processed.

[0040] In step S103, the control unit 401 sets the number of input / output channels (the number of feature maps) and the types of weighting coefficients according to the network structure information 409 stored in the memory 402. In this example, the bit width of the weighting coefficients in the same layer is all 1 bit, and there are two types.

[0041] In step S104, the control unit 401 starts a loop on the output feature map and calculates the output feature data in order.

[0042] In step S105, the control unit 401 performs initialization by setting the product-sum operation result held in the product-sum operation processing unit 406 to zero.

[0043] In step S106, the control unit 401 starts a loop on the input feature map and processes the input feature data in order.

[0044] In step S107, the control unit 401 reads out a part of the input feature map from the memory 402, and transfers it to the feature data storage unit 402. Then, the control unit 401 reads out a part of the weighting coefficients from the memory 402, and transfers them to the correction unit 410 and the product-sum calculation processing unit 406.

[0045] In step S108, the correction unit 410 receives a control signal from the control unit 401 and calculates a correction value that is not dependent on the feature map. The calculation formula for the product-sum calculation (Formula 1) can be divided into a part that is dependent on the feature map and a part that is not dependent on the feature map (Formula 4). The correction value is the calculation result of the part that is not dependent on the feature map, and the correction value can be calculated independently from the product-sum calculation process of the feature map and the weighting coefficient. The main purpose of the inference processing device is not to learn the network, but to perform inference processing, and the input image and the feature map are updated more frequently than the weighting coefficient. In many cases, even if the input image that is the subject of the inference processing changes, a common weighting coefficient is used. When a common weighting coefficient is used, the calculation of the correction value that is not dependent on the feature map is processed by a circuit with a different calculation efficiency from that of the calculation that is dependent on the feature map, thereby making it possible to reduce the calculation cost. The circuit scale can be reduced by reducing the parallelism of the circuit that calculates the correction value that is not dependent on the feature map. The calculation formula for the correction value is Formula 5, which will be described later.

[0046] In step S109, the product-sum operation processing unit 406 receives a control signal from the control unit 401, and calculates a product-sum operation result based on the stored input feature data, weighting coefficients, and the type of weighting coefficient (mode represented by variable m) using Equation 1. "otherwise" in Equation 1 means the case where m=1.

[0047]

number

[0048] FIG. 11 shows a table of the relationship between modes and multiplication. The variable a' is a pixel of the feature map and is multiple bits. The variables a'i,j are parts of the pixel of the feature map divided into 1 bit and have a value of 0 or 1. The variables w' and w are weighting coefficients and are selected by the value of the variable m. When the variable m is set to 0, the weighting coefficient w with a value of -1 or +1 is used as shown in table 1101. When the variable m is set to 1, the weighting coefficient w' with a value of 0 or +1 is used as shown in table 1102. By using two types of binary weighting coefficients in different layers of the same network, three types of weighting coefficients are used in the entire network, which can improve the recognition accuracy by inference. The variable i is an index when performing product-sum calculation processing, and there are I pixels and I weighting coefficients. The variable β is a bias value (or offset value) used when normalizing or quantizing.

[0049] A pixel (feature data) of the feature map has J bits, and a weighting coefficient has 1 bit. The feature data and the weighting coefficient have different bit widths. Figure 14 shows an example of feature data with a bit width of 8 bits (J=8). When decimal feature data a'i with a value of 214 is converted to binary, it becomes an 8-digit value (11010110). That is, feature data a'i can be expressed with 8 bits (a'i,0, a'i,1, ..., a'i,7). The variables a'i,j represent the bits of each digit. The variable a'i,0 is the least significant bit and has a value of 0. The variable a'i,7 is the most significant bit and has a value of 1. In this example, the sign bit is not included in the 8-bit feature data.

[0050] When the variable m is 0, the first type of binary weighting coefficient is selected. When the pixel a' of the feature map is expressed by J-bit data, the product-sum operation process of Equation 1 can be expressed by Equation 2.

[0051]

number

[0052] The variable a'i,j is a part of the pixel of the feature map divided into 1 bit, and the value is 0 or 1. The variable j is the index of the bit of the pixel. For example, when J=8, the 0th to 7th digits (j=0~7) are multiplied by the weight coefficient and bit operation in order, and shifted from 0th power of 2 to 7th power of 2, and then accumulated. The accumulation operation in parentheses in Equation 2 is the multiplication and accumulation of 1 bit of data and 1 bit of weight coefficient. When performing calculations in a digital circuit, data is stored as 0 and 1, so the weight coefficient w' with a value of -1 or +1 must be expressed as a weight coefficient w with a value of 0 or +1. The accumulation operation in parentheses in Equation 2 can be rewritten as Equation 3.

[0053]

number

[0054] The variables ai,j are the part of the pixels of the transformed feature map and have values ​​of -1 or 1. They are used as intermediate variables.

[0055]

number

[0056] means exclusive negative OR (XNOR). By substituting the result of Equation 3 into Equation 2, the multiplication of a multi-bit pixel (synonymous with feature data of 2 or more bits) and a 1-bit weighting coefficient can be expressed by Equation 4.

[0057]

number

[0058] Since the multiplication and accumulation process can be substituted by the bit logic operation and addition of XNOR, there is no need to provide a multiplier, and the circuit scale can be reduced. The variable γ, which does not depend on the feature map, is the correction value and is calculated using Equation 5.

[0059]

number

[0060] Since the correction value γ is calculated only using the weighting coefficients and does not change depending on the pixel values ​​of the feature maps, it is possible to perform the process of calculating the correction value γ separately from the product-sum calculation process. In other words, the correction value used in the product-sum calculation process for pixels of multiple feature maps is the same.

[0061] The correction unit 410 receives a control signal from the control unit 401 and calculates a correction value γ in step S108.

[0062] When the variable m is 1, the second type of binary weighting coefficient is selected. When pixel a' in the feature map is expressed as J-bit data, the product-sum operation process in Equation 1 can be expressed as Equation 6. Unlike when the variable m is 0, multiplication can be substituted with logical product (AND), so the correction value is 0.

[0063]

number

[0064] In 1102 of FIG. 11, the variable a' is a pixel of the feature map and is multiple bits. The variables a'i,j are part of the pixel of the feature map divided into 1 bit and have a value of 0 or 1. The variable w' is a weighting coefficient and has a value of 0 or 1. The variable i is an index when performing product-sum calculation processing, and there are I pixels and I weighting coefficients. The variable β is a bias value when normalizing or quantizing.

[0065] FIG. 5 shows the structure of bit operation unit 405. It is illustrated in detail in stages. Bit operation unit 405 is made up of multiple (J) logic operation units, and combines the output results of each logic operation unit to output the result of a bitwise operation (operation for each bit). The output of the logic operation unit differs depending on the value of variable m. When variable m is 0, the result of the XNOR operation in equation 4 is output from the logic operation unit. When variable m is 1, the result of the AND operation in equation 6 is output from the logic operation unit.

[0066] In step S110, the control unit 401 determines whether the loop of the input feature map has ended. If all input feature data has been processed, the process proceeds to step S111. If not, the process returns to step S107 and starts processing the next input feature map.

[0067] In step S111, the product-sum calculation processing unit 406 transfers the product-sum calculation result to the adder 407, which receives a control signal from the control unit 401 and calculates the sum of the correction value γ in equation 4, the bias β in equation 4 or equation 6, and the product-sum calculation result. Then, the activation / pooling processing unit 408 receives a control signal from the control unit 401 and performs activation processing based on the product-sum calculation result equation 4 held in the adder 407. The activation processing result is calculated using equation 7.

[0068]

number

[0069] f(·) is the activation function, and x is the input data (output from the adder 407). The result of the activation process is converted into a J-bit feature map. In this example, the activation function is realized using ReLU (Rectified Linear Unit), but it is not limited to ReLU and may be realized by other nonlinear functions, or a quantization function that quantizes the product-sum operation result, the correction value, and the bias value to J bits.

[0070] In addition, the activation and pooling unit 408 performs pooling processing based on the result of activation processing according to the layer information, and adjusts the size of the output feature map as necessary.

[0071] In step S112, the control unit 401 stores the activation pooling process result in the feature data storage unit 402, and handles it as a feature map of the next layer.

[0072] In step S113, the control unit 401 determines whether the loop of the output feature map is finished. If the processing of all the output feature maps is finished, the control unit 401 proceeds to step S114. If not, the control unit 401 returns to step S105 and starts processing the next output feature map.

[0073] In step S114, the control unit 401 determines whether the layer loop has ended. If all layers have been processed, the processing of the neural network ends. If not, the process returns to step S103, the processing target layer is changed, and processing of the next layer is started.

[0074] In this embodiment, the multiplication and accumulation of a multi-bit (J-bit) pixel (feature data) and a 1-bit weighting coefficient can be efficiently performed by using a circuit configuration in which the calculation of the correction value γ is separated from the multiplication and accumulation. In addition, since a network including three types of weighting coefficients with values ​​of 0 and ±1 can be processed, it is possible to process a network with higher information expression power and improve recognition accuracy. Furthermore, since the same circuit configuration can process a network including weighting coefficients with values ​​of 0 and ±1, it is possible to reduce the circuit scale.

[0075] <Adjusting the degree of parallelism> FIG. 7 shows an example of a correction unit 410 and a product-sum calculation processor with different parallelism. There is one correction unit 410, and P product-sum calculation processors 702 and adders 703. The product-sum calculation processor 702 has the same function as the product-sum calculation processor 406, and the adder 703 has the same function as the adder 407. As in the configuration shown in FIG. 4, the product-sum calculation processor 702 transfers the product-sum calculation result to the adder 703. After P pieces of feature data and one common weighting factor are read via the memory bus 403, the processor 702 and the adder 703 calculate the P pieces of feature data and one weighting factor in parallel. In order to improve the processing speed, it is necessary to increase the parallelism of the circuit that performs the product-sum calculation of multiple pixels, but it is not necessary to increase the parallelism of the circuit that calculates the correction value. By separating the calculation of the correction value and the product-sum calculation, there is an effect that the circuit scale of the product-sum calculation with high parallelism can be reduced. In an embedded device such as a digital camera, it is desirable to be able to reduce the circuit scale.

[0076] [Second embodiment] <Calculate the correction value before the inference process> In the first embodiment, an example in which the correction value is calculated in step S108 has been described, but the correction value may be calculated before inference begins. The other hardware configurations are the same as those in the first embodiment, and therefore will not be described.

[0077] 9 shows the configuration of the inference unit 305. The inference unit 305 is composed of a control unit 401, a memory 402, a memory bus 403, a product-sum calculation processing unit 406, an addition unit 407, and an activation / pooling processing unit 408. A correction unit 410 is not included in the inference unit 305. The CPU 306 shown in FIG. 3 performs the function of the correction unit 410.

[0078] Each step in the flowchart shown in Fig. 8 will be described based on the configuration of the inference unit 305 shown in Fig. 9. Steps that differ from the first embodiment will be described.

[0079] In step S801, the correction unit 410 receives a control signal from the control unit 401, reads out the weighting coefficient and the network structure information 409 from the RAM 308, and calculates the correction value γ based on the formula 5. Then, the sum of the correction value γ and the bias β is calculated, and the bias β is replaced with this sum. The updated bias β' is stored in the memory 402 as part 901 of the new network structure. The value of the updated bias β' is calculated using the formula 8. It differs depending on the mode of the weighting coefficient (variable m).

[0080]

number

[0081] Since the correction value is included in the updated bias β', step S108 in the first embodiment is omitted, and there is no need to recalculate the correction value before performing the product-sum calculation process in step S109.

[0082] In step S 101 , the control unit 401 reads out feature data of a plurality of input feature maps, weighting coefficients, network structure information 409 , and a part 901 of a new network structure from the RAM 308 , and stores them in the memory 402 via the memory bus 403 .

[0083] In step S111, the adder 407 receives a control signal from the control unit 401 and calculates the updated bias (corrected bias) β′ in Equation 8 and the sum of the product-sum calculation results.

[0084] In this embodiment, the correction value is calculated before the inference process and is stored in advance in the RAM 308. By removing the circuit for calculating the correction value from the inference unit 305, it is possible to reduce power consumption and the circuit size.

[0085] [Other embodiments] <Applications of neural networks> In the first and second embodiments, an example in which the inference processing device is applied to image processing has been described, but the inference processing device may also be applied to applications other than image processing (such as voice recognition).

[0086] <Other than two dimensions> In the first embodiment, although an example applied to a neural network of two-dimensional image data has been described, it may be applied to a neural network of one-dimensional audio data or any data of three dimensions or more. <XOR (Exclusive OR)> In the first and second embodiments, although an example in which the multiplication-accumulation operation process can be substituted with the logical operation of XNOR bits and addition has been described, since XNOR can be decomposed into NOT (logical inversion), the logical operation of bits can also be realized by a combination of XOR (Exclusive OR) and NOT (logical inversion).

[0087] <Addition of correction value> In the first embodiment, in step S111, it has been described that the addition unit 407 receives a control signal from the control unit 401 and calculates the sum of the multiplication-accumulation operation result and the correction value based on Equation 4, but it is not necessarily limited to being performed by the addition unit 407. Equation 4 can be rewritten as Equation 9. The part other than the calculation of XNOR and the addition of the bias β in the equation corresponds to the calculation of the correction value. The calculation of the correction value may be partially performed by the multiplication-accumulation operation processing unit 406.

[0088]

Equation

[0089] Similar to the first embodiment, FIG. 10 shows examples of the correction unit 410 and the multiplication-accumulation operation processing unit with different parallelism degrees. There is one correction unit 410, and there are P each of the multiplication-accumulation operation processing units 1001 and the addition units 1002. Different from FIG. 6, the correction value is transferred to the multiplication-accumulation operation processing unit instead of the addition unit. Even if the parallelism degree of the circuit for performing the multiplication-accumulation operation processing of a plurality of pixels is increased, it is not necessary to increase the parallelism degree of the circuit for calculating the correction value. By separating the calculation of the correction value and the multiplication-accumulation operation processing, the circuit scale of the multiplication-accumulation operation processing with high parallelism can be reduced.

[0090] A modified example of the multiplication process is shown in Fig. 12. The result of the bitwise XNOR of the weighting coefficient and feature data is calculated, and then the result is bit-concatenated with the logically inverted 1-bit weighting coefficient. Then, the sum of the result of the bit-concatenation with the logically inverted 1-bit weighting coefficient is calculated, and cumulative addition is performed. In the multiplication process of the first embodiment shown in Fig. 5, no correction value is calculated, but in the multiplication process shown in Fig. 12, a correction value is also calculated.

[0091] The calculation result of Equation 9 can be output.

[0092] <Change bit operations to multipliers or multiplexers> In the first embodiment, the weight coefficient is described as being substituted for multiplication by a bit operation, but this is not limited to bit operations. The multiplication can also be performed by a multiplier or a multiplexer (selector) instead of being substituted for a bit operation.

[0093] When the variable m is set to 0, the feature data is expressed in two's complement and the product-sum calculation formula when the weighting coefficient w has a value of −1 is Equation 10.

[0094]

number

[0095] The overline of the variable a' means logical inversion (NOT). The result of logical inversion includes a sign bit. Also, it is necessary to deal with overflow and truncation, but this is omitted in Equation 10. In this case, the correction value is the number I of weighting coefficients.

[0096] Figure 13 shows an example of substituting a multiplexer for multiplication. Select positive feature data and negative feature data. The calculation result of Equation 10 can be output.

[0097] <Calculate the correction value outside the inference device> In the first and second embodiments, the correction value is calculated inside the device, but it may be calculated outside the device. In this case, the correction value is included in the network structure information.

[0098] Unlike the second embodiment, in this embodiment, the correction value and the updated bias β′ value are calculated in advance outside the device based on Equation 8. In step S801, instead of calculating the correction value, the weighting coefficient and new network structure information 409 are read from RAM 308, and the bias β′ including the correction value is stored in memory 402.

[0099] In this embodiment, the correction values ​​are calculated by an external device and stored in the storage means together with the network structure information, so there is no need to calculate the correction values ​​inside the device, and the circuit scale can be reduced.

[0100] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable recording medium. [Explanation of symbols]

[0101] This section provides an explanation of the symbols used to indicate the main parts shown in the drawings. S101: A step of storing input feature data, weight coefficients, and network structure information. S102 Layer loop start step S103: Step of setting the bit width of the weighting coefficient S104: Starting a loop of the output feature map S105: Step for initializing the multiply-and-accumulate result S106: A step for starting a loop of input feature maps S107: A step of transferring input feature data and weight coefficients S108 Step for calculating correction value S109: A step for performing multiply-and-accumulate processing S110: A step of determining whether the loop of the input feature map is completed S111: Step for adding bias and correction values, and performing activation and pooling processing S112: A step of storing the result of the activation pooling process in memory. S113: A step for determining whether the loop of the output feature map is completed S114: A step for determining the end of the layer loop

Claims

1. a multiply-and-accumulate calculation means for outputting a result of a multiply-and-accumulate operation between data of a plurality of feature maps and a 1-bit weighting coefficient in a neural network having a plurality of layers; an addition means for adding a correction value for correcting the result of the product-sum operation, the correction value being independent of the data of the feature map, to the result of the product-sum operation.

2. 2. An inference processing apparatus according to claim 1, further comprising a correction means for calculating said correction value independently of said product-sum calculation means.

3. 3. The inference processing device according to claim 2, wherein said correction means processes a smaller number of data items in parallel than said multiply-and-accumulate means.

4. Further comprising a retaining means; 2. An inference processing apparatus according to claim 1, wherein said storage means stores said correction value calculated by an external device and neural network structure information.

5. Further comprising a retaining means; the holding means holds the correction value calculated before performing the product-sum operation; the adding means reads out the correction value from the holding means and performs addition; 2. An inference processing apparatus according to claim 1, comprising no correction means for calculating said correction value.

6. 2. An inference processing apparatus according to claim 1, wherein the values ​​of the weighting coefficients are +1 and -1, or 0 and +1.

7. 2. The inference processing device according to claim 1, wherein the weighting coefficients used in said multiply-and-accumulate means vary for each hierarchy level, either in a mode of +1 and -1 or in a mode of 0 and +1.

8. 2. The inference processing device according to claim 1, wherein said multiplication and accumulation calculation means calculates the multiplication result by a bit logic operation.

9. 2. The inference processing device according to claim 1, wherein the bit logical operation performs a one-bit logical operation for each digit value of the binarized data of the feature map.

10. The correction means calculates a correction value using the weighting coefficient, 3. The inference processing device according to claim 2, wherein said adding means calculates the sum of said product-sum operation result and said correction value.

11. 2. The inference processing device according to claim 1, wherein the correction value includes a bias.

12. 2. The inference processing device according to claim 1, wherein in said sum-of-products calculation means, multiplication is performed by a bit logic operation circuit, and either a logical product (AND) or an exclusive logical sum (XOR) is output according to neural network structure information.

13. 11. The inference processing device according to claim 10, wherein the correction value varies depending on whether the output of a bit logic operation circuit is a logical product (AND) or an exclusive OR (XOR).

14. 11. The inference processing device according to claim 10, wherein the bit logic operation circuit outputs either a logical product (AND) or an exclusive logical sum (XOR) in accordance with the neural network structure information.

Images