Data processing equipment, its operating methods, and computer-readable storage media

Abstract

Embodiments of this disclosure relate to a data processing apparatus, its operating method, and a program. The data processing apparatus includes: an input data determining unit configured to determine whether each binary input data in binary input data is a predetermined value; a storage unit configured to store a plurality of coefficients and coefficient address information, the coefficient address information including information related to the coefficient addresses where the plurality of coefficients are stored; a control unit configured to read coefficient addresses from the storage unit based on the determining result of the input data determining unit, and to read coefficients from the storage unit based on the coefficient addresses; and an arithmetic unit configured to perform arithmetic operations related to the coefficients obtained by the control unit.

Description

[0001] Cross-references to related applications

[0002] The disclosure of Japanese Patent Application No. 2019-200609, filed on November 5, 2019, including its description, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to a data processing apparatus and its operating methods and procedures, and to a data processing apparatus and its operating methods and procedures configured, for example, to perform multiple accumulations using floating-point weighting coefficients in processing performed via a neural network. Background Technology

[0004] In recent years, some embedded devices have emerged that perform inference processing using artificial intelligence (AI). Various neural networks are used for this inference process. Examples of neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), multilayer perceptrons, recurrent neural networks, and so on. Inference processing via neural networks requires significant memory and arithmetic resources to perform numerous multiple accumulations.

[0005] Regarding arithmetic operation resources, multipliers have a larger circuit size than adders and consume a significant amount of power during operation. Therefore, methods have been proposed that use floating-point addition instead of floating-point multiple accumulation, by changing the input data of the multiple accumulation matrix used in neural network operations to a combination of binary data, where floating-point multiple accumulation requires a large amount of computation.

[0006] On the other hand, regarding memory capacity, since the weight parameters are floating-point data, there is a need for large memory capacity to store them. This is a significant issue, especially for embedded microcomputers and SoCs (System-on-a-Chip) with limited memory capacity.

[0007] The following are the technologies that have been disclosed.

[0008] [Non-patent document 1] Speeding-up of Identification Calculation and ModelCompression by Binarized-DCNN, Technical Committee on Pattern Recognition and Media Understanding, 2016. Summary of the Invention

[0009] According to Non-Patent Document 1, a coefficient information is given a bit of base information. The original weight parameter is then approximated by the sum of values ​​obtained by multiplying each of the H coefficient information segments (H is a natural number) by the value 1 or -1. Therefore, a problem arises: the expressible weight parameter is limited to the sum of combinations of positive (+) or negative (-) values ​​of the H coefficients. For example, even if the number of base information segments doubles from H to 2H, the possible coefficient information only doubles. Furthermore, since the expressible weight parameter is limited to the sum of values ​​obtained by multiplying the H coefficient information segments by +1 or -1, the distribution of possible values ​​of the weight parameter is restricted to being symmetrical between the positive (+) and negative (-) sides. In particular, when distorted input values ​​are fed into the neural network, the distortion of the value distribution is accumulated each time multiple accumulation is performed, leading to a decrease in recognition accuracy during inference processing.

[0010] One objective of the embodiments is to achieve high recognition accuracy while suppressing the memory requirements associated with multiple accumulations in machine learning inference processes involving neural networks. Other objectives and novel features will become apparent from the description in the specification and figures.

[0011] A data processing apparatus according to one embodiment includes: an input data determining unit configured to determine whether each binary input data in binary input data is a predetermined value; a storage unit configured to store a plurality of coefficients and coefficient address information, the coefficient address information including information related to the coefficient addresses of the plurality of coefficients stored; a control unit configured to read coefficient addresses from the storage unit based on the determining result of the input data determining unit, and to read coefficients from the storage unit based on the coefficient addresses; and an arithmetic unit configured to perform arithmetic operations related to the coefficients obtained by the control unit.

[0012] An operation method of a data processing apparatus according to another embodiment is a method of operating a data processing apparatus having an arithmetic unit and a memory, and the method includes: a step of determining whether binary input data is a predetermined value; a step of storing a plurality of coefficients and coefficient address information in the memory, the coefficient address information including information related to the coefficient addresses of the plurality of coefficients stored; a step of reading coefficient addresses from the memory based on the determination result of the determination step; a step of reading coefficients from the memory based on the coefficient addresses; and a step of performing arithmetic operations by the arithmetic unit based on the read coefficients.

[0013] According to another embodiment, a computer-readable storage medium storing a program is a computer-readable storage medium storing a program relating to a neural network model executed by a data processing device having a CPU and a memory, and the computer-readable storage medium storing the program causes a computer to perform a process including the following steps: determining whether each of the binary input data in the binary input data is a predetermined value; storing a plurality of coefficients and coefficient address information in memory, the coefficient address information including information relating to coefficient addresses including addresses where a plurality of coefficients are stored; reading coefficient addresses from memory; reading coefficients from memory based on the coefficient addresses; and performing a cumulative addition of the read coefficients.

[0014] According to one embodiment, the data processing device can achieve high recognition accuracy while suppressing memory usage in the inference processing of machine learning, which includes deep neural networks. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating a configuration example of a data processing device according to the first embodiment;

[0016] Figure 2 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the first embodiment;

[0017] Figure 3 This is an explanatory diagram of binary input data stored in the input data storage unit of the data processing device according to the first embodiment;

[0018] Figure 4 This is a diagram illustrating an example of processing of a neural network performed by the inference unit of the data processing apparatus according to the first embodiment;

[0019] Figure 5 This is a diagram illustrating an example of multiple accumulations of first input data by a data processing device according to the first embodiment;

[0020] Figure 6 This is a diagram illustrating an example of multiple accumulation of second input data by a data processing device according to the first embodiment;

[0021] Figure 7 This is a diagram illustrating an example of multiple accumulation of third input data by a data processing device according to the first embodiment;

[0022] Figure 8 This is a diagram illustrating an example of multiple accumulation of fourth input data by a data processing device according to the first embodiment;

[0023] Figure 9 This is a diagram illustrating an example of a multiple accumulation process performed by a data processing device according to the first embodiment;

[0024] Figure 10 This is a flowchart illustrating an example of a multiple accumulation process performed by a data processing device according to the first embodiment;

[0025] Figure 11 This is a diagram illustrating an example of the coefficients stored in the coefficient storage unit according to the first embodiment;

[0026] Figure 12 This is a diagram illustrating how the distribution of output data is improved by the data processing device according to the first embodiment;

[0027] Figure 13 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the second embodiment;

[0028] Figure 14 This is a diagram illustrating an outline of the multiple accumulation processing performed by the data processing apparatus according to the second embodiment;

[0029] Figure 15 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the third embodiment;

[0030] Figure 16 This is an explanatory diagram illustrating the concept of multiple accumulation processing performed by the data processing device according to the third embodiment;

[0031] Figure 17 This is a flowchart illustrating an example of a multiple accumulation process performed by a data processing device according to a third embodiment;

[0032] Figure 18 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the fourth embodiment;

[0033] Figure 19 This is a diagram illustrating an example of the coefficients stored in the coefficient storage unit according to the fourth embodiment;

[0034] Figure 20 This is a diagram illustrating how the distribution of output data is improved by the data processing device according to the fourth embodiment;

[0035] Figure 21 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the fifth embodiment;

[0036] Figure 22This is an explanatory diagram of coefficient adjustment performed by the data processing device according to the fifth embodiment based on the most frequent coefficient address;

[0037] Figure 23 This is a flowchart illustrating an example of a multiple accumulation process performed by a data processing device according to the fifth embodiment;

[0038] Figure 24 This is a diagram illustrating an example of a method for quantizing input data of a neural network into multiple bits according to the sixth embodiment;

[0039] Figure 25 This is a circuit diagram illustrating an example configuration of the inference unit and the storage device of the data processing apparatus according to the sixth embodiment;

[0040] Figure 26 This is an explanatory diagram illustrating the concept of multiple accumulation processing performed by the data processing device according to the sixth embodiment;

[0041] Figure 27 This is a flowchart illustrating an example of a multiple accumulation process performed by a data processing device according to the sixth embodiment; and

[0042] Figure 28 This is an illustration of the distribution of weight parameters under conventional techniques. Detailed Implementation

[0043] For clarity, the following descriptions and figures are omitted and simplified as needed. Additionally, each element of the functional blocks described in the figures for performing various processes can be configured in hardware by a CPU (Central Processing Unit), memory, and other circuitry, and can be implemented in software by programs loaded in memory, etc. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms, solely by hardware, solely by software, or a combination thereof, and are not limited to any of these. In each figure, the same elements are indicated by the same reference numerals, and redundant descriptions are omitted as needed.

[0044] Furthermore, the aforementioned programs are stored using various types of non-transitory computer-readable media and can be provided to a computer. Non-transitory computer-readable media include various types of physical storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read-Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), Flash ROMs, and RAMs (Random Access Memory)). Programs can be provided to a computer via various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can provide programs to a computer via wired communication paths such as wires and optical fibers, or wireless communication paths.

[0045] (First embodiment)

[0046] (Configuration)

[0047] Figure 1 This is a block diagram illustrating a configuration example of the data processing device 1 according to the first embodiment. For example... Figure 1 As shown, the data processing device 1 includes a processor 10, a ROM 11, a bus 12, a RAM 13, and an inference unit 14. For example, the data processing device 1 can be configured as a semiconductor device.

[0048] Processor 10 is a processing circuit that performs arithmetic processing by executing a program (instruction stream), which is read from ROM 11, etc., via bus 12. For example, the processor is a CPU, DSP (digital signal processor), etc.

[0049] ROM 11 is a storage device that stores programs to be executed by processor 10. For example, ROM 11 stores programs related to a deep neural network obtained through machine learning by performing deep learning, and programs for controlling the inference unit 14. Although in Figure 1 In this embodiment, ROM 11 is incorporated into data processing device 1, but data processing device 1 can be configured to perform processing by reading a program from a storage device provided outside of data processing device 1. Furthermore, the storage device for storing the program can be either ROM 11 incorporated into data processing device 1 or a storage device provided outside of data processing device 1.

[0050] Bus 12 is a signal path connecting processor 10, ROM 11, RAM 13, and inference unit 14 to exchange data. Processor 10 and inference unit 14 access ROM 11 or RAM 13 via bus 12.

[0051] RAM 13 is a storage device (storage unit) that stores data for the processor 10 and the inference unit 14 to perform arithmetic processing. The inference unit 14 can store temporary data, etc., required for neural network operations in RAM 13.

[0052] Inference unit 14 uses a neural network to perform inference processing. An example of a neural network processed by inference unit 14 is a deep neural network (DNN), but the neural network is not limited to this. For example, the neural network can be a CNN or an RNN.

[0053] Although Figure 1 An example configuration of the inference unit 14 is shown, but the configuration of the data processing device 1 is not limited to this. That is, the data processing device 1 can be configured such that the inference unit 14 is implemented as software and the program is stored in the ROM 11. Furthermore, the inference unit 14 may have a storage device (e.g., ROM or RAM) therein, which is separate from the ROM 11 and RAM 13.

[0054] Figure 2 This is a diagram illustrating an example configuration of the inference unit 14 and RAM 13 according to the first embodiment. Figure 2 As shown, the inference unit 14 includes an input data determination unit 140, a floating-point unit (FPU) 141, and a control unit 142. Furthermore, the RAM 13 includes an input data storage unit 130, a coefficient address information storage unit 131, and a coefficient storage unit 132. Although... Figure 2 The illustration shows a configuration in which coefficient address information storage unit 131 and coefficient storage unit 132 are provided in RAM 13, but the configuration of coefficient address information storage unit 131 and coefficient storage unit 132 is not limited to this. For example, the following configuration is also possible: in which coefficient address information storage unit 131 and coefficient storage unit 132 are provided in ROM 11.

[0055] The input data storage unit 130 is a storage unit that stores binary input data used to perform inference through the neural network. The data input to the neural network is floating-point data. The input floating-point data is quantized into a 1-bit digital value by a quantization unit (not shown) and stored in the input data storage unit 130 as binary input data. In other words, the floating-point input data input to the neural network is quantized into a value of 0 or 1 and stored in the input data storage unit 130. Figure 3 This is a diagram illustrating an example of floating-point input data being quantized into binary input data. Figure 3 In the example shown, the input values ​​are distributed from 0 to max (maximum value). Figure 3In the example shown, values ​​greater than or equal to 0 and less than th = max / 2 are quantized to 0, and values ​​greater than or equal to th = max / 2 and less than or equal to max are quantized to 1. Figure 3 In the quantization example shown, the threshold used for binary conversion is th = max / 2, but the method for quantizing input data into binary data is not limited to this. For example, by setting the threshold to th = max / 4, the input data can be quantized into binary data. When the threshold is th = max / 4, values ​​greater than or equal to 0 and less than th = max / 4 are quantized as 0, and values ​​greater than or equal to th = max / 4 and less than or equal to max are quantized as 1.

[0056] Refer again Figure 2 The coefficient address information storage unit 131 is a storage unit in which information related to coefficient addresses is stored, which are the addresses where the coefficients of the neural network are stored (hereinafter referred to as "coefficient address information"). In this case, the coefficient address is the address value of the coefficient stored in the coefficient storage unit 132. When the coefficient address includes K bits (K is a natural number), up to 2K different coefficients can be identified. In other words, the inference unit 14 can use up to 2K different coefficients as coefficients of the neural network. On the other hand, the coefficient address information includes one or more coefficient addresses, and also includes information about the order of use (operation order) of the one or more coefficient addresses. In other words, the coefficient address information includes information consisting of one or more coefficient addresses corresponding to the fact that each of the one or more input data and each of the one or more weighting parameters is multiplied in the multiple accumulation of the neural network. Therefore, the coefficient address information includes information about which input data the one or more coefficients are multiplied with. The coefficient address information can be expressed as, for example, an array of one or more coefficient addresses. (Refer to...) Figure 5 This section provides a detailed description of specific examples of coefficient addresses and coefficient address information.

[0057] The coefficient storage unit 132 is a storage unit in which the coefficients of the neural network are stored.

[0058] The input data determination unit 140 determines whether the binary input data is a predetermined value. More specifically, the input data determination unit 140 determines whether the binary input data is 1, where 1 is a predetermined value.

[0059] FPU 141 is an arithmetic circuit that performs floating-point operations. As described later, FPU 141 performs multiple accumulation of input data and coefficients by performing a cumulative addition of coefficients read from coefficient storage unit 132.

[0060] Control unit 142 is a control circuit that controls the transmission and reception of binary input data, coefficient addresses, and coefficients between input data storage unit 130, coefficient address information storage unit 131, coefficient storage unit 132, input data determination unit 140, and FPU 141. More specifically, control unit 142 reads binary input data from the input data storage unit and sends the read input data to input data determination unit 140. Furthermore, based on the determination result of input data being 1 in input data determination unit 140, control unit 142 reads coefficient addresses from coefficient address information storage unit 131. Additionally, based on the read coefficient addresses, control unit 142 reads coefficients from coefficient storage unit 132 and sends these coefficients to FPU 141.

[0061] After reading all input data and performing cumulative addition, the control unit 142 stores the result of the cumulative addition as a multiple accumulation result in the FPU 141 in RAM 13.

[0062] (operate)

[0063] In the data processing apparatus 1 according to the first embodiment, before inference processing is performed using a neural network, learning data is used to perform the process, and optimal weight parameters are obtained. In the data processing apparatus 1 according to the first embodiment, representative L coefficients (L is a natural number) are selected from the distribution of weight parameters obtained during the learning period. The selected L coefficients are stored in coefficient storage unit 132. Here, the L coefficients are floating-point data. Furthermore, in the data processing apparatus 1 according to the first embodiment, coefficient address information is stored in coefficient address information storage unit 131, and the coefficient address information is information related to the coefficient addresses of the L coefficients. For example, the coefficient address is a relative address relative to the base address of the coefficient storage unit 132. By using the value of the relative address as the coefficient address, the addresses of the L coefficients can be expressed with fewer bits.

[0064] In the data processing apparatus 1 according to the first embodiment, K representative values ​​from the distribution of weight parameters obtained through learning are pre-selected and stored in the coefficient storage unit 132. As described above, in the data processing apparatus 1 according to the first embodiment, the weight parameters are expressed using a combination of the coefficients stored in the coefficient storage unit 132 and the coefficient address information stored in the coefficient address information storage unit 131, and the weight parameters are floating-point data.

[0065] Figure 4 This is a diagram illustrating an example of a neural network used in the inference process performed by the data processing device 1 according to the first embodiment. Figure 4 The neural network diagram illustrates the inference process performed by inference unit 14. For example... Figure 4 As shown, the output data y1 is calculated by obtaining the product of the input data and the weight parameters, and then obtaining the sum of these products.

[0066] (Equation 1)

[0067] As described above, in a neural network, a large number of multiple accumulations (hereinafter referred to as "floating-point multiple accumulations") using input data expressed in floating-point values ​​and coefficients expressed in floating-point values ​​are performed to compute output data. Since a large number of floating-point multiple accumulations are performed during inference processing, a large-capacity memory is required to store the floating-point data. Therefore, in the data processing apparatus 1 according to the first embodiment, the cumulative addition of coefficients is performed based on binary input data, rather than multiple accumulations using floating-point values. In other words, the data processing apparatus 1 according to the first embodiment can perform the operation corresponding to floating-point multiple accumulations by performing cumulative addition of coefficients based on binary input data.

[0068] Figures 5 to 8 This is a diagram illustrating the concept of multiple accumulations required in the neural network of the data processing device 1 according to the first embodiment. Figures 5 to 8 Four binary input data sets (1 0 1 1) are shown as an example of binary input data. Additionally, in... Figures 5 to 8 In this context, the coefficient addresses are A0 to A7. Simultaneously, the coefficient address information is generated by... Figures 5 to 8 The information expressed in (A0 A3 A2 A1). In coefficient addresses A0 to A7, in Figures 5 to 8The coefficient addresses used in the multiple accumulation shown are A0, A1, A2, and A3. The coefficient address information includes a combination of coefficient addresses corresponding to the coefficients used in a single, specific multiple accumulation. The coefficient address information also includes information regarding the reading order of the coefficient addresses, which corresponds to the order of operation of the coefficients in the multiple accumulation. In other words, when the coefficient address information includes multiple coefficient addresses, these multiple coefficient addresses are arranged so that they can be read in a predetermined order.

[0069] Reference Figures 5 to 8 To describe the coefficient address information in more detail. Figures 5 to 8 In the given equation, coefficient address A0 is the address where the coefficient multiplied by the leftmost 1 (first input data) in (1 0 1 1) is stored. Coefficient address A3 is the address where the coefficient multiplied by the second 0 (second input data) from the left in (10 1 1) is stored. Coefficient address A2 is the address where the coefficient multiplied by the third 1 (third input data) from the left in (1 0 1 1) is stored. Coefficient address A1 is the address where the coefficient multiplied by the fourth 1 (fourth input data) from the left in (1 0 1 1) is stored. (A0 A3 A2 A1) is... Figure 4 The model of the neural network shown is predetermined. (1 0 1 1) and (A0 A3 A2 A1) each have four elements. In the following text, the coefficient address information, i.e., the array of coefficient addresses, is considered a matrix and is also referred to as the "coefficient address matrix". For example, (A0 A3 A2 A1) is a 1×4 coefficient address matrix. Note that in... Figures 5 to 8 In the diagram, a 1×4 coefficient address matrix is ​​shown as an example, but the number of elements in a coefficient address matrix is ​​not limited to this. A coefficient address matrix can also be a 1×N matrix (where N is a natural number).

[0070] like Figure 5 As shown, when performing multiple accumulations of four binary input data with coefficients, the control unit 142 first reads 1 from the input data storage unit 130. This 1 is the leftmost binary input data in the (1 0 1 1) sequence. The input data determination unit 140 determines that the binary input data read from the input data storage unit 130 is 1, and 1 is a predetermined value. Based on the determination result of the input data determination unit 140, the control unit 142 reads the coefficient address A0 from the coefficient address information storage unit 131. Then, based on the coefficient address A0, the control unit 142 reads the coefficient 0.283 corresponding to address A0 from the coefficient storage unit 132. The coefficient 0.283 read by the control unit 142 from the coefficient storage unit 132 is input to the FPU 141 through the control unit 142, and is cumulatively added by the FPU 141.

[0071] Next, as Figure 6 As shown, the control unit 142 reads 0 from the input data storage unit 130, which is the second binary input data from the left in (1 0 11). The input data determination unit 140 determines that the binary input data is not 1, and 1 is a predetermined value. Based on the determination result of the input data determination unit 140, the control unit 142 performs the following control: does not read the coefficient address (A3) from the coefficient address information storage unit 131.

[0072] Subsequently, as Figure 7 As shown, the control unit 142 reads 1 from the input data storage unit 130. This 1 is the third binary input data from the left in the (1 0 1 1) sequence. The input data determination unit 140 determines that the binary input data read from the input data storage unit 130 is 1, and 1 is a predetermined value. Based on the determination result of the input data determination unit 140, the control unit 142 reads the coefficient address A2 from the coefficient address information storage unit 131. Then, based on the coefficient address A2, the control unit 142 reads the coefficient 1.232 corresponding to address A2 from the coefficient storage unit 132. The coefficient 1.232 read by the control unit 142 from the coefficient storage unit 132 is input to the FPU 141 through the control unit 142, and is cumulatively added by the FPU 141.

[0073] Ultimately, as Figure 8 As shown, the control unit 142 reads 1 from the input data storage unit 130, which is the fourth binary input data from the left in the (1 0 1 1) sequence. The input data determination unit 140 determines that the binary input data read from the input data storage unit 130 is 1, and 1 is a predetermined value. Based on the determination result of the input data determination unit 140, the control unit 142 reads the coefficient address A1 from the coefficient address information storage unit 131. Then, based on the coefficient address A1, the control unit 142 reads the coefficient -0.332 corresponding to address A1 from the coefficient storage unit 132. The coefficient -0.332 read by the control unit 142 from the coefficient storage unit 132 is input to the FPU 141 through the control unit 142, and is cumulatively added by the FPU 141.

[0074] pass Figures 5 to 8 The cumulative addition of the FPU 141 shown in the diagram has been replaced by multiple accumulations of floating-point input data and floating-point weight parameters.

[0075] Figures 5 to 8The state of multiple accumulations related to four binary input data (1 0 1 1) is shown. In the inference process of the neural network, the data processing device 1 sequentially reads the required number of binary input data from the input data storage unit 130 and repeats the above multiple accumulations.

[0076] Figure 9 This is a diagram illustrating an example of a multiple accumulation process performed by the data processing device 1 according to the first embodiment. (Refer to...) Figure 9 The coefficients and coefficient address information are values ​​obtained through learning by the neural network and are fixed values ​​in the inference processing performed by the data processing device 1. Input data is, for example, image input data. When the input data determination unit 140 determines that the input data is not the predetermined value 1, the coefficient address information is not acquired, and therefore, in the case of input data being 0, the multiplication result of (input data) × (coefficient) is represented as 0. The accumulated addition value represents the progress and result of multiple accumulations performed by the FPU 141. Figure 9 In the multiple accumulation shown, 0.468116 was obtained as the final multiple accumulation result.

[0077] Figure 10 This is a flowchart illustrating an example of a multiple accumulation process performed by the data processing device 1 according to the first embodiment. When multiple accumulation begins, the control unit 142 reads binary input data from the input data storage unit 130 and sends the binary input data to the input data determination unit 140 (step S101).

[0078] Upon receiving binary input data, the input data determination unit 140 determines whether the binary input data is a predetermined value (e.g., 1) (step S102). When the input data determination unit 140 determines that the binary input data is 0 (step S102: No), the control unit 142 does not read the coefficient address from the coefficient address information storage unit 131. Then, the control unit 142 reads the next binary input data from the input data storage unit 130 (step S101). On the other hand, when the input data determination unit 140 determines that the binary input data is 1 (step S102: Yes), the control unit 142 obtains the coefficient address from the coefficient address information storage unit 131 (step S103). Here, the coefficient address is information related to the address where the coefficient is stored, and it is integer data.

[0079] Subsequently, the control unit 142 reads and accesses the coefficient storage unit 132 based on the coefficient address obtained from the coefficient address information storage unit 131, and obtains the coefficient (step S104). Here, the coefficient is floating-point data.

[0080] The coefficients obtained in step S104 are sent to the FPU 141 by the control unit 142. When the coefficients are input, the FPU 141 performs cumulative addition of the floating-point data (step S105). By performing cumulative addition of the coefficients by the FPU 141, the multiplication and addition processes for floating-point input data and floating-point coefficients are replaced.

[0081] After the FPU 141 performs the cumulative addition of coefficients, the control unit 142 determines whether the input data is the final input data (step S106). When the input data is the final input data (step S106: Yes), the multiple accumulation process ends. On the other hand, when the input data is not the final input data (step S106: No), the control unit 142 reads the next binary input data from the input data storage unit 130 and sends the next binary input data to the input data determination unit 140 (step S101).

[0082] (Effect)

[0083] The data processing device 1 according to the first embodiment includes a coefficient address information storage unit 131, a coefficient storage unit 132, an input data determination unit 140, and an FPU 141. Based on the determination result of the input data determination unit 140, the data processing device 1 obtains coefficient addresses from the coefficient address information storage unit 131. Furthermore, based on the obtained coefficient addresses, the data processing device 1 obtains coefficients from the coefficient storage unit 132. Then, the data processing device 1 can replace multiple accumulations of floating-point data by performing cumulative additions of the obtained coefficients by the FPU 141. Since the data processing device 1 can reference 2K coefficients (K is a natural number) based on K-bit coefficient addresses, it can achieve high recognition accuracy in neural network inference processing while reducing the required memory capacity.

[0084] Furthermore, in the data processing apparatus 1 according to the first embodiment, the distribution of expressible coefficients is more diverse than that in conventional techniques. Regarding this, reference will be made to... Figure 28 , Figure 11 and Figure 12 To describe.

[0085] Figure 28 This is a graph showing the distribution of weight parameters in conventional techniques. Additionally, Figure 11 This is a diagram illustrating an example of a coefficient distribution that can be implemented in the data processing device 1 according to the first embodiment. For example... Figure 28 As shown, according to conventional techniques, the weight parameters are decomposed into a basis matrix and coefficient information, where the basis matrix has binary elements of 1 and -1. Therefore, the distribution of the implemented weight parameters is always symmetrical about the origin (0). On the other hand, as... Figure 11As shown, in the data processing apparatus 1 according to the first embodiment, coefficients are referenced via coefficient addresses. For example... Figure 11 As shown, in the data processing apparatus 1 according to the first embodiment, for example, +0.3 can be consistently added to the coefficients. Therefore, in the data processing apparatus 1 according to the first embodiment, the distribution of the implemented weight parameters can be asymmetric with respect to the origin (0). In this way, in the data processing apparatus 1 according to the first embodiment, independent settings (e.g., adding individual deviation values) can be performed for the coefficients stored in the coefficient storage unit 132.

[0086] Figure 12 It is used to describe the input to Figure 4 The diagram shows the relationship between the distribution of input values, weight parameters, and output values ​​of the neural network. Figure 12 The upper side corresponds to conventional technology, while Figure 12 The lower side corresponds to the first embodiment. For example... Figure 12 As shown above, when a distorted distribution of input values ​​is input, if the weight parameters are symmetrical between the positive (+) and negative (-) sides, the output values ​​are output as a more distorted distribution. The reason for this more distorted distribution is that multiple accumulations are performed multiple times in a neural network, and each time a multiple accumulation is performed, the distortion of the distribution increases. As a result, the dispersion increases, which leads to a deterioration in the accuracy of the inference process.

[0087] On the other hand, such as Figure 12 As shown below, when the asymmetric weighting parameters between the positive (+) and negative (-) sides are expressed by adding deviation values ​​to the weighting parameters, the distortion of the output value distribution can be reduced. Since settings such as adding deviation values ​​to coefficients can be performed in the data processing device 1 according to the first embodiment, batch normalization processing can also be performed simultaneously with multiple accumulations, and batch normalization processing corrects the dispersion and average value of the output values.

[0088] Furthermore, when the input data determination unit 140 determines that the binary input data is not a predetermined value (e.g., 1) but 0, the data processing device 1 does not retrieve the coefficient address from the coefficient address information storage unit 131, nor does it retrieve the coefficient from the coefficient storage unit 132. Therefore, unnecessary memory accesses and floating-point operations can be reduced. In other words, the data processing device 1 can achieve acceleration of multiple accumulations and a reduction in the power consumption required in the neural network.

[0089] (Second Embodiment)

[0090] Next, the second embodiment will be described. The data processing device 1A according to the second embodiment differs from the data processing device 1 according to the first embodiment in that the coefficient multiplied with each input data is expanded from 1 to M (M is a natural number greater than or equal to 2) to improve the accuracy of multiple accumulations. Since the coefficient multiplied with each input data is expanded from 1 to M (M is a natural number greater than or equal to 2), the coefficient address of each input data is also expanded from 1 to M. Since the configuration and operations are the same as those described in the first embodiment, except that the number of coefficient addresses for a binary input data is expanded from 1 to M, the same configuration is indicated by the same reference numerals, and redundant descriptions are omitted.

[0091] (Configuration)

[0092] Figure 13 This is a diagram illustrating an example configuration of the inference unit 14 and RAM 13A according to the second embodiment. Figure 13 As shown, the RAM 13A according to the second embodiment includes a coefficient address information storage unit 131A, which replaces the coefficient address information storage unit 131 according to the first embodiment.

[0093] Figure 14 This is an explanatory diagram illustrating the general outline of the multiple accumulation processing performed by the data processing apparatus according to the second embodiment. (As shown...) Figure 14 As shown, in the data processing apparatus 1A according to the second embodiment, the coefficient address information can be expressed as an M×N matrix. (See reference...) Figure 14 Before inference processing, the addresses of the M coefficients in each column of the M×N matrix are predetermined such that the sum of the M coefficients referenced by the M coefficient addresses is closest to the original weight parameters. The leftmost column of the reference matrix (the first column) is adjusted such that the sum of the values ​​of the coefficients referenced by coefficient address A0 + (the values ​​of the coefficients referenced by coefficient address A3) + (the values ​​of the coefficients referenced by coefficient address A0) + ... + (the values ​​of the coefficients referenced by coefficient address A1) is substantially equal to the first element (feature) of the weight parameters before quantization. In other words, the coefficient address information is determined to correspond to the fact that the weight parameters before quantization are decomposed, thus substantially equal to the sum of one or more coefficients. As a method for adjusting the combination of coefficient addresses, for example, a table of weight parameter values ​​is pre-formed, which can be expressed according to the combination of the M coefficient addresses, and the combination of the M coefficient addresses can be stored in the coefficient address information storage unit 131A. Through this combination of M coefficient addresses, the value of the table closest to the original weight parameters is generated.

[0094] (Effect)

[0095] According to the second embodiment, the data processing device 1A stores multiple coefficient addresses for a binary input data in the coefficient address information storage unit 131A. Therefore, compared with the data processing device 1A according to the first embodiment, the weighting parameters can be more accurately approximated. Therefore, the data processing device 1A according to the second embodiment can achieve higher recognition accuracy in the inference processing of neural networks.

[0096] (Third embodiment)

[0097] (Configuration)

[0098] Figure 15 This is a block diagram illustrating an example configuration of the inference unit 14B and RAM 13 of the data processing apparatus 1B according to a third embodiment. The data processing apparatus 1B according to the third embodiment differs from the data processing apparatus 1A according to the second embodiment in that the inference unit 14B further includes an access frequency accumulation unit 143, and the FPU 141B performs multiple accumulation based on the number of accesses to the coefficient address accumulated in the access frequency accumulation unit 143. Since the configuration and operation, apart from this, are the same as those of the data processing apparatus 1 described in the first embodiment, the same configuration is indicated by the same reference numerals, and redundant descriptions are omitted.

[0099] refer to Figure 15 The access frequency accumulation unit 143 is a circuit that counts the reference frequency of each coefficient address among the plurality of coefficient addresses stored in the coefficient address information storage unit 131A. More specifically, the access frequency accumulation unit 143 counts the number of times each coefficient address is accessed during the calculation of an output data by the input data determination unit 140. In other words, the access frequency accumulation unit 143 is a circuit that counts the reference frequency of each coefficient address among the plurality of coefficient addresses.

[0100] (operate)

[0101] Figure 16 This is a diagram illustrating an example of the operation of the access frequency accumulation unit 143 in the data processing apparatus 1B according to the third embodiment when obtaining output data. (See reference) Figure 16 For the four binary input data, the weight parameters are represented by a coefficient address matrix, which has 4×4 coefficient addresses as elements.

[0102] exist Figure 16In the input data, the first binary input data is 1, which is shown as the leftmost 1 in (1 0 1 1). Therefore, the input data determination unit 140 determines to obtain the coefficient address from the coefficient address information storage unit 131A through the control unit 142B. Based on the determination result of the input data determination unit 140, the control unit 142B sequentially obtains the coefficient addresses A0, A3, A0, and A1 included in the first column of the 4×4 coefficient address matrix from the coefficient address information storage unit 131A. The control unit 142B sequentially sends the obtained coefficient addresses A0, A3, A0, and A1 to the access frequency accumulation unit 143. The access frequency accumulation unit 143 counts as follows: 2 is counted as the reference frequency of coefficient address A0, 1 is counted as the reference frequency of coefficient address A1, and 1 is counted as the reference frequency of coefficient address A3, and its information is stored.

[0103] Subsequently, the second binary input data is the second 0 from the left in (1 0 1 1). Therefore, based on the determination result of the input data determination unit 140, the control unit 142B performs the following control: does not access the coefficient address information storage unit 131A. Therefore, the count value stored in the access frequency accumulation unit 143 is not updated.

[0104] Next, the third binary input data is the third 1 from the left in (1 0 1 1). Therefore, the input data determination unit 140 determines to obtain the coefficient address from the coefficient address information storage unit 131A via the control unit 142B. Based on the determination result of the input data determination unit 140, the control unit 142B sequentially obtains coefficient addresses A2, A0, A2, and A6 from the coefficient address information storage unit 131A. The control unit 142B sequentially sends the obtained coefficient addresses A2, A0, A2, and A6 to the access frequency accumulation unit 143. The access frequency accumulation unit 143 counts as follows: 3 as the reference frequency of coefficient address A0, 1 as the reference frequency of coefficient address A1, 2 as the reference frequency of coefficient address A2, 1 as the reference frequency of coefficient address A3, and 1 as the reference frequency of coefficient address A6, and stores this information.

[0105] Ultimately, the fourth binary input data is the fourth 1 from the left in (1 0 1 1). Therefore, the input data determination unit 140 determines that the coefficient address is obtained from the coefficient address information storage unit 131A by the control unit 142B.

[0106] like Figure 16As shown, finally, the access frequency accumulation unit 143 stores the counting result information 3 as the reference frequency of coefficient address A0, stores the counting result information 3 as the reference frequency of coefficient address A1, stores the counting result information 2 as the reference frequency of coefficient address A2, stores the counting result information 2 as the reference frequency of coefficient address A3, stores the counting result information 0 as the reference frequency of coefficient address A4, stores the counting result information 0 as the reference frequency of coefficient address A5, stores the counting result information 1 as the reference frequency of coefficient address A6, and stores the counting result information 1 as the reference frequency of coefficient address A7.

[0107] According to the FPU 141B of the third embodiment, based on the accumulation result of the access frequency accumulation unit 143 and the coefficients stored in the coefficient storage unit 132, multiple accumulation is performed to calculate the output data for the input data (1 0 1 1). Specifically, the output data is obtained by calculating as follows: (reference frequency of coefficient address A0) × (coefficient stored in coefficient address A0) + (reference frequency of coefficient address A1) × (coefficient stored in coefficient address A1) + (reference frequency of coefficient address A2) × (coefficient stored in coefficient address A2) + (reference frequency of coefficient address A3) × (coefficient stored in coefficient address A3) + (reference frequency of coefficient address A4) × (coefficient stored in coefficient address A4) + (reference frequency of coefficient address A5) × (coefficient stored in coefficient address A5) + (reference frequency of coefficient address A6) × (coefficient stored in coefficient address A6) + (reference frequency of coefficient address A7) × (coefficient stored in coefficient address A7). Figure 15 In the example, the output data is calculated by the FPU 141B performing the following operation: 3×0.283+3×(-0.332)+2×1.232+2×0.432+0×(-0.543)+0×0.123+1×(-1.121)+1×0.635.

[0108] Figure 17 This is a flowchart illustrating an example of a multiple accumulation process performed by the data processing apparatus 1B according to the third embodiment. When multiple accumulation begins, the control unit 142B reads binary input data from the input data storage unit 130. The control unit 142B sends the acquired binary input data to the input data determination unit 140 (step S101).

[0109] When the input data determination unit receives binary input data, the input data determination unit 140 determines whether the binary input data is 1, where 1 is a predetermined value (step S102). When the input data determination unit 140 determines that the binary input data is 0 (step S102: No), the control unit 142B does not read the coefficient address from the coefficient address information storage unit 131A. Then, the control unit 142B reads the next binary input data from the input data storage unit 130 (step S101). On the other hand, when the input data determination unit 140 determines that the binary input data is 1 (step S102: Yes), the control unit 142B obtains the coefficient address from the coefficient address information storage unit 131A (step S103). Here, the coefficient address is the address value where the coefficient is stored, and it is integer data.

[0110] The control unit 142B sends the coefficient address obtained from the coefficient address information storage unit 131A to the address frequency accumulation unit 143. The access frequency accumulation unit 143 increments the count value associated with the received coefficient address by 1 (step S304). The counting operation of the access frequency accumulation unit 143 is integer arithmetic.

[0111] After step S304, the control unit 142B determines whether the binary input data is the final input data (step S305). When the input data is the final input data (step S305: Yes), the control unit 142B obtains the following coefficient from the coefficient storage unit 132: the count value of this coefficient is greater than or equal to 1 in the access frequency accumulation unit 143 (step S306). On the other hand, when the binary input data is not the final input data (step S305: No), the control unit 142B reads the next binary input data from the input data storage unit 130 (step S101).

[0112] In step S306, the coefficients obtained from the coefficient storage unit 132 are sent to the FPU 141B. After step S306, the FPU 141B calculates the multiplication of each coefficient with the reference frequency counted by the access frequency accumulation unit 143 (step S307). Furthermore, the FPU 141B calculates the sum of the multiplications calculated in step S307 (step S308). When the FPU 141B completes the accumulation in step S308, the multiple accumulation ends.

[0113] (Effect)

[0114] Compared to the data processing device 1A according to the second embodiment, the data processing device 1B according to the third embodiment further includes an access frequency accumulation unit 143. The access frequency accumulation unit 143 counts the number of read accesses performed on the coefficient address read from the coefficient address information storage unit 131A. In other words, since the access frequency accumulation unit 143 counts the number of times each coefficient stored in the coefficient storage unit 132 is used for multiple accumulations, the number of cumulative additions for each coefficient in the FPU 141B can be determined. Therefore, in the data processing device 1B, it is not necessary to access the coefficient storage unit 132 to read the coefficient corresponding to the coefficient address every time a coefficient address is obtained from the coefficient address information storage unit 131A. That is, in the data processing device 1B, for multiple accumulations in the FPU 141B, the coefficient storage unit 132 only needs to be accessed once for each coefficient. Since the power required for memory access is generally greater than the power required for counting the reference frequency, the data processing device 1B according to the third embodiment can suppress the power required for inference processing via a neural network. In neural networks, the number of weight parameters used in multiple accumulation processing is large, so the data processing device 1B according to the third embodiment has a good power suppression effect.

[0115] (Revise)

[0116] In the third embodiment, an example of further providing the access frequency accumulation unit 143 to the configuration of the second embodiment has been described; however, the application of the access frequency accumulation unit 143 is not limited to the second embodiment. That is, by further providing the access frequency accumulation unit 143 to the inference unit 14 of the first embodiment, a similar effect to the third embodiment can be achieved even when using a 1×N coefficient address matrix.

[0117] (Fourth embodiment)

[0118] In the first to third embodiments, it is assumed that the binary input data is 0 or 1. Based on this assumption, the input data determination unit 140 determines whether to obtain the coefficient address from the coefficient address information storage unit 131 in the data processing device 1 and the data processing device 1A. In neural networks, a ramp function called ReLU (Rectified Linear Unit) is mainly integrated as an activation function for each output data, and in many cases, the result is used as the input to the next stage. Here, ReLU is a function that outputs the value of the input value when the input value is greater than or equal to 0, and outputs 0 when the input value is less than 0. Therefore, since the input data is 0 or a positive value, it is assumed in the first to third embodiments that the binary input data is 0 or 1.

[0119] Incidentally, functions other than ReLU are sometimes used as activation functions, and in such cases, the input data can become negative. As described above, the data processing device 1C according to the fourth embodiment is suitable for input data that is binary, thus taking two values, positive and negative.

[0120] (Configuration)

[0121] Figure 18 This is a diagram illustrating an example configuration of the inference unit 14C and RAM 13C according to a fourth embodiment. The inference unit 14C according to the fourth embodiment includes a control unit 142C, which is different from the control unit 142 according to the first embodiment. Furthermore, the RAM 13C according to the fourth embodiment includes a coefficient address information storage unit 131C, which is different from the coefficient address information storage unit 131. The RAM 13C according to the fourth embodiment also includes a coefficient storage unit 132C, which is different from the coefficient storage unit 132.

[0122] Figure 19 This is a diagram illustrating an example configuration of the coefficient address information storage unit 131C and coefficient storage unit 132C according to the fourth embodiment. The data processing device 1C according to the fourth embodiment differs from the first and second embodiments in that coefficients are stored in the coefficient storage unit 132C for each of the following two cases: when the binary input data is positive, and when the binary input data is negative. Since the coefficients acquired by the control unit 142C differ depending on whether the binary input data is positive or negative, the coefficient address information storage unit 131C also stores different coefficient addresses depending on whether the input data is positive or negative. Since the configuration and operation are otherwise identical to those of the data processing devices 1 and 1A described in the first and second embodiments, the same configurations are indicated by the same reference numerals, and redundant descriptions are omitted.

[0123] like Figure 19 As shown, the binary input data can take two positive and negative values: 1 and -1. When the input data is 1, the control unit 142C obtains the coefficient address An_1 instead of the coefficient address An (n is an integer greater than or equal to 0 and less than or equal to 7). On the other hand, when the input data is -1, the control unit 142C obtains the coefficient address An_-1 instead of the coefficient address An (n is an integer greater than or equal to 0 and less than or equal to 7).

[0124] (operate)

[0125] exist Figure 19In the example, (1 -1 1 1) is sequentially read from the input data storage unit 130 as binary input data. The first binary input data is the leftmost 1 in (1 -1 1 1). Since the first binary input data is a positive value, the control unit 142C obtains four coefficient addresses A0_1, A3_1, A0_1, and A1_1, corresponding to the first column of the 4×4 coefficient address matrix. For example, the control unit 142C obtains the coefficient 0.283 based on the obtained coefficient address A0_1.

[0126] On the other hand, the second binary input data is the second -1 from the left in (1 -1 1 1). Since the second binary input data is negative, the control unit 142C obtains four coefficient addresses A0_-1, A1_-1, A0_-1, and A2_-1 to correspond to the second column of the 4×4 coefficient address matrix. For example, the control unit 142C obtains the coefficient -0.332 based on the obtained coefficient address A0_-1.

[0127] (Effect)

[0128] Figure 20 This illustrates an example of the distribution of output values ​​when distorted input values ​​are fed into a neural network. As mentioned above, functions other than ReLU can be used as activation functions, and in this case, the input data may become negative. In this situation, if the ratio between the input and output data is acceptable, the method of changing the sign of the reference coefficients solely based on the sign of the input data is also applicable. However, as... Figure 20 As shown in the figure above, when the distribution of input values ​​is distorted and the weight parameters are proportional to the input values, the dispersion of the output value distribution increases, and the inference accuracy of the neural network deteriorates.

[0129] On the other hand, by performing adjustments to the weight parameters and deviation values ​​(batch normalization), it is possible to achieve the following: Figure 20 As shown in the figure below, the mean becomes 1 and the dispersion becomes 1, and the distortion in the distribution of the input values ​​can be suppressed in the distribution of the output values.

[0130] According to the fourth embodiment, the coefficient address information storage unit 131C and the coefficient storage unit 132C store coefficient address information and coefficients respectively based on the positive or negative sign of the binary input data. Therefore, in the data processing device 1C according to the fourth embodiment, cumulative addition of different coefficients can be performed based on the positive or negative sign of the binary input data. In other words, the data processing device 1C can simultaneously perform batch normalization and cumulative addition. As a result, the data processing device 1C can suppress the degradation of the inference accuracy of the neural network.

[0131] (Fifth Embodiment)

[0132] (Configuration)

[0133] Figure 21 This is a block diagram illustrating a configuration example of a data processing device 1D according to a fifth embodiment. The data processing device 1D according to the fifth embodiment differs from the data processing device 1 according to the first embodiment in that a most frequent index determination unit 144 is further provided, coefficients adjusted by the control unit 142D are output to the FPU 141D, and the control unit 142D controls the FPU 141D to perform offset addition. Since the configuration and operation are otherwise identical to those of the data processing device 1 described in the first embodiment, the same configurations are indicated by the same reference numerals, and redundant descriptions are omitted.

[0134] like Figure 21 As shown, the inference unit 14D according to the fifth embodiment includes an input data determination unit 140, a most frequent index determination unit 144, an FPU 141D, and a control unit 142D.

[0135] In the fifth embodiment, among the coefficient addresses included in the coefficient address information, the coefficient address with the highest frequency of occurrence is referred to as the "most frequent coefficient address". The most frequent index determination unit 144 stores the most frequent coefficient address to determine whether the coefficient address read by the control unit 142D is the most frequent coefficient address. The method for determining the most frequent coefficient address will be described later. When the control unit 142D reads a coefficient address from the coefficient address information storage unit 131 based on the determination result of the input data determination unit 140, the most frequent index determination unit 144 determines whether the read coefficient address is the most frequent coefficient address. Furthermore, the most frequent index determination unit 144 uses a counter (not shown) to count the number of times the coefficient address with the highest frequency of occurrence is read.

[0136] The control unit 142D differs from the control unit 142 according to the first embodiment in that it has the function of controlling the FPU 141D to perform offset addition related to the address of the coefficient with the highest occurrence frequency. Offset addition will be referred to later. Figure 22 and Figure 23 Describe it.

[0137] The difference between FPU 141D and FPU 141 according to the first embodiment is that, prior to the inference process, cumulative addition is performed based on pre-adjusted coefficients, and the aforementioned offset addition is performed.

[0138] (operate)

[0139] Figure 22This is an explanatory diagram of the method for adjusting the coefficients according to the fifth embodiment. Figure 22 The coefficient adjustments shown can be performed before the inference process. In other words, Figure 22 The coefficient adjustment shown can be performed before multiple accumulations.

[0140] Regarding the adjustment of coefficients, firstly, the address of the most frequent coefficient is determined. Before multiple accumulation, the control unit 142D reads coefficient address information from the coefficient address information storage unit 131 and sends the coefficient address information to the most frequent index determination unit 144. Based on the coefficient address information read from the coefficient address information storage unit 131, the most frequent index determination unit 144 counts the frequency of occurrence of coefficient addresses A0 to A7 using a counter (not shown). Figure 22 In the example, coefficient address A3 was retrieved 162 times, and it can be seen that the coefficient stored at coefficient address A3 is used most frequently in the multiple accumulation. Therefore, the most frequent index determination unit 144 determines that the most frequent coefficient address is coefficient address A3. Note that although an example of the most frequent index determination unit 144 determining the most frequent coefficient address has been described here, the method for determining the most frequent coefficient address is not limited to this. For example, the most frequent coefficient address can be determined by a program outside the personal computer and data processing device 1D. In this case, for example, the processor 10 executes the program and writes the most frequent coefficient address into a register (not shown) that has the most frequent coefficient address provided in the most frequent index determination unit 144.

[0141] After the address of the most frequent coefficient is determined, the control unit 142D retrieves the stored coefficients based on coefficient addresses A0 to A7 in order to adjust the coefficients. Then, based on information received from the most frequent index determination unit 144, which indicates that coefficient address A3 is the most frequent coefficient address, the control unit 142D adjusts the retrieved coefficients. More specifically, the adjustment is performed so that the value of the most frequently used coefficient becomes 0. Figure 22 In the example, control unit 142D adjusts the coefficient -0.13 stored at coefficient address A3 to 0.00. Based on this adjustment, control unit 142D adjusts the coefficient -0.79 stored at coefficient address A0 to -0.66 (-0.79 + 0.13 = -0.66). Similarly, control unit 142D adjusts the coefficient -0.46 stored at coefficient address A1 to -0.33 (-0.46 + 0.13 = -0.33). In the same manner, control unit 142D adjusts each coefficient stored at coefficient addresses A2 and A4 through A7 by +0.13.

[0142] Next, the offset value determined before the inference process performed by the data processing device 1D according to the fifth embodiment will be described. As described above, the most frequent index determination unit 144 determines the most frequent coefficient address and further counts the number of times the most frequent coefficient address included in the coefficient address information appears using a counter (not shown). The offset value is calculated as (offset value) = (number of times the most frequent coefficient address is read) × (most frequently used coefficient) and stored in the coefficient storage unit 132.

[0143] After the inference process begins, the FPU 141D according to the fifth embodiment performs cumulative addition for multiple accumulations based on adjusted coefficients, which are adjusted and stored in coefficient storage unit 132 before the inference process. After the cumulative addition based on the adjusted coefficients, the FPU 141D adds the offset value read from coefficient storage unit 132 to the result of the cumulative addition.

[0144] Figure 23 This is a flowchart illustrating an example of a multiple accumulation process performed by the data processing apparatus 1D according to the fifth embodiment. When multiple accumulation begins, the control unit 142D reads binary input data from the input data storage unit 130. The control unit 142D sends the acquired binary input data to the input data determination unit 140 (step S101).

[0145] When the input data determination unit 140 receives binary input data, it determines whether the binary input data is a predetermined value (e.g., 1) (step S102). When the input data determination unit 140 determines that the binary input data is 0 (step S102: No), the control unit 142D does not read the coefficient address from the coefficient address information storage unit 131. Then, the control unit 142D reads the next binary input data from the input data storage unit 130 (step S101). On the other hand, when the input data determination unit 140 determines that the binary input data is 1 (step S102: Yes), the control unit 142D reads the coefficient address from the coefficient address information storage unit 131 (step S103).

[0146] The control unit 142D sends the coefficient address obtained from the coefficient address information storage unit 131 to the most frequent index determination unit 144. The most frequent index determination unit 144 determines whether the received coefficient address is the most frequent coefficient address (step S501). When the received coefficient address is not the most frequent coefficient address (step S501: No), the control unit 142D reads the coefficient from the coefficient storage unit 132 based on the determination result of the most frequent index determination unit 144 (step S104). On the other hand, when the received coefficient address is the most frequent coefficient address (step S501: Yes), the control unit 142D does not read the coefficient from the coefficient storage unit 132 based on the determination result of the most frequent index determination unit 144, and executes control to read the next input data (step S101).

[0147] The coefficients obtained in step S104 are sent to the FPU 141D by the control unit 142D. When the coefficients are input, the FPU 141D performs cumulative addition of the floating-point data (step S105). By performing cumulative addition of the coefficients by the FPU 141D, the multiplication and addition processes for the floating-point input data and floating-point coefficients are replaced.

[0148] After the cumulative addition of coefficients is performed by the FPU 141D, the control unit 142D determines whether the input data is the final input data (step S106). When the input data is not the final input data (step S106: No), the control unit 142D reads the next binary input data from the input data storage unit 130 and sends the next binary input data to the input data determination unit 140 (step S101). On the other hand, when the input data is the final input data (step S106: Yes), the control unit 142D controls the FPU 141D to add the offset value to the result of the cumulative addition calculated in step S105 (step S502). When the offset addition is performed by the FPU 141D, a multiple accumulation ends.

[0149] (Effect)

[0150] Compared to the data processing device 1 according to the first embodiment, the data processing device 1D according to the fifth embodiment further includes a most frequent index determination unit 144. During multiple accumulations, the most frequent index determination unit 144 determines whether the coefficient address read from the coefficient address information storage unit 131 is the most frequent coefficient address. Since the FPU 141D can omit the cumulative addition for coefficients adjusted to 0, the power consumption and execution time required for cumulative addition can be reduced.

[0151] (Sixth embodiment)

[0152] In the first through fifth embodiments, the input data is binary, for example, 0 and 1 or 1 and -1. The sixth embodiment relates to multiple accumulation processing when the input data includes multiple bits.

[0153] Figure 24 This is a diagram illustrating an example of a method for quantizing input data into multiple bits. Figure 24 Depending on which segment the input value belongs to, the input value is quantized as 000, 001, 010, 011, 100, 101, 110, or 111. The above segments are obtained by dividing (the maximum value of the input value (max) - 0) into eight equal parts.

[0154] (Configuration)

[0155] Figure 25 This figure illustrates an example configuration of the inference unit 14E and RAM 13 of the data processing apparatus 1E according to a sixth embodiment. The data processing apparatus 1E according to the sixth embodiment differs from the data processing apparatus 1 according to the first embodiment in that the data processing apparatus 1E further includes an integer conversion unit 145, a bit extraction unit 146, and a counter 147. Additionally, the data processing apparatus 1E according to the sixth embodiment differs from the first embodiment in that a control unit 142E and an FPU 141E are provided to replace the control unit 142 and FPU 141, respectively. Since the configuration and operation are otherwise identical to those of the data processing apparatus 1 described in the first embodiment, the same configurations are indicated by the same reference numerals, and redundant descriptions are omitted.

[0156] refer to Figure 25 The integer conversion unit 145 converts the input data into a predetermined L-bit (L is a natural number) integer, which is floating-point data.

[0157] Bit extraction unit 146 extracts the bits indicated by counter 147 from the integer data obtained by quantization by integer conversion unit 145.

[0158] Counter 147 is a circuit used to determine which bit of integer data is processed. At the start of multiple accumulation, counter 147 is 0, and the value increments by 1 sequentially until a predetermined number of bits is reached. Figure 25 In this process, counter 147 increments the value of J (an integer greater than or equal to 0) by 1. The value indicated by counter 147 is sent to input data determination unit 140 and bit extraction unit 146.

[0159] (operate)

[0160] Figure 26 This is a diagram conceptually illustrating a method of multiple accumulation performed by the data processing apparatus 1E according to the sixth embodiment. Figure 26 In the example of multiple accumulation shown, three input data points, 1.3, 0.2, and 3.1, are stored in the input data storage unit 130. The floating-point data 1.3, 0.2, and 3.1, read from the input data storage unit 130 via the control unit 142E, are sent to the integer conversion unit 145. The integer conversion unit 145 quantizes the input data 1.3, 0.2, and 3.1 into 100, 001, and 110, respectively. The bit extraction unit 146 extracts the 0, 1, and 0 corresponding to the 0th bit from the quantized integer data 100, 001, and 110, where 0, 1, and 0 are the LSBs of each of the integer data 100, 001, and 110. The extracted 0, 1, and 0 are sequentially sent to the input data determination unit 140 via the control unit 142E.

[0161] The input data determination unit 140 retrieves the coefficient address from the coefficient address information storage unit 131 based on the input 0, 1, and 0. First, the input data determination unit 140 determines that the 0th bit of the integer 100 is 0, obtained by quantizing the integer 100 by 1.3. The control unit 142E performs control to omit retrieving the coefficient address from the coefficient address information storage unit 131 based on the determination result of the input data determination unit 140. Subsequently, the input data determination unit 140 determines that the 0th bit of the integer 001 is 1, obtained by quantizing the integer 001 by 0.2. Based on the determination result of the input data determination unit 140, the control unit 142E retrieves the coefficient address A0 from the coefficient address information storage unit 131. Based on the retrieved coefficient address A0, the control unit 142E retrieves the coefficient 0.283 from the coefficient storage unit 132. The FPU 141E performs a cumulative addition on the coefficient 0.283. Finally, the input data determination unit 140 determines that the 0th bit of the integer 110 is 0, which is obtained by 3.1 quantization. The control unit 142E performs control to omit retrieving the coefficient address from the coefficient address information storage unit 131 based on the determination result of the input data determination unit 140.

[0162] Based on information from control unit 142E that processing of bit 0 has been completed, counter 147 increments the value of J by 1. Inference unit 14E performs processing for 0, 0, and 1, which are the first bits of the quantized integers 100, 001, and 110. At this point, accumulation by FPU 141E is performed with the following consideration: the value of the first bit of the input data is twice the value of the 0th bit. More specifically, control unit 142E, based on the value of counter 147, shifts the result of the cumulative addition of bit J to the left by J bits (equivalent to multiplying by 2J), and then controls FPU 141E to perform the accumulation.

[0163] Inference unit 14E repeats a similar process until the (L-1)th bit. Figure 26 In the example, L is 3 (L = 3), and the inference unit 14E repeats a similar process until the second bit. In this way, the inference unit 14E can perform multiple accumulations on input data that has been quantized into multiple bits.

[0164] Figure 27 This is a flowchart illustrating an example of a multiple accumulation process performed by the data processing apparatus 1E according to the sixth embodiment. At the beginning of the multiple accumulation, the count value J of the counter 147 is initialized to, for example, 0 (step S600). Subsequently, the control unit 142E reads the following value from the input data storage unit 130 as binary input data: the Jth bit value of the input data quantized into multiple bits (step S601). The control unit 142E sequentially sends the acquired Jth bit value as binary input data to the input data determination unit 140 (step S602).

[0165] When the input data determination unit 140 receives binary input data, it determines whether the binary input data is a predetermined value (e.g., 1) (step S102). When the input data determination unit 140 determines that the binary input data is 0 (step S102: No), the control unit 142E does not read the coefficient address from the coefficient address information storage unit 131. Then, the control unit 142E reads the next binary input data from the input data storage unit 130 (step S602). On the other hand, when the input data determination unit 140 determines that the binary input data is 1 (step S102: Yes), the control unit 142E obtains the coefficient address from the coefficient address information storage unit 131 (step S103).

[0166] Subsequently, the control unit 142E reads and accesses the coefficient storage unit 132 based on the coefficient address obtained from the coefficient address information storage unit 131, and obtains the coefficient (step S104).

[0167] The coefficients obtained in step S104 are sent to the FPU 141E by the control unit 142E. When the coefficients are input, the FPU 141E performs cumulative addition of the floating-point data (step S105). By performing cumulative addition of the coefficients by the FPU 141, the multiplication and addition processes for the floating-point input data and floating-point coefficients are replaced.

[0168] After the FPU 141E performs the cumulative addition of the coefficients, the control unit 142E determines whether the input data is the final input data (step S106). When the input data is not the final input data (step S106: No), the control unit 142E reads the next binary input data from the input data storage unit 130 and sends the next binary input data to the input data determination unit 140 (step S602). On the other hand, when the input data is the final input data (step S106: Yes), the multiple accumulation processing of the J-th bit is completed. When the cumulative addition of the J-th bit is completed (step S106: Yes), the control unit 142E instructs the FPU 141E to shift the result of the cumulative addition to the left by J bits (equivalent to multiplying by 2J) (step S607). Subsequently, the FPU 141E calculates the cumulative result up to the J-th bit by adding the cumulative addition result of the J-th bit to the cumulative result up to the (J-1)-th bit (step S608). When the accumulated result up to the Jth bit is calculated (step S608), counter 147 increments the value of J by 1 (step S609). If the incremented value of J is less than L (step S610: Yes), control unit 142E acquires the input data of the Jth bit based on the incremented value of J (step S601). On the other hand, if the incremented value of J is equal to or greater than L (step S610: No), control unit 142E completes the multiple accumulation of the input data quantized into multiple bits.

[0169] In the above description, the method of converting floating-point input data to integers and then performing multiple accumulations has been described as being applied to the first embodiment; however, embodiments to which the method can be applied are not limited to the first embodiment. That is, the method of converting floating-point input data to integers and then performing multiple accumulations can be applied to any of the second to fifth embodiments.

[0170] The present invention has been specifically described above based on embodiments, but the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the fourth embodiment can be combined with the third embodiment.

Claims

1. A data processing device, comprising: The input data determination unit is configured to determine whether each binary input data in the binary input data input to the neural network is a predetermined value; A storage unit is configured to store multiple coefficients of the neural network and coefficient address information, wherein the coefficient address information includes information related to the coefficient addresses where the multiple coefficients are stored; The control unit is configured to read the coefficient address from the storage unit based on the determination result of the input data determination unit, and to read the coefficient from the storage unit based on the coefficient address; as well as An arithmetic unit is configured to perform arithmetic operations relating to coefficients obtained by the control unit. The arithmetic unit performs cumulative addition on the coefficients obtained by the control unit.

2. The data processing device according to claim 1, The control unit reads the coefficient address based on the following determination result of the input data determination unit: the determination result is that the input data is the predetermined value.

3. The data processing device according to claim 1, The coefficient address information includes 1×N coefficient addresses, where N is a natural number greater than 1.

4. The data processing device according to claim 1, The coefficient address information includes M×N coefficient addresses, where M and N are natural numbers greater than 1.

5. The data processing device according to claim 1 further includes an access frequency accumulation unit, the access frequency accumulation unit being configured to count the number of times each coefficient address in the coefficient addresses is acquired; The arithmetic unit performs multiple accumulations based on the count value of the access frequency accumulation unit and the coefficient.

6. The data processing device according to claim 1, The binary input data includes two values, a positive value and a negative value, and the coefficient address information includes a first coefficient address corresponding to the positive value and a second coefficient address corresponding to the negative value.

7. The data processing device according to claim 6, When the binary input data is a positive value, the control unit reads the first coefficient address; and when the binary input data is a negative value, the control unit reads the second coefficient address.

8. The data processing device according to claim 1, The control unit adjusts the values ​​of the plurality of coefficients based on the coefficient address with the highest frequency of occurrence included in the coefficient address information, and the arithmetic unit performs cumulative addition on the adjusted plurality of coefficients.

9. A method of operating a data processing device, the data processing device having an arithmetic unit and a memory, the method comprising: The step of determining whether the binary input data fed into the neural network is a predetermined value; The step of storing multiple coefficients and coefficient address information of the neural network in the memory, wherein the coefficient address information includes information related to the coefficient addresses where the multiple coefficients are stored; The step of reading the coefficient address from the memory based on the determination result of the determination step; The step of reading the coefficient from the memory based on the coefficient address; as well as The step of the arithmetic unit performing arithmetic operations based on the read coefficients. The step of performing the arithmetic operation further includes the following step: performing cumulative addition of the coefficients obtained by reading the coefficients.

10. The operation method of the data processing device according to claim 9, The step of reading the coefficient address includes the following steps: based on the determination result that the input data is the predetermined value, read the coefficient address.

11. The method of operating the data processing device according to claim 9, The determining step further includes determining whether each of the N binary input data is the predetermined value, and the step of reading the coefficient address further includes reading 1×N coefficient addresses, where N is a natural number greater than 1.

12. The operation method of the data processing device according to claim 9, The step of reading the coefficient address further includes the following steps: reading M×N coefficient addresses, where M and N are natural numbers greater than 1.

13. The operation method of the data processing device according to claim 9 further includes the following step: counting the number of times each coefficient address in the coefficient addresses is acquired. The steps for performing arithmetic operations further include the following steps: performing multiple accumulations based on the counting results of the counting steps and the multiple coefficients.

14. The method of operating the data processing device according to claim 9, The determining step further includes the following steps: determining whether the binary input data is the predetermined value, the binary input data having a positive or negative value, and the step of reading the coefficient address further includes the following steps: when the binary input data is the positive value, reading the first coefficient address, and when the binary input data is the negative value, reading the second coefficient address.

15. The operation method of the data processing device according to claim 9 further includes the following step: adjusting the values ​​of the plurality of coefficients based on the coefficient address with the highest frequency of occurrence included in the coefficient address information. The steps of performing arithmetic operations also include the following steps: performing cumulative addition of the adjusted plurality of coefficients.

16. The method of operating the data processing device according to claim 9, The steps for performing arithmetic operations also include the following: performing multiple accumulations required for the inference processing of the neural network.

17. The method of operating the data processing device according to claim 9, The determining step further includes the following steps: determining whether each of the multiple binary input data constituting the multi-bit data is a predetermined value, and the step of performing arithmetic operations further includes the following steps: shifting the arithmetic operation result of the step of performing arithmetic operations based on the bit position of the binary input data in the multi-bit data.

18. A computer-readable storage medium storing a program relating to a neural network model executed by a data processing device having a CPU and a memory, the computer-readable storage medium storing the program causing a computer to perform processing comprising the following steps: The step of determining whether each binary input data in the binary input data is a predetermined value; The step of storing multiple coefficients and coefficient address information in the memory, wherein the coefficient address information includes information related to the following coefficient addresses, wherein the coefficient addresses include the addresses where the multiple coefficients are stored; The step of reading the coefficient address from the memory based on the determination result of the determination step; The step of reading the coefficient from the memory based on the coefficient address; as well as Perform the step of cumulative addition of the read coefficients.

19. The computer-readable storage medium for storing a program according to claim 18, The step of reading the coefficient address further includes the following step: reading the coefficient address based on the following determination result of the determination step, wherein the determination result is that the input data is the predetermined value.

Images