Neural network implementation device, method and program thereof, and neural network device

Abstract

The present invention provides a neural network implementation device, method, and program that can reduce the amount of data transfer in a supermask when implementing a subset network of a neural network in an arithmetic circuit, as well as a neural network device equipped therewith. [Solution] The neural network implementation device of the present invention is a device that implements a subset network extracted from a neural network, which is equipped with pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into an arithmetic circuit, and comprises an external memory circuit EM that stores supermask information representing the supermask, and the external memory circuit EM does not output the supermask to the arithmetic circuit when the edge weights of the subset network satisfy predetermined conditions.

Description

Technical Field

[0001] The present invention relates to a neural network implementation device for implementing a neural network in an arithmetic circuit, a neural network implementation method and a neural network implementation program, and a neural network device provided with the neural network implementation device.

Background Art

[0002] In recent years, so-called artificial intelligence has been researched, developed, and advanced. One of them is the hidden neural network theory (Strong Lottery Ticket Hypothesis), which is disclosed in, for example, Patent Document 1. In this hidden neural network theory, each weight assigned to each edge (connection) between nodes in a neural network is generated by a pseudo-random number generator, each weight is fixed, and the importance (score) of the edge is machine-learned. Edges having a score equal to or higher than a predetermined threshold are extracted, and a network (sub-network, sub-neural network) formed by the extracted edges and their nodes is selected (extracted) as a neural network of an inference model (prediction model) actually used for inference (prediction) from the neural network. The learning dataset used for the machine learning of the score is the same as the learning dataset used when machine-learning each weight of each edge in a neural network by ordinary machine learning. The score (importance) increases as the edge contributes (influences) more to reducing the error between the correct information (teacher data) of the learning data and the output result of the neural network for a plurality of inputs.

[0003] In an ordinary neural network, each weight of each edge is obtained by machine learning, but in a hidden neural network, each weight of each edge is fixed and machine learning of each weight is not performed. Instead, as described above, the score of the edge is machine-learned.

[0004] When the neural network of the inference model described above is implemented in an arithmetic circuit that performs calculations, such as an inference LSI (Large Scale Integration), it is necessary to transfer the weights of each edge in the neural network of the inference model, which are stored in an external memory circuit, to the arithmetic circuit. In hidden neural network theory, by providing the arithmetic circuit with a pseudo-random number generator that generates the weights of each edge, it is only necessary to transfer a supermask indicating whether or not each edge is valid from the external memory circuit to the arithmetic circuit. Therefore, the amount of data transferred from the external memory circuit to the arithmetic circuit (data transfer amount) can be reduced compared to the case of a normal neural network that uses the entire network. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-078975 [Overview of the project] [Problems that the invention aims to solve]

[0006] As mentioned above, while the amount of data transferred can be reduced in the hidden neural network theory compared to that of a normal neural network, there is a desire to further reduce that amount of data transfer.

[0007] The present invention was made in view of the above circumstances, and its object is to provide a neural network implementation device, a neural network implementation method, and a neural network implementation program that can reduce the amount of data transfer of a supermask when a subset network of a neural network is implemented in an arithmetic circuit, as well as a neural network device equipped with the neural network implementation device. [Means for solving the problem]

[0008] As a result of various studies, the inventors have found that the above objective can be achieved by the present invention as described below. That is, a neural network implementation device according to one aspect of the present invention is a device that implements a subset network extracted from a neural network having pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into an arithmetic circuit, and comprises an external memory circuit that stores supermask information representing the supermask, wherein the external memory circuit does not output the supermask to the arithmetic circuit when the edge weights of the subset network satisfy predetermined conditions.

[0009] The inventor investigated the frequency of occurrence of each weight at each edge in a subset network and found that each weight is concentrated within a range that satisfies predetermined conditions. Therefore, the inventor conceived the idea that the amount of data transferred for the supermask could be reduced by assigning pseudorandom numbers generated by a pseudorandom number generator as the weights to the edges, without relying on the supermask, instead of outputting the supermask corresponding to the weights of the edges that satisfy the predetermined conditions from the external memory circuit to the arithmetic circuit. Based on this idea, the neural network implementation device does not output the supermask to the arithmetic circuit when the weights of the edges of the subset network satisfy predetermined conditions, thus reducing the amount of data transferred for the supermask when implementing a subset network of a neural network in the arithmetic circuit.

[0010] In another embodiment, the neural network implementation device described above includes a pseudorandom number generator that generates pseudorandom numbers, a condition determination device that determines whether the pseudorandom numbers generated by the pseudorandom number generator satisfy the predetermined conditions, and a selector that, as a result of the determination, outputs the pseudorandom numbers generated by the pseudorandom number generator to the arithmetic circuit if the predetermined conditions are satisfied and the external memory circuit does not output the supermask, and outputs the pseudorandom numbers generated by the pseudorandom number generator to the arithmetic circuit according to the supermask if the predetermined conditions are not satisfied and the external memory circuit outputs the supermask.

[0011] Such a neural network implementation device outputs pseudorandom numbers to the arithmetic circuit when the pseudorandom numbers generated by the pseudorandom number generator satisfy predetermined conditions, even when the supermask is not output from the external memory circuit. Therefore, even when the supermask is not output from the external memory circuit, a subset network can be implemented in the arithmetic circuit.

[0012] In another embodiment, in these neural network implementation devices, the predetermined condition is that the value exceeds the lower limit of a predetermined numerical range and falls below the upper limit of the predetermined numerical range, or that the value is greater than or equal to the lower limit of a predetermined numerical range and less than or equal to the upper limit of the predetermined numerical range, wherein the predetermined numerical range is the range of weights in which the number of occurrences of each weight of each edge in the subset network is greater than or equal to a predetermined threshold.

[0013] Since the predetermined conditions are defined based on a numerical range of weights in which the number of occurrences (frequency) of each weight is above a predetermined threshold, the number of weights that satisfy the predetermined conditions, i.e., the number of edges, increases, and therefore the neural network implementation device can further reduce the amount of data transferred in the supermask.

[0014] Another aspect of the present invention relates to a neural network implementation method, which involves implementing a subset network extracted from a neural network having pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not the edge is valid, into an arithmetic circuit, wherein the subset network is output from an external memory circuit that stores supermask information representing the supermask to the arithmetic circuit, and the mask output step does not output the supermask to the arithmetic circuit if the edge weights of the subset network satisfy predetermined conditions.

[0015] The neural network implementation method based on the above concept does not output the supermask to the arithmetic circuit when the edge weights of the subset network satisfy predetermined conditions, thus reducing the amount of data transferred by the supermask when a subset network of the neural network is implemented in the arithmetic circuit.

[0016] Another aspect of the present invention is a neural network implementation program that causes a computer to function as one of the neural network implementation devices described above.

[0017] According to this, a neural network implementation program can be provided, and this neural network implementation program will have the same effects as the neural network implementation devices described above.

[0018] Another aspect of the present invention provides a neural network device comprising any of the above-described neural network implementation devices and the arithmetic circuit.

[0019] According to this, a neural network device including any of the above-described neural network implementation devices can be provided. Since this neural network device includes any of the above-described neural network implementation devices, when implementing a partial network of a neural network in an arithmetic circuit, the data transfer amount of the super mask can be reduced.

[0020] A neural network device according to another aspect of the present invention includes the above-described neural network implementation device and the arithmetic circuit, and the arithmetic circuit includes the pseudo-random number generator, the condition determination device, and the selector. Preferably, in the above-described neural network device, the arithmetic circuit includes the pseudo-random number generator, the condition determination device, the selector, and an arithmetic unit in which the neural network is implemented (that is, an arithmetic unit in which a partial network of the neural network is implemented).

[0021] Such a neural network device can be composed of, for example, an external memory circuit that stores super mask information and an arithmetic circuit such as an inference LSI, since the arithmetic circuit includes the pseudo-random number generator, the condition determination device, and the selector.

Effects of the Invention

[0022] The neural network implementation device, the neural network implementation method, and the neural network implementation program according to the present invention can reduce the data transfer amount of the super mask when implementing a partial network of a neural network in an arithmetic circuit. According to the present invention, a neural network device including the neural network implementation device can be provided.

Brief Description of the Drawings

[0023] It is a diagram for explaining the idea of the present invention. [Figure 1] It is a block diagram showing the configuration of a neural network device including the neural network implementation device in an embodiment. [Figure 3] As an example, it is a diagram for explaining an original super mask (original super mask sequence) and a shortened super mask (shortened super mask sequence) obtained by shortening a part thereof. [Figure 4] As an example, it is a diagram for explaining a neural network and a partial network thereof. [Figure 5] As an example, it is a diagram for explaining an edge where a super mask is not output and an edge where a super mask is output. [Figure 6] As an example, it is a graph showing the relationship between condition setting, model accuracy, and the internal processing ratio of an arithmetic circuit. [Figure 7] As an example, it is a graph showing each power consumption associated with data transfer in each of an example and a comparative example.

Embodiments for Carrying Out the Invention

[0026] A pseudorandom number generator typically generates pseudorandom numbers using a normal distribution, as shown in Figure 1A. In the neural network of Hidden Neural Network Theory, each edge is assigned a weight based on one of these normally distributed pseudorandom numbers. In Hidden Neural Network Theory, the scores of each edge in the neural network, each assigned a pseudorandom number as a weight, are machine-learned. Edges with scores above a predetermined threshold are extracted, and a subset network is selected (extracted) from the neural network. The inventor investigated the frequency of occurrence of each weight of each edge in this subset network and found that each weight exhibits a bimodal distribution symmetrical with respect to weight value 0, as shown in Figure 1B, and that each weight appears frequently and is unevenly distributed within specific numerical ranges. In the example shown in Figure 1B, each weight appears frequently and is unevenly distributed within the first and second numerical ranges RG1 and RG2. Therefore, the inventor conceived the idea that by setting this unevenly distributed numerical range as a predetermined condition, and instead of outputting a supermask corresponding to the weight edge satisfying this predetermined condition from the external memory circuit to the arithmetic circuit, pseudorandom numbers generated by a pseudorandom number generator without relying on a supermask can be assigned to the edge, thereby reducing the amount of data transferred by the supermask.

[0027] The neural network device in this embodiment comprises a neural network implementation device and an arithmetic circuit. This neural network implementation device is a device that implements a subset network extracted from a neural network, which has pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into the arithmetic circuit. The neural network implementation device based on the above concept comprises an external memory circuit that stores supermask information representing the supermask, and this external memory circuit does not output the supermask to the arithmetic circuit when the edge weights of the subset network satisfy predetermined conditions. The neural network device comprising such a neural network implementation device, as well as the neural network implementation method and neural network implementation program implemented in the neural network implementation device, will be described in more detail below.

[0028] Figure 2 is a block diagram showing the configuration of a neural network device equipped with a neural network implementation device in an embodiment. Figure 3 is a diagram illustrating, as an example, the original supermask (original supermask sequence) and a shortened supermask (shortened supermask sequence) obtained by shortening a part of it. The upper part of Figure 3 shows the original supermask (original supermask sequence) for extracting (selecting) a subset network from a neural network using hidden neural network theory, and the lower part of Figure 3 shows a shortened supermask (shortened supermask sequence) obtained by shortening a part of the original supermask shown in the upper part by removing supermasks that are not output.

[0029] The neural network device 1000 in this embodiment includes, for example, a neural network implementation device as shown in Figure 2, and comprises an external memory circuit EM and an arithmetic circuit AC.

[0030] The external memory circuit EM is connected to the arithmetic circuit AC by a data bus, stores predetermined data, and outputs predetermined data to the arithmetic circuit AC. The external memory circuit EM is composed of memory elements (memory circuits) and peripheral circuits, such as rewritable non-volatile memory elements like EEPROM (Electrically Erasable Programmable Read Only Memory) and volatile memory elements like RAM (Random Access Memory). Examples of the predetermined data stored in the external memory circuit EM include shortened supermask information and seed number information.

[0031] The shortened supermask information represents a shortened supermask (shortened supermask) obtained by removing the supermask corresponding to an edge whose weight in a subset network satisfies a predetermined condition from the original supermask. The original supermask is a flag (edge ​​validity flag) SP that indicates whether or not an edge is valid for each edge in the neural network, in order to represent a subset network of the neural network in Hidden Neural Network Theory. The original supermask is, for example, as shown in the upper part of Figure 3, a numerical sequence (original supermask sequence) in which the edge validity flag SP, which indicates whether or not an edge is valid, is arranged sequentially in a column from the first edge of the first layer to the last edge of the final layer in the neural network. An edge validity flag SP = "1" indicates that the edge is valid as an edge in the subset network, and an edge validity flag SP = "0" indicates that the edge in the subset network is not valid (invalid). Note that the edge validity flags "1" and "0" may be defined in the opposite way to those described above. In the example shown in the upper part of Figure 3, the pseudo-random number weights WI assigned to the 1st, 9th, 10th, 12th, and 17th edges satisfy the predetermined conditions, and the supermasks (edge ​​validity flags SP) for these 1st, 9th, 10th, 12th, and 17th edges are removed from the original supermask, thereby generating the shortened supermask shown in the lower part of Figure 3. In the example shown in Figure 3, the shortened supermask information represents the numerical sequence (shortened supermask sequence) of each edge validity flag SP in the shortened supermask shown in the lower part of Figure 3.

[0032] The seed number information represents the seed number used in the pseudorandom number generator 1. Pseudorandom numbers are random numbers obtained by a sequence of pseudorandom numbers calculated by a deterministic calculation based on a so-called pseudorandom number generation method, and the sequence of pseudorandom numbers is specified and identified by a so-called seed number (seed value). Therefore, each weight of each edge of the neural network in Hidden Neural Network Theory, which is assigned by the pseudorandom numbers, can always be reproduced by the seed number. That is, by using the same seed number, each weight of each edge of the neural network will always be the same.

[0033] Various algorithms are known for the aforementioned pseudorandom number generation methods, such as the squared random number generation method, the linear congruential random number generation method, the Mersenne Twister, and the Lagged Fibonacci method.

[0034] The arithmetic circuit AC is a circuit that performs operations such as arithmetic operations and logical operations, and is composed of an integrated circuit such as an inference LSI or FPGA (Field Programme Gate Array) and its peripheral circuits. The arithmetic circuit AC comprises a pseudo-random number generator (pseudo-random number generation circuit) 1, an arithmetic unit 2, a synchronous unit 3, a conditional determination unit 4, and a selection switch 5.

[0035] The pseudo-random number generator 1 generates pseudo-random numbers and outputs them to the condition determination unit 4 and the selection switch 5, respectively. The pseudo-random number generator 1 generates pseudo-random numbers in a sequence corresponding to the seed number represented by the seed number information output from the external memory circuit EM.

[0036] The condition determiner 4 determines whether the pseudo-random number generated by the pseudo-random number generator 1 satisfies a predetermined condition. As a result of this determination, when the condition determiner 4 satisfies the predetermined condition, it outputs a flag (determination flag) JF = "1" indicating that the predetermined condition is satisfied to each of the selection switch 5 and the synchronizer 3. When the predetermined condition is not satisfied, a determination flag JF = "0" indicating that the predetermined condition is not satisfied is output to each of the selection switch 5 and the synchronizer 3. Note that "1" and "0" of the determination flag JF may be defined conversely to the above. The predetermined condition is, for example, exceeding the lower limit value of a predetermined numerical range and being less than the upper limit value of the predetermined numerical range, or being greater than or equal to the lower limit value of the predetermined numerical range and being less than or equal to the upper limit value of the predetermined numerical range. The predetermined numerical range is a weight range in which the number of occurrences of each weight of each edge in the partial network is equal to or greater than a predetermined threshold value (range setting threshold value). The range setting threshold value is appropriately set in advance. For example, in the example shown in FIG. 1C, when the lower limit value in the first numerical range RG1 is RL1, the upper limit value is RU1, the lower limit value in the second numerical range RG2 is RL2, and the upper limit value is RU2 (RL1 < RU1 < 0, 0 < RL2 < RU2), the predetermined condition is that the weight WI satisfies at least one of RL1 < WI < RU1 and RL2 < WI < RU2, or the weight WI satisfies at least one of RL1 ≦ WI ≦ RU1 and RL2 ≦ WI ≦ RU2.

[0037] Synchronizer 3 retrieves the supermask (edge ​​effectiveness flag SP) represented by the shortened supermask information from the external memory circuit EM, and outputs the retrieved shortened supermask (edge ​​effectiveness flag SP) to the selection switch 5 in synchronization with the output timing of the original supermask, based on the judgment result of the condition determination unit 4. As described above, the shortened supermask has the supermask corresponding to the edge that satisfies a predetermined condition removed from the original supermask, so it is necessary to output the shortened supermask at the output timing of the original supermask. Synchronizer 3 is provided for this purpose. In the example shown in Figure 3, the supermasks (edge ​​effectiveness flag SP) for the 1st, 9th, 10th, 12th, and 17th edges (1st, 9th, 10th, 12th, and 17th edges) have been removed, so at the respective output timings TC1, TC9, TC10, TC12, and TC17, the supermasks (edge ​​effectiveness flag SP) are not retrieved from the external memory circuit EM and are not output from synchronous 3 to the selection switch 5. Furthermore, the synchronous unit 3 may be equipped with an internal FIFO memory that functions as a buffer memory for storing one or more supermasks (edge ​​validity flags) of the shortened supermask, in order to retrieve the supermask (edge ​​validity flag SP) from the external memory circuit EM at each output timing and output the shortened supermask's supermask (edge ​​validity flag SP) to the selection switch 5 at the output timing of the original supermask.

[0038] More specifically, if the condition determination unit 4 inputs the determination flag JF="1", the synchronous unit 3 does not output the supermask (edge ​​effectiveness flag SP) to the select switch 5. If the condition determination unit 4 inputs the determination flag JF="0", the synchronous unit 3 outputs the supermask (edge ​​effectiveness flag SP) to the select switch 5. This allows the synchronous unit 3 to output the supermask (edge ​​effectiveness flag SP) of the shortened supermask to the select switch 5 at the output timing of the original supermask, based on the determination result of the condition determination unit 4.

[0039] The selection switch 5 outputs pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic unit 2 based on the supermask (edge ​​effectiveness flag SP) represented by the shortened supermask information output from the external memory circuit EM via the synchronous unit 3, and the judgment flag JF output from the condition determination unit 4. More specifically, if the condition determination unit 4 determines that the predetermined conditions are met and the external memory circuit EM has not output the supermask of the shortened supermask, the selection switch 5 outputs pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic unit 2 of the arithmetic circuit AC. If the predetermined conditions are not met and the external memory circuit EM has output the supermask of the shortened supermask, the selection switch 5 outputs pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic unit 2 of the arithmetic circuit AC according to this supermask. In other words, if the judgment flag JF = "1" and the external memory circuit EM has not output the supermask of the shortened supermask, the selection switch 5 outputs pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic unit 2 of the arithmetic circuit AC. Then, when the determination flag JF = "0" and the supermask of the shortened supermask is output from the external memory circuit EM, the selection switch 5 outputs the pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic unit 2 of the arithmetic circuit AC when the edge validity flag of the supermask = "1", and outputs the value 0, which masks the edges, to the arithmetic unit 2 of the arithmetic circuit AC when the edge validity flag of the supermask = "0".

[0040] More specifically, in this embodiment, as explained with reference to Figure 3, the supermask corresponding to an edge whose weight in the subset network satisfies a predetermined condition is removed. Therefore, whether or not the supermask of the shortened supermask is output from the external memory circuit EM can be determined by the determination flag JF. That is, if the determination flag JF = "1", it means that the pseudorandom numbers generated by the pseudorandom number generator, i.e., the edge weights, satisfy the predetermined condition, and the supermask of the shortened supermask is not output from the external memory circuit EM. If the determination flag JF = "0", it means that the pseudorandom numbers generated by the pseudorandom number generator, i.e., the edge weights, do not satisfy the predetermined condition, and the supermask of the shortened supermask is output from the external memory circuit EM. Therefore, if the determination flag JF = "1", the selection switch 5 outputs the pseudorandom numbers generated by the pseudorandom number generator to the arithmetic unit 2 of the arithmetic circuit AC. When the determination flag JF = "0", the selection switch 5 outputs a pseudorandom number generated by the pseudorandom number generator to the arithmetic unit 2 of the arithmetic circuit AC if the edge effectiveness flag of the shortened supermask output from the external memory circuit EM via the synchronous unit 3 is "1", and when the determination flag JF = "0", the selection switch 5 outputs a value of 0 to mask the edges to the arithmetic unit 2 of the arithmetic circuit AC if the edge effectiveness flag of the shortened supermask is "0".

[0041] If the external memory circuit EM is not outputting the supermask of the shortened supermask, the input terminal of the synchronous unit 3 may maintain the output voltage output from the external memory circuit EM at the previous output timing, and the input terminal of the selector switch 5 may maintain the output voltage output from the synchronous unit 3 at the previous output timing. If the output voltage at the previous output timing was at a low level, the low level is maintained, and if the output voltage at the previous output timing was at a high level, the high level is maintained. This prevents power consumption due to current flowing when switching between low and high levels (for example, current flowing through parasitic capacitance), thereby reducing power consumption.

[0042] Calculator 2 is a circuit within the arithmetic circuit AC that implements a subset of the neural network in the Hidden Neural Network Theory.

[0043] In a neural network device 1000 equipped with such a neural network implementation device, the external memory circuit EM and the arithmetic circuit AC can be configured, for example, by a computer such as a desktop or notebook computer.

[0044] Next, we will explain how to implement a subset of a neural network in a computational circuit, as described in Hidden Neural Network Theory.

[0045] Figure 4 is a diagram illustrating a neural network and its subsystems as an example. Figure 4A shows the neural network, and Figure 4B shows the subsystems. Figure 5 is a diagram illustrating edges where a supermask is not output and edges where a supermask is output as an example. Figure 5A shows edges where a supermask is not output, and Figure 5B shows edges where a supermask is output.

[0046] Firstly, a subset of the neural network in Hidden Neural Network Theory is generated. More specifically, a training dataset containing multiple training data sets is prepared. The training data set includes input data and output data that serves as training data corresponding to the input data. Next, a neural network is prepared, and the weights of each edge in the neural network are assigned by pseudo-random numbers generated with a predetermined seed number. Subsequently, each score of each edge is machine-learned using the training dataset, and the scores of each edge are obtained. Next, scores that are above a predetermined score threshold are extracted from the scores of each edge. Then, a subset of the neural network formed by the extracted edges and their nodes is selected and generated from the neural network. This generates a subset of the neural network in Hidden Neural Network Theory. For example, the subset shown in Figure 4B is selected and generated from the neural network shown in Figure 4A. The neural network shown in Figure 4A is a fully connected 4-layer network. The first layer (input layer) has two nodes ND11 and ND12, the second layer (hidden layer) has three nodes ND21, ND22 and ND23, the third layer (hidden layer) has three nodes ND31, ND32 and ND33, and the fourth layer (output layer) has one node ND41. Since each of these nodes ND is fully connected, there are 18 first to 18th edges EG111, EG112, EG113, EG121, EG122, EG123; EG211, EG212, EG213, EG221, EG222, EG223, EG231, EG232, EG233; EG311, EG321, and EG331.For this neural network, machine learning is performed on the scores of each edge to extract the first edge EG111, the second edge EG112, the fifth edge EG122, the sixth edge EG123, the ninth edge EG213, the tenth edge EG221, the eleventh edge EG222, the twelfth edge EG223, the thirteenth edge EG231, the fifteenth edge EG233, the sixteenth edge EG311, the seventeenth edge EG321, and the eighteenth edge EG331, and the subset network shown in Figure 4B is selected and generated.

[0047] The supermask (original supermask sequence) for generating the subset network shown in Figure 4B is as follows, as shown in the upper part of Figure 3: "1", "1", "0", "0", "1", "1", "0", "0", "1", "1", "1", "1", "0", "1", "1", "1", "1".

[0048] Secondly, a shortened supermask is generated. More specifically, first, a predetermined condition is set (defined) by the user (operator). The predetermined condition may be set, for example, by examining the number of occurrences (frequency of occurrence) of each weight of each edge in the subset network and setting it according to the results of the examination, or it may be set, for example, based on the user's past experience. In this embodiment, a range setting threshold is set by the user, a numerical range is set according to the range determination threshold, and the predetermined condition is set according to the numerical range. Subsequently, the user examines each weight of each edge in the subset network to see whether the weight of that edge satisfies the predetermined condition, and edges that satisfy the predetermined condition are extracted. For example, in the subset network shown in Figure 4B, as shown in Figure 5A, the weights of the first edge EG111, the ninth edge EG213, the tenth edge EG221, the twelfth edge EG223, and the seventeenth edge EG321 all satisfy the predetermined condition. Then, a shortened supermask is generated by the user. In the above example, a shortened supermask is generated for each of the remaining edges shown in Figure 5B, namely the second edge EG112, the fifth edge EG122, the sixth edge EG123, the eleventh edge EG222, the thirteenth edge EG231, the fifteenth edge EG233, the sixteenth edge EG311, and the eighteenth edge EG331, as shown in Figure 5B, after removing the first edge EG111, the ninth edge EG213, the tenth edge EG221, the twelfth edge EG223, the sixteenth edge EG311, and the eighteenth edge EG331, as shown in Figure 5B. In this example shown in Figure 5B, the shortened supermask (shortened supermask sequence) is "1", "0", "0", "1", "1", "0", "0", "1", "1", "0", "1", "1", "1", "1".

[0049] Thirdly, a subset network is implemented in the arithmetic circuit AC based on the shortened supermask information and the pseudo-random numbers generated by the pseudo-random number generator 1.

[0050] More specifically, the external memory circuit EM first stores (remembers) the shortened supermask information representing the shortened supermask generated by the user (operator), and the seed number information representing the seed number.

[0051] When implementation begins, the seed number, represented by seed number information, is read from the external memory circuit EM to the pseudo-random number generator 1 of the arithmetic circuit AC, and pseudo-random numbers are generated by the pseudo-random number generator 1 according to the pseudo-random number sequence corresponding to the seed number. Subsequently, the pseudo-random number generator 1 outputs the generated pseudo-random numbers to the condition determination unit 4 and the selection switch 5, respectively, and the condition determination unit 4 determines whether the pseudo-random numbers satisfy the predetermined conditions. Subsequently, if the predetermined conditions are satisfied, the condition determination unit 4 outputs the determination flag JF="1" to the selection switch 5 and the synchronous unit 3, respectively, and if the predetermined conditions are not satisfied, it outputs the determination flag JF="0" to the selection switch 5 and the synchronous unit 3, respectively. Subsequently, if the condition determination unit 4 inputs the determination flag JF="1", the synchronous unit 3 does not output the supermask (edge ​​effectiveness flag SP) to the selection switch 5, and if the condition determination unit 4 inputs the determination flag JF="0", the synchronous unit 3 outputs the supermask (edge ​​effectiveness flag SP) to the selection switch 5. Then, the selection switch 5 outputs a pseudo-random number generated by the pseudo-random number generator 1 to the arithmetic unit 2 based on the edge validity flag SP of the shortened supermask read from the external memory circuit EM and the determination flag JF output from the condition determination unit 4. In this embodiment, when the determination flag JF = "1", the selection switch 5 outputs a pseudo-random number generated by the pseudo-random number generator 1 to the arithmetic unit 2 of the arithmetic circuit AC. When the determination flag JF = "0", if the edge validity flag SP = "1", the selection switch 5 outputs a pseudo-random number generated by the pseudo-random number generator to the arithmetic unit 2 of the arithmetic circuit AC, and when the edge validity flag SP = "0", it outputs 0 to mask the edges to the arithmetic unit 2 of the arithmetic circuit AC. As a result, the subset network is implemented in the arithmetic unit 2 of the arithmetic circuit AC.

[0052] In the examples shown in Figures 3 to 5, the seed number, represented by seed number information, is read from the external memory circuit EM to the pseudo-random number generator 1 of the arithmetic circuit AC. The pseudo-random number generator 1 generates pseudo-random numbers according to the pseudo-random number sequence corresponding to the seed number, and these pseudo-random numbers are output to the condition determination unit 4 and the selection switch 5, respectively. At output timing TC1, the condition determination unit 4 determines that the pseudo-random numbers satisfy the predetermined conditions and outputs the determination flag JF=“1” to the selection switch 5 and the synchronous unit 3, respectively. Since the determination flag JF=“1” is input from the condition determination unit 4, the synchronous unit 3 does not read the supermask (edge ​​effectiveness flag SP) of the shortened supermask, and the external memory circuit EM does not output the supermask (edge ​​effectiveness flag SP) of the shortened supermask. Since the determination flag JF=“1”, the selection switch 5 outputs the pseudo-random numbers generated by the pseudo-random number generator 1 to the arithmetic unit 2. Next, at output timing TC2, the condition determination unit 4 determines that the pseudo-random number does not satisfy the predetermined conditions and outputs the determination flag JF="0" to the select switch 5 and the synchronous unit 3, respectively. Since the condition determination unit 4 inputs the determination flag JF="0", the synchronous unit 3 reads the supermask of the shortened supermask (edge ​​effectiveness flag SP="1") and outputs it to the select switch 5. Since the determination flag is "0" and the edge effectiveness flag SP="1", the select switch 5 outputs the pseudo-random number generated by the pseudo-random number generator 1 to the arithmetic unit 2. Next, at output timing TC3, the condition determination unit 4 determines that the pseudo-random number does not satisfy the predetermined conditions and outputs the determination flag JF="0" to the select switch 5 and the synchronous unit 3, respectively. Since the condition determination unit 4 inputs the determination flag JF="0", the synchronous unit 3 reads the supermask of the shortened supermask (edge ​​effectiveness flag SP="0") and outputs it to the select switch 5. Since the selection switch 5 has a judgment flag of "0" and an edge validity flag SP of "0", it outputs a value of 0, which masks the edge, to the arithmetic unit 2 of the arithmetic circuit AC. Subsequently, at output timing TC4, since the pseudorandom number does not satisfy the predetermined conditions, the same processing as at output timing TC3 is performed.Next, at output timing TC5, since the pseudorandom number satisfies the predetermined conditions, the same processing as at output timing TC2 is performed. Next, at output timing TC6, since the pseudorandom number satisfies the predetermined conditions, the same processing as at output timing TC2 is performed. Next, at output timing TC7, since the pseudorandom number does not satisfy the predetermined conditions, the same processing as at output timing TC3 is performed. Next, at output timing TC8, since the pseudorandom number does not satisfy the predetermined conditions, the same processing as at output timing TC3 is performed. Next, at output timing TC9, since the pseudorandom number satisfies the predetermined conditions, the same processing as at output timing TC1 is performed. The same processing is performed for each output timing TC thereafter, and each edge is assigned a pseudorandom number as a weight, resulting in the subset network shown in Figure 4B being implemented in the arithmetic unit 2 of the arithmetic circuit AC.

[0053] When the aforementioned subsystem is implemented in the arithmetic unit 2 of the arithmetic circuit AC, the Activation signal, which is the output of the processing circuit (not shown) preceding the arithmetic circuit AC, is input to the arithmetic unit 2 as input data, and the subsystem implemented in the arithmetic unit 2 of the arithmetic circuit AC performs inference and outputs it as an inference model.

[0054] Furthermore, the synchronous unit 3 and the selection switch 5 are examples of selectors that, as a result of the determination, if the predetermined conditions are met and the supermask is not output from the external memory circuit, output the pseudorandom numbers generated by the pseudorandom number generator to the arithmetic circuit, and if the predetermined conditions are not met and the supermask is output from the external memory circuit, output the pseudorandom numbers generated by the pseudorandom number generator according to the supermask to the arithmetic circuit.

[0055] Furthermore, the external memory circuit EM, pseudo-random number generator 1, synchronous unit 3, conditional determination unit 4, and selection switch 5 correspond to an example of a neural network implementation device that implements a subset network extracted from a neural network, which has pseudo-random numbers generated by a pseudo-random number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into an arithmetic circuit.

[0056] In the above description, the synchronous unit 3 was provided with the arithmetic circuit AC, but it may be configured as a separate circuit from the arithmetic circuit AC. That is, the synchronous unit 3 may be configured as a circuit placed between the external memory circuit EM and the arithmetic circuit AC.

[0057] As described above, the neural network implementation device provided in the neural network device 1000 in the embodiment, as well as the neural network implementation method and neural network implementation program implemented therein, are based on the above concept. Therefore, when the edge weights of the subset network satisfy predetermined conditions, the supermask (edge ​​effectiveness flag SP) is not output to the arithmetic circuit AC. Thus, when a subset network of a neural network is implemented in the arithmetic circuit AC, the amount of data transfer for the supermask (edge ​​effectiveness flag SP) can be reduced. The neural network implementation device, method, and program can reduce the amount of data transfer for the supermask (edge ​​effectiveness flag SP) by the amount of data that is not output to the arithmetic circuit AC. Since the amount of data transfer can be reduced, the power consumption associated with data transfer can be reduced.

[0058] The above-described neural network implementation device, method, and program output the pseudorandom numbers generated by the pseudorandom number generator 1 to the arithmetic circuit AC when the pseudorandom numbers generated by the pseudorandom number generator 1 satisfy predetermined conditions, even when the external memory circuit EM does not output a supermask (edge ​​effectiveness flag SP). Therefore, even when the external memory circuit EM does not output a supermask (edge ​​effectiveness flag SP), a subset network can be implemented in the arithmetic circuit AC.

[0059] The above-described neural network implementation device, method, and program define the predetermined conditions based on a numerical range of weights in which the number of occurrences (frequency of occurrence) of each weight is equal to or greater than a predetermined threshold. As a result, the number of weights that satisfy the predetermined conditions, i.e., the number of edges, increases, thus further reducing the amount of data transferred for the supermask (edge ​​effectiveness flag SP).

[0060] Since the above-mentioned neural network device 1000 includes a neural network implementation device, the amount of data transfer for the supermask (edge ​​effectiveness flag SP) can be reduced when implementing a subset of the neural network in the arithmetic circuit AC.

[0061] The above-described neural network device 1000 includes an arithmetic circuit AC which comprises a pseudo-random number generator 1, a conditional determiner 4, a synchronous circuit 3 as an example of a selector, and a selection switch 5. Therefore, it can be composed of, for example, an external memory circuit EM for storing supermask information and an arithmetic circuit AC such as an inference LSI.

[0062] An embodiment and a comparative example will be described. Figure 6 is a graph showing, as an example, the relationship between the condition setting, model accuracy, and the internal processing rate of the arithmetic circuit. The horizontal axis of Figure 6 represents the width of the numerical range for the predetermined conditions, the left vertical axis represents the model accuracy, and the right vertical axis represents the internal processing rate of the arithmetic circuit for supermask generation. Figure 7 is a graph showing, as an example, the power consumption associated with data transfer in the embodiment and comparative example, respectively. The vertical axis of Figure 7 represents power consumption [μW·s].

[0063] The neural network in this embodiment is a fully connected 4-layer network, implemented in Python, a programming language with a rich library suitable for implementing artificial intelligence. The first layer, the input layer, has 1 input and 32 outputs, and the convolution process is performed by shifting one step at a time using a 3x3 canelle. The second layer, the hidden layer, has 32 inputs and 64 outputs, and the convolution process is performed by shifting one step at a time using a 3x3 canelle. The third layer, the hidden layer, has 9216 inputs and 128 outputs. The fourth layer, the output layer, has 128 inputs and 10 outputs. The reason the fourth layer has 10 outputs is that, as will be described later, the input data of the MINIST training dataset is handwritten images with integer values ​​from 0 to 9, and its output data is also integer values ​​from 0 to 9. An NVIDIA Jetson AGX Xavier computer was used. The MINIST (Mixed National Institute of Standards and Technology database) dataset was used as the training dataset, with 60,000 data points used for machine learning training and 10,000 data points used for validation. Pseudorandom weights were assigned to each edge of the neural network, and the scores were machine-learned using the training dataset. Subnetworks were then extracted from the neural network based on the scores.

[0064] In the subset networks extracted in this way, the width (range setting threshold) of the first and second numerical ranges RG1 and RG2 subject to the predetermined conditions was changed so that they included pseudorandom numbers with high occurrence frequency as weights, and the model accuracy and the internal processing ratio of the arithmetic circuit for generating the supermask were investigated. A general-purpose desktop personal computer was used for this investigation of the internal processing ratio of the arithmetic circuit. The results are shown in Figure 6. In Figure 6, the line graph α represents the model accuracy, and the bar graph represents the internal processing ratio of the arithmetic circuit. The model accuracy was obtained by running the simulation multiple times on the general-purpose desktop personal computer while changing the numerical ranges RG1 and RG2. The internal processing ratio of the arithmetic circuit is the ratio of the number of weights in the numerical ranges RG1 and RG2 to the number of weights in the neural network ((internal processing ratio of the arithmetic circuit) = (number of weights in numerical ranges RG1 and RG2) / (number of weights in the neural network)). The line β represents the model accuracy of the comparative example. This comparative example implements a subset network using the original supermask (original supermask sequence). In Figure 6, the width of the first and second numerical ranges RG1 and RG2 are varied in eight different ways. In this example, as can be seen from Figure 6, the narrower the first and second numerical ranges RG1 and RG2 are (the larger the range setting threshold), the smaller the internal processing ratio of the arithmetic circuit. Conversely, as the first and second numerical ranges RG1 and RG2 are widened (the smaller the range setting threshold), the internal processing ratio of the arithmetic circuit increases. In this example, the model accuracy does not fluctuate much even when the width of the first and second numerical ranges RG1 and RG2 is changed, but in the widest case (the rightmost example in Figure 6), the model accuracy decreases. In this example, it can be expected that the model accuracy will be maintained even if the internal processing ratio of the arithmetic circuit is set to 40% (the second example from the right in Figure 6).

[0065] The power consumption associated with data transfer was investigated for both the example (second from the right in Figure 6) and the comparative example when the internal processing ratio of this arithmetic circuit was 40%. The results are shown in Figure 7. In this example, a power consumption reduction of approximately 36.4% can be expected compared to the comparative example.

[0066] To illustrate the present invention, the embodiments have been adequately and fully described above with reference to the drawings. However, those skilled in the art should recognize that it is easy to modify and / or improve upon the embodiments described above. Therefore, unless such modifications or improvements implemented by those skilled in the art fall outside the scope of the claims, such modifications or improvements shall be considered to be included within the scope of the claims. [Explanation of Symbols]

[0067] 1000 Neural Network Devices EM external memory circuit AC Calculation Circuit 1. Pseudorandom number generator 2 Arithmetic unit 3 Synchronizer 4 Condition judger 5 selection switches

Claims

1. A neural network implementation device that implements a subset network extracted from a neural network, which has pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into an arithmetic circuit, It includes an external memory circuit that stores supermask information representing the aforementioned supermask, The external memory circuit does not output the supermask to the arithmetic circuit when the edge weights of the subset network satisfy predetermined conditions. A neural network implementation device.

2. A pseudo-random number generator that generates the aforementioned pseudo-random numbers, A condition determination device that determines whether the pseudorandom numbers generated by the pseudorandom number generator satisfy the predetermined conditions, The system includes a selector that, as a result of the determination, if the predetermined conditions are met and the supermask is not output from the external memory circuit, outputs a pseudorandom number generated by the pseudorandom number generator to the arithmetic circuit, and if the predetermined conditions are not met and the supermask is output from the external memory circuit, outputs a pseudorandom number generated by the pseudorandom number generator according to the supermask to the arithmetic circuit. The neural network implementation device according to claim 1.

3. The aforementioned predetermined conditions are that the value exceeds the lower limit of a predetermined numerical range and falls below the upper limit of the predetermined numerical range, or that the value is greater than or equal to the lower limit of a predetermined numerical range and less than or equal to the upper limit of the predetermined numerical range. The predetermined numerical range is the range of weights in which the number of occurrences of each weight at each edge in the subset network is equal to or greater than a predetermined threshold. The neural network implementation device according to claim 1.

4. A neural network implementation method that involves implementing a subset network extracted from a neural network, which uses pseudorandom numbers generated by a pseudorandom number generator as edge weights, by masking each edge with a supermask indicating whether or not that edge is valid, into an arithmetic circuit, The system includes a mask output step that outputs the supermask from an external memory circuit that stores supermask information representing the supermask to the calculation circuit, The mask output step does not output the supermask to the calculation circuit if the edge weights of the subset network satisfy predetermined conditions. Methods for implementing neural networks.

5. A neural network implementation program for causing a computer to function as a neural network implementation device according to any one of claims 1 to 3.

6. A neural network implementation device according to any one of claims 1 to 3, The arithmetic circuit is provided as described above. A neural network device.

7. A neural network implementation device according to claim 2, The arithmetic circuit is provided, The calculation circuit comprises the pseudo-random number generator, the conditional determination unit, and the selector. A neural network device.

Images