Neural network circuit device

Abstract

The present invention comprises: an AND operation circuit for calculating the AND of a vector from a learning phase and a vector from an inference phase; a first counter that counts the number of instances of input in which the value is 1 in the vector input at the time of the inference phase; a second counter that counts the number of instances of input in which the value is 1 in an AND vector obtained as a result of AND operation by the AND operation circuit; a third counter that counts the number of instances of input in which the value is 1 in the vector at the time of the learning phase; an addition circuit (507) that adds the output of the first counter and the output of the third counter; a shift resistor (508) that shifts the result of the second counter by 1 bit toward the higher side; and a division circuit (509) that divides an output vector from the shift resistor (508) by an output vector from the addition circuit (507).

Description

[Technical Field]

[0001] The present invention relates to a neural network circuit device. [Background technology]

[0002] In recent years, artificial intelligence technology using artificial neural networks has advanced, and various industrial applications are being developed. Such neural networks are characterized by the use of a network of connected perceptrons, which model nerve cells. A neural network performs calculations based on inputs to the entire network and outputs the calculation results.

[0003] Perceptrons, used in artificial neural networks, are an extension of earlier models of neurons.

[0004] FIG. 32 illustrates the operation of a perceptron 200 with a variable constant input. As shown in Figure 32, the N+1 input values ​​are b, x1, x2, ... x N are input to the perceptron 200. Of these, there are N external inputs to the entire neural network, and input i has the input value x i is input. b is a constant value stored inside the neural network. Also, one output y is output from the perceptron as the output of the neural network. For input i (i=1, 2, ...N), a value called a weight w i are assigned (hereinafter referred to as synaptic weights). In this case, the output y is expressed by equation (1).

[0005]

number

[0006] Here, f(·) represents the activation function. Commonly used activation functions include nonlinear functions such as the sigmoid function and tanh function, and the ReLU (Rectified Linear Unit function). In equation (1), w i x i To eliminate the difference in notation of and b and to make the equation easier to read, a circuit like that shown in Figure 13, where the constant input is 1 and the corresponding synaptic weight w0 is b, and the following equation (2) are often used. Figure 33 shows the operation of a perceptron 200, which generalizes the representation of input and synaptic weights.

[0007]

number

[0008] As shown in equation (2), the value passed to the activation function is calculated based on the input value, and the output value is calculated by the activation function. In the following explanation, the value passed to the activation function will be called the activation. When the activation function is expressed as f(a), a is the activation. Typically, when performing machine learning using an artificial neural network, a network in which one or more perceptrons 200 are connected hierarchically is used, as shown in Figure 34. Figure 34 is a diagram showing a multi-layered artificial neural network.

[0009] The artificial neural network is i There are multiple combinations (i=1,2,…,N). Each combination is represented by j, and the input value x of combination j is i When each of (i=1,2,…,N) is considered as a component of a vector, x i (i=1,2,…,N) is a vector x j Here, x j component of x j =(x j1 ,x j2 ,…,x jN ) T Let (x j =(x j1,x j2 ,…,x jN ) T The T in (means that the vector is converted to a column vector) will be expressed as follows.

[0010] Then, for each x j For the target value l j We prepare multiple sets of data with w i The value of is determined. This value is determined by minimizing the error for the entire training data, using the difference between the value calculated by the neural network and the target value as the error.

[0011] In machine learning methods using this type of artificial neural network, the training data itself is not stored within the neural network. On the other hand, there is a machine learning method called k-nearest neighbor method, which stores training data, calculates the similarity between the input and memory patterns, and outputs labels using the k most similar memories. This k-nearest neighbor method is known to enable relatively stable learning even when there is little training data, and has advantages in some applications.

[0012] Furthermore, as described in Non-Patent Document 4, the brain is thought to have a function called pattern completion, which, when multiple external inputs are received, completely recalls a close memory that has already been established in the brain, even if an input pattern that perfectly matches the combination of those inputs is not stored in memory. Finding memories that are close to an external input pattern is one of the functions of human intelligence, and calculating the similarity between an input and a memory pattern provides basic information for finding the most similar memory. Therefore, technology for calculating the similarity between an input and a memory pattern is also important as a component technology for realizing this pattern completion.

[0013] As described above, neural networks are a fundamental technology for artificially realizing intelligent functions such as machine learning and the recall of similar memories, which are thought to be inherent in humans.

[0014] In neurons and neural networks that are the basis for perceptrons and artificial neural networks, the associative network is a technology that learns and stores information that was previously input, and compares that memory with the current input to determine similarity. Examples of the associative network are described in Non-Patent Documents 1, 2, and 3. Examples of neurons used in the associative network and the associative network are shown in Figures 35 and 36, respectively.

[0015] Figure 35 is a diagram showing an example of a simple associative network. In Figure 35, a neuron 300 is represented by a combination of an arrow and a black triangle. The upper side of the triangle (the side without the arrowhead) corresponds to the input part of the neuron, and the lower side of the triangle (the side with the arrowhead) corresponds to the output part of the neuron.

[0016] Suppose a neural network has neuron 300 that changes to a firing state (indicating that the neuron's membrane potential rises and exceeds a threshold) when input A is applied. If input B is repeatedly applied at the same time as input A, a phenomenon occurs in which neuron 300 changes to a firing state simply by input B. This phenomenon is explained by Hebb's law, which states that the simultaneous firing of neuron 300 with the neuron that generates input B strengthens the synaptic connection between input B and neuron 300. The phenomenon in which neuron 300 changes to a firing state simply by input B is called classical conditioning, and inputs A and B are called the unconditioned stimulus and the conditioned stimulus, respectively.

[0017] FIG. 36 is a diagram showing an example of an associative network including multiple unconditioned stimuli. Figure 36 shows a case where different unconditioned stimuli P, Q, and R are related to one conditioned stimulus C by classical conditioning. Unconditioned stimulus P and conditioned stimulus C are input to neuron 301. Unconditioned stimulus Q and conditioned stimulus C are input to neuron 302. Unconditioned stimulus R and conditioned stimulus C are input to neuron 303.

[0018] Next, a technique for determining similarity using an associative network will be described. Fig. 37 is a diagram illustrating a neuron 300, which is a component of the technology for determining similarity using an associative network. Fig. 37 shows the setting of synaptic weights in a simple associative network. Four input values ​​x1, x2, x3, and x4 are input to the neuron 300 in Figure 37. Here, if the input i has the input value x i are input. These input values ​​are either 0 or 1. This is related to the state of the preceding neuron that generates each input, with 0 corresponding to the non-firing state of the preceding neuron (a state in which the membrane potential of the nerve cell has not reached the threshold membrane potential state), and 1 corresponding to the firing state of the preceding neuron. In a non-firing state, neurotransmitters do not reach the connected neuron, and in a firing state, neurotransmitters do reach the neuron. Since the combination of input values ​​to a neuron can be considered as a vector with each of these as components, let x be the vector with components x1, x2, x3, and x4, and let x=(x1,x2,x3,x4). T Hereafter, this x will be called the input vector.

[0019] Synaptic weights are assigned to synapses, which are the parts where inputs connect to neurons. Inputs 1, 2, 3, and 4 are assigned w1, w2, w3, and w4, respectively. Since the combination of these synaptic weights can also be considered as a vector, we will use the same notation as the inputs and define the synaptic weight vector w as w=(w1, w2, w3, w4). T It is expressed as:

[0020] 38A to 38F are diagrams for explaining similarity calculation in the prior art. Figure 38A shows the state of the associative network during learning. Six inputs are connected to a neuron 300 in Figure 38A. In Figure 38A, the input vector x l is x l =(1,0,0,1,0,1) T By this learning, the synaptic weight vectors are set as shown in FIG. 38B. This is because when the neuron 300 shown in FIG. 38A is in the firing state, the input vector x l =(1,0,0,1,0,1) T is added, and for the components of this input vector whose value is 1, the weight of the corresponding synapse is set to 1 based on the Hebbian law. That is, w=x l This becomes:

[0021] As an example of the first similarity determination, as shown in FIG. 38C, the input vector x1 is x1=(1,0,0,1,0,1). T In other words, the same input vector as during training is also added during similarity judgment. In the Associative Network, x1 and the input x l The similarity of x is calculated as the dot product of both vectors. l ·x1. w=x l Therefore, the dot product can be rewritten as w·x1. The degree of similarity calculated in this way (hereafter referred to as dot product similarity) is 3. In this case, the activity of the neuron in Figure 43C, i.e., the value passed to the neuron's activation function to determine the output, is considered to be equal to the dot product similarity. If neuron 300 in Figure 38C has a step function with a threshold of 3 as its activation function, this neuron 300 will output 1.

[0022] As a second example of similarity determination, as shown in FIG. 38D, an input vector x2 is x2=(1,0,0,1,1,0). TIn this case, the dot product similarity is 2, and the input vector x l This indicates that there is one less input with a value of 1. If neuron 300 in FIG. 43D has the same activation function as when the above input vector x2 is input, this dot product similarity does not reach the threshold value of 3, and so outputs 0.

[0023] As a third example of similarity determination, as shown in FIG. 38E, an input vector x3 is x3=(1,0,0,1,0,0). T In this case, the dot product similarity is also 2, and the input vector x l This means that there is one less input whose value is 1. In this case, too, 0 will be output, just like in Figure 43D.

[0024] Now, looking at the difference between input vectors x2 and x3, in x2 there is one input where the learning input is 0 and the similarity judgment input is 1, and one input where the learning input is 1 and the similarity judgment input is 0. In other words, there are two inputs where a difference occurs. In contrast, in x3 there is only one input where the learning input is 1 and the similarity judgment input is 0. In other words, there is only one input where a difference occurs. Therefore, in reality, x3 is closer to x l However, the dot product similarity is the same value.

[0025] As a fourth example of similarity determination, as shown in FIG. 38F, an input vector x4 is x4=(1,1,1,1,0,1). T In this case, the dot product similarity is 3, and the input vector x l is the same as the first similarity judgment example that is input as is. However, x1 is x l In contrast to the above, in the case of x4, even though there are two inputs where the learning input is 0 and the similarity judgment input is 1, the result is the same as in the case of x1. [Prior art documents] [Non-patent literature]

[0026] [Non-Patent Document 1] BL McNaughton, RGM Morris, "Hippocampal synaptic enhancement and information storage within a distributed memory system", Trends in Neuroscience, volume 10, Issue 10, pp. 408-415, 1987. [Non-patent document 2] Thomas Trappenberg, “Fundamentals of Computational Neuroscience”, Oxford University Press, 2010. [Non-patent document 3] Edmund T. Roll, “Cerebral Cortex: Principles of Operation,” Oxford University Press, 2016. [Non-patent document 4] Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, AJ Hudspeth, "PRINCIPLES OF NEURAL SCIENCE: Fifth Edition," McGraw-Hill Education, 2012. [Non-Patent Document 5] David J. Heeger, "Normalization of cell responses in cat striate cortex," Visual Neuroscience, vol. 9, pp. 181-197, 1992. [Non-patent document 6] T. Tanimoto, "An elementary mathematical theory of classification and prediction.", Technical report, International Business Machines Corporation, New York, 1958. [Non-Patent Document 7] P. Jaccard, “The distribution of the flora in the alpine zone”, Phytologist, 1912;11(2):37-50. https: / / doi.org / 10.1111 / j.1469-8137.1912.tb05611.x. [Non-patent document 8] GA Carpenter, S. Grossberg, N. Markuzon, JH Reynolds, DB Rosen, "Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analog Multidimensional Maps," IEEE Transactions of Neural Networks, Vol. 3, No. 5, pp. 698-713, 1992. https: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=&arnumber=159059 [Non-Patent Document 9] L. Zadeh, "Fuzzy sets", Information and Control, Vol. 8, No. 3, pp. 338-353, 1965. Summary of the Invention [Problem to be solved by the invention]

[0027] In an associative network, the input to the neural network is a vector (input vector), and similarity is determined by calculating the dot product of the input vector used during training and the input vector used to determine similarity. In reality, the dot product similarity may be the same for two input vectors whose similarity is being determined, even if there is a difference in the distance from the input vector used during training. For example, in the third similarity judgment example shown in FIG. 38E, x3 is actually closer to x l Although the result is close to x1, the dot product similarity may end up being the same value. Also, as in the fourth similarity determination example shown in Figure 38F, in the case of x4, even though there are two inputs where the learning input is 0 and the similarity determination input is 1, the result may be the same as in the case of x1. As described above, in the similarity calculation in the prior art, there was a problem in that the dot product similarity may not be able to accurately determine the difference between the input vector at the time of learning and the input vector at the time of similarity determination.

[0028] The present invention has been made in view of the above circumstances, and an object of the present invention is to realize a circuit device that, when determining inner product similarity, accurately determines the difference between an input vector during learning and an input vector during similarity determination. [Means for solving the problem]

[0029] In order to solve the above-mentioned problems, a neural network circuit device is provided that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron that models a nerve cell, and is characterized by comprising: a logical product operation circuit that operates the logical product of a vector in the learning phase and a vector in the inference phase; a first counter that counts the number of inputs having a value of 1 among the vectors input during the inference phase; a second counter that counts the number of inputs having a value of 1 among the logical product vectors obtained by the logical product operation in the logical product operation circuit; a third counter that counts the number of inputs having a value of 1 among the vectors during the learning phase; an adder circuit that adds the output of the first counter and the output of the third counter; a shift register that shifts the result of the second counter by one bit to the higher order; and a divider circuit that divides the output vector of the shift register by the output vector of the adder circuit. [Effects of the Invention]

[0030] According to the present invention, it is possible to realize a circuit device that, when determining the dot product similarity, accurately determines the difference between the input vector at the time of learning and the input vector at the time of similarity determination. [Brief explanation of the drawings]

[0031] [Figure 1] 1 shows an example of a neural circuit that performs division normalization calculations in a division normalization-type similarity determination method according to a first embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a circuit for performing a division normalization type similarity determination method according to a first embodiment of the present invention. FIG. [Figure 3] FIG. 2 is a diagram illustrating the setting of synapse weights in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a similarity determination phase in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 5]FIG. 1 is a diagram showing a neural network circuit device when the activation function is a "step function" in the division normalization type similarity calculation method according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram for explaining a method for implementing a Bitwise-AND in a Bitwise-AND circuit of a neural network circuit device according to a first embodiment of the present invention. [Figure 7] 1A to 1C are diagrams illustrating a method for realizing a T counter of a neural network circuit device according to a first embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing a neural network circuit device when the activation function is a "linear function" in the division normalization type similarity calculation method according to the second embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of a memory configuration for storing the reciprocal of the divisor in a method for storing the reciprocal of the denominator in equation (6) of a neural network circuit device according to a third embodiment of the present invention in memory. [Figure 10] FIG. 10 is a circuit diagram showing an example of the configuration of a division circuit that uses a method of storing the reciprocal of the denominator of equation (6) in a memory in a neural network circuit device according to a third embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing the division normalization type similarity calculation method of the neural network circuit device according to the fourth embodiment of the present invention, and the activity (N=100) of the perceptron that outputs the diffusion information network when only a diffusion type learning network is used. [Figure 12] FIG. 10 is a diagram showing the division normalization type similarity calculation method of the neural network circuit device according to the fourth embodiment of the present invention, and the activity (N=1000) of the perceptron that outputs the diffusion information network when only a diffusion type learning network is used. [Figure 13]FIG. 10 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method are used in a neural network circuit device according to a fourth embodiment of the present invention (changes in output when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). [Figure 14] FIG. 10 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method are used in a neural network circuit device according to a fourth embodiment of the present invention (changes in output when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed). [Figure 15] FIG. 10 is a graph comparing the activity (change in output when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed) of the perceptron that outputs the diffusion information network when the division normalization type similarity calculation method, the diffusion type learning network, and the noise addition type sensitivity characteristic improvement method of the neural network circuit device according to the fourth embodiment of the present invention are used, with the raised Tanimoto similarity. [Figure 16] FIG. 10 is a diagram comparing the activity (change in output when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed) of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method are used in a neural network circuit device according to a fourth embodiment of the present invention, with the raised Tanimoto similarity. [Figure 17] FIG. 10 is a diagram showing the output of a perceptron when a sigmoid function is used as the activation function in division normalization type similarity calculation of a neural network circuit device according to a fourth embodiment of the present invention (when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). [Figure 18]FIG. 10 is a diagram showing the output of a perceptron when a sigmoid function is used as the activation function in division normalization type similarity calculation of a neural network circuit device according to a fourth embodiment of the present invention (when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed). [Figure 19] FIG. 10 is a diagram showing the expected value of the output of a perceptron when using a noise-addition type sensitivity characteristic improvement method for a neural network circuit device according to a fourth embodiment of the present invention (when the number of inputs that result in an input value of 1 during learning and 0 during similarity determination is changed). [Figure 20] FIG. 10 is a diagram showing the expected value of the output of a perceptron when using a noise-addition type sensitivity characteristic improvement method for a neural network circuit device according to a fourth embodiment of the present invention (when the number of inputs that result in an input value of 0 during learning and 1 during similarity determination is changed). [Figure 21] FIG. 10 is a diagram showing a neural network circuit device in a fourth embodiment of the present invention in which the activation function is a step function for which an arbitrary threshold can be set, and in which a division normalization-type similarity calculation method and a noise-addition-type sensitivity characteristic improvement method are combined. [Figure 22] 10 is a parallel circuit in which a plurality of neural network circuit devices are connected, which combine a division normalization type similarity calculation and a noise addition type sensitivity characteristic improvement method of a neural network circuit device according to a fourth embodiment of the present invention. [Figure 23] FIG. 10 is a diagram showing a neural network circuit device in a fourth embodiment of the present invention in which the activation function is a linear function for which an arbitrary threshold can be set, and in which a division normalization-type similarity calculation method and a noise-addition-type sensitivity characteristic improvement method are combined. [Figure 24] 10 is a parallel circuit in which a plurality of neural network circuit devices are connected, which combine a division normalization type similarity calculation and a noise addition type sensitivity characteristic improvement method of a neural network circuit device according to a fourth embodiment of the present invention. [Figure 25]FIG. 10 is a diagram showing a comparison between the expected value of the perceptron output (the change in output when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed) and the raised Tanimoto similarity when the division normalization type similarity calculation method and the noise addition type sensitivity characteristic improvement method of the neural network circuit device according to the fourth embodiment of the present invention are used. [Figure 26] FIG. 10 is a diagram showing a comparison between the expected value of the perceptron output (the change in output when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed) and the raised Tanimoto similarity when the division normalization type similarity calculation method and the noise addition type sensitivity characteristic improvement method of the neural network circuit device according to the fourth embodiment of the present invention are used. [Figure 27] FIG. 11 is a diagram illustrating an example of similarity obtained by a division normalization type similarity calculation method using fuzzy logic in a neural network circuit device according to a fifth embodiment of the present invention. [Figure 28] FIG. 10 is a diagram showing a neural network circuit device that realizes division normalization-type similarity calculation using fuzzy logic when the activation function according to a fifth embodiment of the present invention is a step function for which an arbitrary threshold can be set. [Figure 29] FIG. 10 is a diagram showing a neural network circuit device that realizes division normalization-type similarity calculation using fuzzy logic when the activation function according to a fifth embodiment of the present invention is a linear function for which an arbitrary threshold can be set. [Figure 30] FIG. 11 is a diagram showing a neural network circuit device in which a division normalization type similarity calculation using fuzzy logic and a noise-addition type sensitivity characteristic improvement method are combined, in a case where the activation function according to a fifth embodiment of the present invention is a step function for which an arbitrary threshold can be set. [Figure 31] FIG. 11 is a diagram showing a neural network circuit device in which a division normalization type similarity calculation using fuzzy logic and a noise-addition type sensitivity characteristic improvement method are combined, in a case where the activation function according to a fifth embodiment of the present invention is a step function for which an arbitrary threshold can be set. [Figure 32]FIG. 1 illustrates the operation of a perceptron with variable constant inputs. [Figure 33] This figure shows the operation of a perceptron with a generalized representation of inputs and synaptic weights. [Figure 34] Figure 1 shows a multi-layered artificial neural network. [Figure 35] FIG. 1 is a diagram illustrating an example of a simple associative network. [Figure 36] FIG. 1 is a diagram showing an example of an associative network including multiple unconditioned stimuli. [Figure 37] FIG. 1 is a diagram illustrating neurons that are components of a technology for determining similarity using an associative network. [Figure 38A] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 38B] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 38C] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 38D] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 38E] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 38F] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0032] A neural network circuit device and the like in an embodiment for carrying out the present invention (hereinafter referred to as "the present embodiment") will be described below with reference to the drawings. (First embodiment) The present invention is realized by combining the division normalization type similarity determination method and the diffusion type learning network method. [Division normalization type similarity judgment method] First, the division normalization type similarity determination method (similarity determination method) will be described. In the similarity determination using the Associative Network, which was explained as an existing technology, similarity is calculated by taking the dot product of the input vector during learning and the input vector during similarity determination. Therefore, each neuron has the ability to calculate the product (i.e., multiplication as an operation) of the input value and the synaptic weight value for each input, and to add the product values ​​for all inputs. Generally speaking, if the input value can take any real value, and the input value and synaptic weight value can also be negative, it actually has the ability to perform multiplication, addition, and subtraction.

[0033] In contrast, the division normalization similarity determination method incorporates into the perceptron model not only multiplication, addition, and subtraction, but also operations caused by a phenomenon called the shunt effect of nerve cells (neurons) (Non-Patent Document 4). The shunt effect occurs in nerve cells due to inhibitory synapses formed near the cell body. The shunt effect is an effect in which the entire summed signal transmitted to a neuron is divided by the signal transmitted via an inhibitory synapse formed near the cell body. This division caused by the shunt effect is also used in a model called division normalization that explains the adjustment of visual sensitivity, as described in Non-Patent Document 5.

[0034] Figure 1 shows an example of a division normalization-type similarity calculation unit for division normalization, representing an example of a neural circuit that performs division normalization calculations. In Figure 1, neurons 001, 002, and 003, which are represented by black triangles, form excitatory synapses with neurons 005, 006, and 007, respectively. Neuron 004, which is represented by a white triangle (△), forms inhibitory synapses 008, 009, and 010. Here, an excitatory synapse is a synapse that has the effect of shifting the activation state of the neuron receiving the synapse toward firing. Conversely, an inhibitory synapse is a synapse that has the effect of shifting the activation state toward quiescence. In Figure 1, the inhibitory synapses 008, 009, and 010 formed by neuron 004 are connected to the black triangles, which indicates that the inhibitory synapses 008, 009, and 010 exhibit a shunt effect.

[0035] Neurons 001, 002, and 003 in Figure 1 receive inputs 1 and 2, 3 and 4, and 5 and 6, respectively, and input values ​​x1 and x2, x3 and x4, and x5 and x6, respectively. Assume that these inputs cause the output values ​​of neurons 001, 002, and 003 to become e1, e2, and e3, respectively. The output values ​​e1, e2, and e3 are sent to neurons 005, 006, and 007, respectively. Assume that these output values ​​are transmitted as is to neurons 005, 006, and 007, and become their respective activity values. Neuron 004 also receives e1, e2, and e3 as is, and sets its activity value to Σ 3 j=1 e j Then, the activity of neuron 004 is output as is and sent to neurons 005, 006, and 007, causing a shunt effect at synapses 008, 009, and 010. In this case, the effect of division normalization is expressed by the following equation, and neurons 005, 006, and 007 have activity expressed by this equation (3). Here, k is 1, 2, or 3.

[0036]

number

[0037] In this case, the activity of neurons 005, 006, and 007 is the value when the numerators are e1, e2, and e3, respectively, in equation (3). In this way, in division normalization, the activity of a neuron is divided by the sum of the outputs of multiple neurons called the neuron pool (neurons 001, 002, and 003 in the example in Figure 1). This effect explains the adjustment of visual sensitivity. The division normalization model does not take into account changes in synaptic weights due to learning. Furthermore, the value of C is determined experimentally to prevent saturation of the current visual input, so there is no clear method for determining it based on the input during learning, etc.

[0038] The division normalization similarity determination method of the present invention is realized by (A) a method for determining synaptic weights, (B) a method for determining the division normalization constant C, and (C) a method for determining a perceptron set (hereinafter referred to as a perceptron pool) that corresponds to a neuron pool in division normalization, which are described below.

[0039] 2 is a diagram showing an example of a division normalization type similarity calculation unit (similarity calculation unit) that performs the division normalization type similarity determination method, and illustrates the learning phase in the example of the division normalization type similarity determination method. Hereinafter, the module that executes the processing of the division normalization type similarity determination method will be referred to as the division normalization type similarity calculation unit 100 (similarity calculation unit). The input values ​​x1, x2, x3, x4, x5, and x6 to inputs 1, 2, 3, 4, 5, and 6 shown in FIG. 2 represent the input values ​​to the division normalization type similarity calculation unit 100. These are input equally to perceptrons 001 and 002. In this way, in the division normalization type similarity determination method, only the inputs to the division normalization type similarity calculation unit are used as the perceptron pool in division normalization (C). Each input takes two types of values ​​when the previous perceptron is in a resting state and when it is in a firing state, and in this specification these are represented by 0 and 1, respectively. That is, x i ∈{0,1}(i=1,2,3,4,5,6).

[0040] Fig. 3 shows the setting of synaptic weights in the division normalization type similarity determination method. Fig. 3 shows that as a result of the learning phase in Fig. 2, the synaptic weights formed in perceptron 001 by input values ​​x1, x2, x3, x4, x5, and x6 are w1, w2, w3, w4, w5, and w6.

[0041] In the division normalization type similarity judgment method (A), the synaptic weight determination method is i =x iIn other words, the weight of a synapse that receives an input signal corresponding to the firing state during the learning phase is 1, and the weight of a synapse that receives an input signal corresponding to the quiet state is 0.

[0042] Figure 4 shows the similarity determination phase in the division normalization type similarity determination method. Figure 4 shows the similarity determination phase when input values ​​y1, y2, y3, y4, y5, and y6 arrive. At this time, the input to the perceptron 001 is Σ 6 j=1 y j ·w j On the other hand, there is no change in the synaptic weights of Perceptron 002, and Σ 6 j=1 y j The output of the perceptron 002 generates a shunt effect on the perceptron 001 through a synapse 003 formed between the perceptron 002 and the perceptron 001, and calculates the following operation.

[0043]

number

[0044] Furthermore, as a method for determining the constant C for division normalization (B), the constant C is set to a value calculated as follows in the learning phase:

[0045]

number

[0046] where x=(x1,x2,x3,x4,x5,x6) T where ||x|| represents the norm of vector x. When equation (5) is substituted into equation (4), equation (4) is transformed into the following equation (6).

[0047]

number

[0048] However, y=(y1,y2,y3,y4,y5,y6) T and w=(w1,w2,w3,w4,w5,w6) T is. Equation (6) includes the square of the norm and the dot product of two vectors as vector operations. In general, a vector v = (v1, v2, ..., v N ) T , and vector u=(u1,u2,…,u N ) T When there was ||u|| 2 =u1 2 +u2 2 +…+u N 2 and u·v=u1v1+u2v2+…+u N v N is.

[0049] Now, u i ∈{0,1} and v i ∈{0,1}, then ||u|| 2 =u1 2 +u2 2 +…+u N 2 =u1+u2+…+u N and u v =u1v1+u2v2+…+u N v N =Σ N i=1 u i v i =Σ N i=1 (u i ANDv i ) can also be calculated as u i ANDv i u i and v i represents the logical AND operation of

[0050] where n 11 , n 10 , n 01 , and n 00 respectively, x i =1 and y i= 1, x i =1 and y i = 0, x i =0 and y i = 1, and the number of inputs x i =0 and y i = 0. Also, N = n 11 +n 10 +n 01 +n 00 is assumed to be constant since it represents the total number of inputs. The above equation (6) can be transformed as follows:

[0051]

number

[0052] In the calculation of formula (7), if the denominator is 0, n 11 , n 10 , n 01 Since all of these are 0, the numerator is also n 11 , the value is also 0. In this case, the calculation result of equation (7) is calculated as 0 because there is no similarity between the two vectors. Now, when the same input is used in the learning phase and the similarity judgment phase, n 10 =n 01 =0, so we get equation (8).

[0053]

number

[0054] Next, consider the case where the input differs between the learning phase and the similarity judgment phase. f =n 11 +n 10 is the number of inputs of 1 during learning, and remains constant in the similarity judgment phase after the learning phase. f Using this, equation (7) can be transformed as follows:

[0055]

number

[0056] From this equation (9), the value calculated by equation (9) is n 10 and n 01 From this, we can see that it changes only depending on n 10 and n 01 We explain how the value of equation (9) changes with the change of

[0057] <n 10 Changes in First, n 10 Consider the change in equation (9) with respect to the change in . Equation (9) is transformed into the following equation (10).

[0058]

number

[0059] In equation (10), n 01 If we keep constant, n 10 It can be seen that the value of the above equation decreases monotonically as increases.

[0060] <n 01 Changes in Second, n 01 Consider the change in equation (9) with respect to the change in n 10 If we keep constant, n 01 It can be seen that the value of equation (9) decreases monotonically with increasing From the above, equation (7) is 10 =n 01 =0 gives the value 1, and n 10 and n 01 It monotonically decreases with the increase of n, which represents the degree of similarity. 10 and n 01 It can be seen that this solves the problem that the degree of similarity does not change even if the value of

[0061] <The exact meaning of the value calculated by the division normalization similarity calculation method> Next, the exact meaning of the value calculated by the division normalization type similarity calculation method will be explained. Two equations, S d , and ,S c Think about it.

[0062]

number

[0063] Equation (11) is 11 +n 10 This is the equation that becomes the division normalization type similarity calculation method of the present invention when

[0064]

number

[0065] Equation (12) is the case where c2 is n 11 +n 10 When x is a vector, it represents the cosine similarity between vectors x and y. Cosine similarity represents the degree of similarity between two vectors. Specifically, it is the cosine value of the angle between two vectors in vector space. This value is calculated by dividing the dot product of two vectors (an operation in which the products of corresponding components of two vectors are added for all components) by the product of the magnitudes (norms) of the two vectors.

[0066] First, let u and v be 11 , and n 01 Substituting these into the above formulas (11) and (12), we get S d , and ,S c is expressed as a function of u and v as follows:

[0067]

number

[0068]

number

[0069] Now, in general, as a Taylor expansion of a function f(u,v) around (u,v), if we consider up to the first-order terms, we get the Taylor series f (1) (u+h,v+k) is expressed as follows:

[0070]

number

[0071] Using this, S d (u,v), and S c Taylor series S of (u,v) around (u,v) up to first order d (1) (u+h,v+k), and S c (1) Calculating (u+h,v+k) gives us the following:

[0072]

number

[0073]

number

[0074] In the above equations (16) and (17), c1 = c2 = n 11 +n 10 =N f , u=N f , and substituting v=0, we get the following.

[0075]

number

[0076]

number

[0077] Therefore, c1=c2=n 11 +n 10 =N f , u=N f , and when v=0, the following equality holds:

[0078]

number

[0079] From the above, it can be seen that the value calculated by the division normalization type similarity determination method of the present invention is an approximation of the cosine similarity, and therefore the similarity calculated by the division normalization type similarity determination method can calculate the recognized similarity more accurately than the existing technology.

[0080] [Implementation method] Below, we will explain how to implement the division normalization type similarity calculation method using a neural network circuit device. FIG. 5 is a diagram showing a neural network circuit device in which the activation function is a "step function" for which an arbitrary threshold can be set in the division normalization type similarity calculation method. As shown in FIG. 5, the neural network circuit device 500 includes a demultiplexer (DEMUX) 501, registers 502 and 510, a bitwise-AND circuit 504 (logical AND operation circuit), a T counter 503 (first counter), a T counter 505 (second counter), a T counter 506 (third counter), an adder circuit 507, a shift register 508, a divider circuit 509, and a comparator circuit 511.

[0081] The demultiplexer (DEMUX) 501 demultiplexes the input vector x=(x1, x2, ..., x N ) T (feature) and converts the input vector x signal into outputs A1 to A2 specified by the phase switching signal S. N , and B1~BN The phase switching signal S is a signal for switching between the learning phase and the similarity determination phase (inference phase). The demultiplexer (DEMUX) 501 outputs the input vector x signal to one of B1 to B2 in the learning phase. M In the similarity determination phase (inference phase), A1 to A M Output to the side. The demultiplexer (DEMUX) 501 receives an input vector x in the learning phase, and outputs the input signal to either a first output or a second output specified by a phase switching signal.

[0082] Registers 502 and 510 are circuits that temporarily hold input signals and output them at a predetermined timing.

[0083] The Bitwise-AND circuit 504 receives two input vectors A1 to A M and B1~B M A logical product (AND) operation is performed for each corresponding bit of OUT1 to OUT M (See Figure 6 below.) The Bitwise-AND circuit 504 performs part of the calculation of the vector dot product by calculating the logical product (AND) of the stored learning phase vector and inference phase vector in 1-bit units. It is also possible to calculate the vector dot product by combining the Bitwise-AND circuit 504 with calculations by the T counter. The Bitwise-AND circuit 504 is a logical product calculation circuit that calculates the logical product of the learning phase vector and the inference phase vector.

[0084] T counters 503, 505, and 506 are connected to inputs IN1 to IN N Calculate the number of inputs that are 1 among the values ​​of the logical variables input to OUT1~OUT M The T counter 503 counts the number of inputs with a value of 1 among the input vector signals that come into the inference phase. The T counter 506 counts the number of input vector signals during learning that have a value of 1. The T counter 505 counts the number of 1s among the logical product operations (AND) performed by the Bitwise-AND circuit 504.

[0085] The adder circuit 507 adds the output of the T counter 503 and the output of the T counter 506. The denominator of equation (6), ||w|| 2 +||y|| 2 Calculate and output.

[0086] When the result of the T counter 505 is an integer value expressed in binary, the shift register 508 shifts the input vector signal by one bit toward the MSB (Most Significant Bit) side, thereby outputting a value twice the value calculated by the T counter 505. Here, the MSB side refers to the higher side, which is the left side when expressed in binary. The shift register 508 outputs a value twice the value calculated by the second counter 505 as the numerator of equation (6).

[0087] The division circuit 509 receives the input values ​​2(w·y) and ||w|| from the shift register 508 and the addition circuit 507, respectively. 2 +||y|| 2 , respectively, and 2(w·y) as ||w|| 2 +||y|| 2 Perform division by dividing by. A register 510 is a circuit that temporarily holds and outputs an input signal that is a threshold value.

[0088] The comparison circuit 511 compares the division result (calculated similarity) of the division circuit 509 with the value (threshold) stored in the register 510, and outputs 1 if the similarity is greater than the threshold, and outputs 0 otherwise.

[0089] [Operation] The operation of the neural network circuit device 500 configured as described above will now be described. <Learning Phase> First, the input vector x=(x1,x2,…,x N ) T The demultiplexer 501 receives the input signal and outputs A1 to A N , and B1~B N The output to either of these is specified by a phase switching signal S input to the demultiplexer 501. The phase switching signal S is a signal that distinguishes between the learning phase and the similarity determination phase. When this signal has a value indicating the learning phase, the input vector x is output to one of the outputs B1 to B2. N At this time, the register 502 stores the value of the input vector x, and N Output from

[0090] In this embodiment, since the synapse weight is determined as w=x, the x stored in the register 502 is expressed as w=(w1, w2, . . . , w N ) T The output of the register 502 is transmitted to a bitwise-AND circuit 504 (logical AND operation element) and a T counter 506. The bitwise-AND circuit 504 performs a logical AND operation (AND) on the two inputs for each corresponding bit. T counters 503, 505, and 506 are connected to inputs IN1 to IN N Calculate the number of inputs with a value of 1 among the input vector signals of the logical variables input to OUT1 to OUT M Output from Of the Bitwise-AND circuit 504 and the T counters 503, 505, and 506, an example of the Bitwise-AND circuit 504 will be described with reference to FIG.

[0091] FIG. 6 is a diagram for explaining how the Bitwise-AND circuit 504 implements the Bitwise-AND. The Bitwise-AND circuit 504 includes AND circuits 521 to 528 that perform a logical AND operation on two sets of inputs, A1, A2, A3, A4, A5, A6, A7, and A8, and B1, B2, B3, B4, B5, B6, B7, and B8, and output one set of outputs, OUT1, OUT2, OUT3, OUT4, OUT5, OUT6, OUT7, and OUT8.

[0092] The AND circuits 521 to 528 calculate one set of outputs, OUT1, OUT2, OUT3, OUT4, OUT5, OUT6, OUT7, and OUT8, from two sets of inputs, A1, A2, A3, A4, A5, A6, A7, and A8, and B1, B2, B3, B4, B5, B6, B7, and B8. i The value of the logical variable A i and B i where i is an integer between 1 and 8.

[0093] Returning to Figure 5, T counters 503, 505, and 506 can be realized by a look-up table (LUT). A look-up table is a circuit that has a table that outputs any combination of logic variables for a combination of logic variables. This circuit can be realized by memory.

[0094] Figure 7 is a diagram for explaining how to implement the T counters 503, 505, and 506. Figure 7 is an example showing which values ​​are stored in which addresses when creating a lookup table using memory. The memory receives an address represented by a combination of multiple logical variables as an input, stores data represented by an arbitrary combination of logical variables, and outputs the data stored at the specified address when reading. In the example of Figure 7, the addresses are A0 to A 15 Each address corresponds to the storage location of one byte of data. When reading data from this memory, the eight bytes (D0 to D 63 For example, in Figure 7, A 15 A14 A 13 A 12 A 11 A 10 A9A8A7A6A5A4A3A2A1A0 stores the value 1 at the address 0000000000001000. This indicates that data representing 1 is stored in the 8 bytes (64 bits) from address 0000000000001000 to 0000000000001111.

[0095] For this memory, connect a signal that always inputs 0 to A2A1A0, and input X from the outside 12 X 11 X 10 X9X8X7X6X5X4X3X2X1X0 to A of memory 15 A 14 A 13 A 12 A 11 A 10 When connected to A9A8A7A6A5A4A3, any 8-byte data can be output for any combination of external input logic variables. 12 X 11 X 10 Considering X9X8X7X6X5X4X3X2X1X0 as a binary number, if the value when expressed in decimal is 0, 1, 2, 3, 4, 5, 6, 7, 8, or 9, then X 12 X 11 X 10 The number of 1s contained in the bit string X9X8X7X6X5X4X3X2X1X0 is D0 to D 63 In FIG. 7, D0 to D 63 Regarding what value to put into the 64 bits of 63 When is the MSB (Most Significant Bit), D0 to D 63 The decimal values ​​of the binary numbers are shown in Figure 7.

[0096] Returning to FIG. 5, based on the above operation, the T counter 506 calculates ||w|| included in the above equation (6) for the synaptic weight w. 2 The following is calculated:

[0097] <Similarity determination phase> Next, in the similarity determination phase, when the input vector y is input to the demultiplexer 501, the input vector y is divided into A1 to A2 based on the phase switching signal S. N This output is sent to the T counter 503 and the Bitwise-AND circuit 504. The T counter 503 operates in the same manner as the T counter 506 from the input y to obtain ||y|| included in the above equation (6). 2 Calculate.

[0098] The Bitwise-AND circuit 504 receives the synaptic weight w and the input vector y of the similarity determination phase, and the vectors (w1y1, w2y2, ..., w N y N ) T This result is input to the T counter 505. i y i is 0 or 1, the result of the T counter 505 is w1y1+w2y2+...+w included in the above equation (6). N y N The result of the calculation of T counter 505 is further sent to shift register 508. If the result of T counter 505 is an integer value expressed in binary, shift register 508 can shift the input vector signal by one bit toward the MSB to obtain a value twice the value calculated by T counter 505. This value becomes the numerator value 2(w·y) of equation (6) above.

[0099] The outputs of the T counter 504 and the T counter 506 are ||y||, respectively. 2 , and ||w|| 2 and is sent to the adder circuit 507. The adder circuit 507 calculates the denominator ||w|| 2 +||y|| 2 Calculate and output. The division circuit 509 receives the input value 2(w·y) and ||w|| from the shift register 508 and the addition circuit 507, respectively. 2 +||y|| 2 respectively. Then, it performs a division operation of dividing 2(w·y) by ||w|| 2 +||y|| 2 Through the above processing, the division circuit 509 calculates the similarity and outputs the result.

[0100] The register 510 has the threshold value of the activation function input in advance and stores that value. As a result, the calculated similarity and the threshold value are input to the comparison circuit 511 as IN-A1~IN-A M and IN-B1~IN-B M respectively and compared. As a comparison result, when the value input to IN-A1~IN-A M is greater than the value input to IN-B1~IN-B M , the output A>B becomes 1; when they are equal, the output A=B becomes 1; when it is smaller, the output A<B becomes 1. Thus, the comparison circuit 511 outputs 1 when the similarity is greater than the threshold value, and outputs 0 otherwise. That is, the comparison circuit 511 compares the numerical value represented by a plurality of bits of the division circuit 509 (representing "similarity") with the numerical value represented by a plurality of bits (representing "threshold value").

[0101] (Second Embodiment) FIG. 8 is a diagram showing a neural network circuit device 600 when the activation function is a "linear function" in which an arbitrary threshold value can be set in the division normalization type similarity calculation method according to the second embodiment of the present invention. The same components as those in FIG. 5 are denoted by the same reference numerals, and the description of overlapping parts is omitted. As shown in FIG. 8, the neural network circuit device 600 includes a demultiplexer (DEMUX) 501, registers 502 and 510, a bitwise-AND circuit 504, T counters 503, 505, and 506, an adder circuit 507, a shift register 508, a divider circuit 509, a comparator circuit 511, a register 601, a multiplexer (MUX) 602, and a subtractor circuit 603. A register 601 stores an output value when the similarity is less than the threshold value.

[0102] The subtraction circuit 603 subtracts the threshold value of the activation function stored in the register 510 from the calculation result of the division circuit 509, and outputs the difference indicating how much the calculation result of the division circuit 509 exceeds the threshold value to the multiplexer 602.

[0103] Based on the A / B switching signal (1 if the similarity is greater than the threshold, 0 otherwise) from the comparison circuit 511, the multiplexer (MUX) 602 outputs the calculation result of the subtraction circuit 603 if the A / B switching signal is 1 and the similarity is greater than the threshold, and outputs the output value stored in the register 601 if the A / B switching signal is 0.

[0104] [Operation] The operation of the neural network circuit device 500 configured as described above will now be described. 8, an input is input to a demultiplexer 501. The operation from this input to the output of a division circuit 509 is the same as the operation from the demultiplexer 501 to the output of the division circuit 509 in FIG.

[0105] The output of the division circuit 509 represents the similarity. The output of the division circuit 509 is sent to the subtraction circuit 603 and the comparison circuit 511. In addition to the subtraction circuit 603, the comparison circuit 511 also receives input from a register 510. Similar to the register 510 in FIG. 5, the register 510 stores the threshold value of the activation function. The threshold value is input in advance, similar to the case in FIG. 5, and the stored threshold value is output.

[0106] The inputs from the division circuit 509 and the register 510 are received at IN-A1 to IN-A M and IN-B1 to IN-B M of the comparison circuit 511, respectively. The operation of the comparison circuit 511 is the same as that of the comparison circuit 511 in FIG. 5, and compares the output vector signal of the division circuit 509 with the threshold value of the activation function (stored in the register 510). Then, as a comparison result, when the value input to IN-A1 to IN-A M is greater than the value input to IN-B1 to IN-B M , output A>B becomes 1; when they are equal, output A=B becomes 1; and when it is smaller, output A<B becomes 1. Accordingly, the output of the comparison circuit 511 becomes 1 when the output of the division circuit 509 is greater than or equal to the threshold value stored in the register 510, and becomes 0 otherwise.

[0107] The output of the comparison circuit 511 is connected to the multiplexer 602, and outputs either one of the two sets of inputs A1 to A M and B1 to B M as OUT1 to OUT M . Which of the two sets is output is switched by the value of the output of the comparison circuit 511 input to the multiplexer. When this value is 1, A1 to A M are output to OUT1 to OUT M , and when this value is 0, B1 to B M are output to OUT1 to OUT M . The output of the subtraction circuit 603 is connected to the inputs A1 to A M of the multiplexer 602. The division circuit 509 calculates the similarity. This value is sent to the inputs IN-A1 to IN-A M of the subtraction circuit 603. Also, the threshold value stored in the register 510 is input to the inputs IN-Bl to IN-B M of the subtraction circuit 603. As a result, the output of the subtraction circuit 603 becomes the value obtained by subtracting the threshold value from the similarity, and this value is sent to the multiplexer 602. The inputs B1 to B MThe value stored in register 601 is sent to the output terminal 602. The output value when the similarity is less than the threshold is stored in register 601 before the circuit is used. When a linear function is used as the activation function, the value 0 is stored in register 601.

[0108] (Third embodiment) In this embodiment, an example will be described in which the first and second embodiments are replaced with a division circuit that performs division at high speed. The divisor used in the division in this embodiment is ||w|| as described in the above equation (6). 2 +||y|| 2 The divisor ||w|| is 2 is to write any i-th component of w as w i Then, w i The value of w i = 0, or w i = 1, so ||w|| 2 = w1 + w2 + w3 + .... Also, ||y|| 2 For y, any i-th component of y is expressed as y i Then, y i = 0 or y i = 1, so ||y|| 2 = y1 + y2 + y3 + .... Therefore, it is the number of components that are 1 among the components of the synaptic weight vector w and the input vector y in the similarity determination phase. Therefore, if the number of inputs to the perceptron is N, then the range of the number of components that are 1 among the components of w and y is an integer between 0 and N. Therefore, ||w|| 2 +||y|| 2 The range of is an integer from 0 to 2N. This means that the number of different divisors is at most 2N+1, and this can be used to perform division quickly as follows:

[0109] Generally, when comparing division and multiplication, multiplication can perform calculations faster. Division can be achieved by calculating the reciprocal of the divisor and multiplying that value by the dividend, and this characteristic can be used to speed up the calculation. That is, first, the reciprocals of all divisor candidates are calculated in advance and the calculated reciprocals are stored in memory. Then, in the division normalization similarity calculation, division is performed using the reciprocal values ​​stored in memory and a multiplication circuit.

[0110] Figure 9 shows the denominator ||w|| in Eq. (6). 2 +||y|| 2 10 is a diagram illustrating an example of a memory configuration for storing the reciprocal of a divisor in a method for storing the reciprocal of a divisor in memory. As shown in Figure 10, A 15 , A 14 The reciprocal of each divisor is stored in 8-byte memory 701, which has a 16-bit address signal represented by A0, ...A1. Since the reciprocal of each divisor is represented by 8 bytes, the addresses where the reciprocal of each divisor is stored are 8 bytes apart. Therefore, the addresses required to actually identify each divisor are A1, A2, A3, A4, A5, A6, A7, A8, A9, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22, A23, A24, A25, A26, A27, A28, A29, A30, A31, A32, A33, A34, A35, A36, A37, A38, A39, A40, A41, A42, A43, A44, A45, A46, A47, A48, A49, A50, A51, A52, A53, A54, A55, A56, A57, A58, A59, A60, A61, A62, A63, A64, A65, A66, A67, A68, A69, A71, A72, A73, A74, A75, A76, A77, A78, A79, A80, A81, A82, A83, A84, A85, A85, A86, A87, A88, A89, A91, A92, A93, A94, A95, A96, A97, A98, A99, A99, A99, A99, A100, A99, A110, A99, A120 15 , A 14 , ...A3. In contrast, ||w|| 2 +||y|| 2 The integer representing the value is expressed as X 13 , X 12 , …X1 represents each bit, then X i A (i+2) By connecting to A, the divisor expressed as an integer can be input and the reciprocal of the divisor can be extracted as a data signal. 15 , A 14 , ...A0 stores 0 as data for all addresses where A0 is all 0. This is because ||w|| 2 +||y|| 2 = 0. In this case, the components of w and y are all 0, which does not affect the similarity, so it is assumed that such an input does not exist.

[0111] Figure 10 shows the denominator ||w|| in Eq. (6).2 +||y|| 2 7 is a circuit diagram showing an example of the configuration of a division circuit 509 using a method of storing the reciprocal of As shown in FIG. 10, the division circuit 509 includes a memory 701 and a multiplication circuit 702. As shown in FIG. 10, in the division circuit 509, the divisors are IN-D1, IN-D2, . . . IN-D M , and are input to A3, A4, ...A of the memory 701, respectively. M+2 The memory 701 stores the reciprocal of the divisor in D0, D1, ... D M-1 The outputs are input to the multiplication circuits 702, IN-B1, IN-B2, . . . IN-B M Dividends IN-N1, IN-N2, ... IN-N M are directly connected to the IN-A1, IN-A2, . . . IN-A of the multiplication circuit 702. M Based on these inputs, the product of the dividend and the reciprocal of the divisor is calculated by the multiplication circuit 702, and the results are output as OUT1, OUT2, . . . OUT M is output from

[0112] [Effects of the first to third embodiments] As described above, the neural network circuit device 500 calculates the degree of similarity between the input in the learning phase and the input in the inference phase using a perceptron modeled on a nerve cell, and includes a logical product operation circuit (Bitwise-AND circuit 504) that performs a logical product operation between the vector in the learning phase and the vector in the inference phase, a first counter (T counter 503) that counts the number of inputs having a value of 1 among the vectors input during the inference phase, a second counter (second counter 505) that counts the number of inputs having a value of 1 among the logical product vectors obtained by the logical product operation performed by the logical product operation circuit, a third counter (third counter 506) that counts the number of inputs having a value of 1 among the vectors during the learning phase, an adder circuit 507 that adds the output of the first counter and the output of the third counter, a shift register 508 that shifts the result of the second counter one bit toward the most significant side, and a divider circuit 509 that divides the output vector of the shift register 508 by the output vector of the adder circuit 507.

[0113] For example, a neural network circuit device that calculates the degree of similarity between an input in the learning phase and an input in the similarity judgment phase using a perceptron modeled on a nerve cell receives one or more input values, each of which is input with either a value L or a value H, and calculates the value of a logical variable representing the i-th input of the N inputs in the learning phase as x i The value of the logical variable representing the i-th input among the N inputs in the similarity judgment phase is set as y i Let the weight value assigned to the i-th input in the similarity judgment phase be w i In this case, in the similarity judgment phase, a logic circuit is provided that implements equation (6), which incorporates the calculation caused by a phenomenon called the shunt effect of neurons into the perceptron model, and the logic circuit calculates the division normalization type similarity.

[0114] By doing so, the similarity calculated by the division normalization similarity calculation method can calculate the recognition similarity more accurately than existing technologies. As a result, the division normalization similarity calculation method can accurately measure the similarity between the information memorized in the learning phase and the information input to the similarity judgment phase. As a result, it is possible to realize a circuit device that accurately judges the similarity between the information memorized in the network and the information newly input to the network in an artificial neural network composed of perceptrons modeled on neurons.

[0115] In the neural network circuit device 500 (FIGS. 5 to 10) according to the first to third embodiments, the logic circuit includes a logical AND operation circuit (Bitwise-AND circuit 504) that calculates a vector inner product by performing a logical AND operation on a bit-by-bit basis between a vector in the learning phase and a vector in the inference phase, a first counter (T counter 503) that counts the number of 1s among the values ​​of the logical variables input during the inference phase, and a second counter (T counter 504) that counts the number of 1s in the vector inner product obtained by the logical AND operation performed by the logical AND operation circuit. a third counter (third counter 506) that counts the number of 1s in the vector during the learning phase; an adder circuit 507 that adds the output of the first counter 503 and the output of the third counter 506 to calculate the denominator of equation (6); a shift register 508 that shifts the result of the second counter 505 by one bit toward the MSB and outputs a value that is twice the value calculated by the second counter 505 as the numerator of equation (6); and a divider circuit 509 that receives input values ​​2(w·y) and ||w|| from the shift register 508 and the adder circuit 507, respectively. 2 +||y|| 2 , respectively, and input value 2(w·y) is expressed as ||w|| 2 +||y|| 2 and a division circuit 509 for performing division by dividing by .

[0116] In this way, a circuit device can be realized by a logic circuit that accurately determines the difference between the input vector at the time of learning and the input vector at the time of similarity determination when determining the dot product similarity.

[0117] In the neural network circuit device 500 (FIGS. 5 to 10) according to the first to third embodiments, the logic circuit includes a demultiplexer 501 that receives an input vector x in the learning phase and outputs the input signal to either a first output or a second output specified by a phase switching signal S.

[0118] In this way, the demultiplexer 501 can demultiplex the input vector x=(x1, x2, ..., x N ) T (feature amount), and outputs A1 to A2 as specified by the phase switching signal S. N , and B1~B N The output can be either

[0119] In the neural network circuit device 500 (FIGS. 5 to 10) according to the first to third embodiments, the division circuit 509 includes a storage unit (memory 701) that stores the reciprocal of the divisor, and a multiplication circuit 702 that multiplies the reciprocal of the divisor read from the storage unit.

[0120] In this way, a look-up table (LUT) is used instead of a logic gate as the multiplication circuit 702. The LUT is a basic component of a field programmable gate array (FPGA), which is an accelerator, and has high compatibility when synthesizing FPGAs, making it easy to implement using an FPGA. The accelerator may also be a graphics processing unit (GPU) / application specific integrated circuit (ASIC), etc.

[0121] The neural network circuit device 500 (FIGS. 5 to 10) according to the first to third embodiments further includes a comparison circuit 511 that compares the output vector of the division circuit 509 with a threshold vector.

[0122] By doing this, the comparison circuit 511 can compare the division result (calculated similarity) of the division circuit 509 with the value (threshold value) stored in the register 510, and output 1 if the similarity is greater than the threshold value, and output 0 otherwise.

[0123] The neural network circuit device 500 (FIGS. 5 to 10) according to the first to third embodiments includes a subtraction circuit 603 that subtracts a threshold value from the output vector of the division circuit 509, and a multiplexer 602 that switches between the output vector of the subtraction circuit 603 and a predetermined value according to the output of the comparison circuit 511 and outputs the selected value.

[0124] In this way, the value stored in register 601 is sent to the input of multiplexer 602. The register 601 can store the output value when the similarity is below the threshold before using the circuit, and can flexibly and adaptively handle cases such as when a linear function or other activation function is used as the activation function.

[0125] (Fourth embodiment) In the fourth embodiment, the division normalization similarity determination method and the diffusion learning network method of the first to third embodiments are further combined with a noise addition sensitivity characteristic improvement method.

[0126] The division normalization similarity determination method and the diffusion learning network method are the same as those in the first to third embodiments, and therefore their explanations will be omitted. First, a noise-addition type sensitivity characteristic improvement method will be described. Generally, the sensitivity of a measuring device is expressed as the ratio of the measuring device's indicated amount to the observed value. On the other hand, the division normalization type similarity assessment method and the diffusion type learning network method described in the first to third embodiments can be regarded as measuring devices for measuring the similarity between data in the learning phase and the similarity assessment phase. In order to explain the characteristics of these as a measuring instrument, FIGS. 11 and 12 are used.

[0127] Fig. 11 shows the activity (N=100) of a perceptron that outputs a diffusion information network when using only a division normalization similarity calculation method and a diffusion learning network. Fig. 12 shows the activity (N=1000) of a perceptron that outputs a diffusion information network when using only a division normalization similarity calculation method and a diffusion learning network. In Figures 11 and 12, the difference between the data in the learning phase and the similarity assessment phase increases as the horizontal axis moves to the right. The vertical axis represents the similarity calculated using the division normalization similarity assessment method and the diffusion-based learning network method. The activation function used in Figures 11 and 12 is a sigmoid function. The sigmoid function is expressed by the following equation (21). In this equation, β and τ are a parameter representing the slope and a threshold, respectively.

[0128]

number

[0129] The parameters included in equation (21) are p = 0.05 and β = 1.0 × 10 4 , τ=0.9. The values ​​of N are 100 and 1000 in Figures 11 and 12, respectively. As shown in dashed box a in Figure 11 and dashed box b and c in Figure 12, where the perceptron activity is close to 0.0 and where the perceptron activity is close to 1.0, the slope of the curve is almost zero and is nearly horizontal.

[0130] The fact that the slope of the curve shown in Fig. 11 is horizontal means that the calculated similarity does not change depending on the difference in data between the learning phase and the similarity determination phase, resulting in poor sensitivity. Thus, when only the division normalization type similarity determination method and the diffusion type learning network method of the first embodiment are used, a problem occurs in that there are some areas where the sensitivity for measuring similarity is poor (Note 1).

[0131] Furthermore, comparing Figures 11 and 12, we see that the curves differ depending on the value of N, which represents the square of the norm of the training data. For example, when the value on the horizontal axis is 0.3, the values ​​on the vertical axis are 0.302 and 0.0287 in Figures 11 and 12, respectively. Therefore, if various training data have different values ​​of N, different similarities will be output even if the difference in proportion to the training data is the same. This creates a problem in that it becomes difficult to compare the similarities between different training data with different values ​​of N (Note 2).

[0132] Furthermore, Equation (7) used in the division normalization similarity determination method of the first embodiment is an approximation of cosine similarity, which is mathematically defined, its characteristics are thoroughly analyzed, and its effectiveness has been demonstrated. However, after calculating the activity using Equation (7), the conversion is performed using an activation function, and then the result is processed using the diffusion-based learning network method of the first embodiment. This causes a problem in that the mathematically defined characteristics become unclear (Note 3).

[0133] The noise-addition type sensitivity characteristic improving method described below in the fourth embodiment is a technology that solves these points 1 to 3. The noise-addition type sensitivity characteristic improvement method calculates the similarity Sd expressed by equation (7) used in the division normalization type similarity determination method and the diffusion type learning network method of the first embodiment, and then calculates the similarity Sg by adding noise to Sd as shown in equation (22) below.

[0134]

number

[0135] Here, when the probability density function that generates the random variable X is expressed as P(X), G is the value of the random variable randomly generated according to this probability density function. This value is generated anew each time Sg is calculated. After Sg is calculated, Sg is used instead of Sd when performing the [Division Normalization Similarity Judgment Method] and [Diffusion Learning Network Method].

[0136] In this way, consider the expected value of the output of the division normalization similarity calculation unit when Sg is used instead of Sd. In a certain division normalization similarity calculation unit, the probability that random variable X occurs is P(X)dX. If the activity and activation function when no noise is added are S(n,d,l) and f(·), respectively, when S is used, the output of this division normalization similarity calculation unit is f(S(n,d,l)+X). In this, the value G of the random variable generated above is represented by X.

[0137] Now, if there are a sufficiently large number of division normalization type similarity calculation units, it can be considered that there are also a sufficient number of division normalization type similarity calculation units that have the same activity S(n,d,l). Therefore, the expected value of the output of a division normalization type similarity calculation unit whose activity is S(n,d,l) is given by equation (23).

[0138]

number

[0139] Furthermore, the probability that the activity is S(n,d,l) can be expressed as the expected value of the output of the division normalization type similarity calculation unit using the probability that the activity is S(n,d,l) as follows:

[0140]

number

[0141] The characteristics of the similarity actually calculated by the division normalization type similarity calculation unit using equation (24) will be described with reference to FIGS.

[0142] Fig. 13 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization similarity calculation method, a diffusion learning network, and a noise-addition sensitivity characteristic improvement method are used (the output changes when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). Fig. 14 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization similarity calculation method, a diffusion learning network, and a noise-addition sensitivity characteristic improvement method are used (the output changes when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed). In Figures 13 and 14, the vertical axis represents the activity of the perceptron that outputs the diffusion-type learning network, and the horizontal axis represents the rate at which the data in the similarity judgment phase differs from the data in the learning phase.

[0143] In Figures 13 and 14, a sigmoid function is used as the activation function, and the parameters included in equation (24) and the parameters included in equation (21) representing f(·) included in equation (24) are p = 0.05, β = 1.0 × 10 4 , τ=0.9. The values ​​of N are 25, 50, 100, and 1000. Furthermore, the probability density function P(X) in equation (24) is a Gaussian probability density function with a mean value and standard deviation of 0.01 and 0.5, respectively.

[0144] 13 and 14, the difference between the data in the learning phase and the similarity judgment phase increases as the horizontal axis moves to the right. The vertical axis represents the activity of the perceptron that outputs the diffusion-type learning network, calculated using equation (24).

[0145] As can be seen from Figures 13 and 14, the activity of the perceptron that outputs the diffusion-based learning network always has a negative slope as the value on the horizontal axis increases. This shows that by using the activity of the perceptron that outputs the diffusion-based learning network as the similarity, the problem of insufficient sensitivity for measuring similarity (Note 1) can be solved. Furthermore, Figures 13 and 14 show that when N = 100 or greater, it is almost independent of N, solving the problem of difficulty in comparing similarities between different training data with different values ​​of N (Note 2).

[0146] To explain that (Note 3) is resolved, we will explain a method for expressing the degree of similarity between two sets, called Tanimoto similarity or Jaccard similarity, which is described in Non-Patent Document 6 and Non-Patent Document 7. In this specification, these similarities, which have equivalent definitions, will be abbreviated as Tanimoto similarity. Consider two sets A and B. The Tanimoto similarity S T is expressed by the following equation (25).

[0147]

number

[0148] In equation (25), |A| represents the number of elements contained in set A. Here, using the symbols used in equation (7), the Tanimoto similarity S T In this case, if we consider the two sets as the set of components whose value is 1 in the input vector w in the learning phase and the set of components whose value is 1 in the input vector y in the similarity determination phase, then, using the symbols used in equation (7), |A∩B|=n 11 , |A|=n 11 +n 10 , |B|=n 11 +n 01 Substituting these into equation (25), we get the following equation (26).

[0149]

number

[0150] The number of components in w that have a value of 1, N, is N=n 11 +n 10 Therefore, this formula is transformed into n 11 =Nn 10 Substituting this into equation (26), the Tanimoto similarity S T is expressed as the following equation (27).

[0151]

number

[0152] Here, by introducing a constant C, we obtain S RT Define

[0153]

number

[0154] S in equation (28) RT Hereafter, we call the Tanimoto similarity the raised Tanimoto similarity. Now, let us define the Tanimoto similarity included in two raised Tanimoto similarities as S T (1) and S T (2) In this case, the difference in the raised Tanimoto similarity calculated from these is given by the following equation (29).

[0155]

number

[0156] From the above, we can see that the difference in raised Tanimoto similarity is a constant multiple of the difference in Tanimoto similarity. This means that when comparing the magnitude of the difference between two sets, we can use either Tanimoto similarity or raised Tanimoto similarity. Tanimoto similarity is a mathematically defined and widely applied similarity measure, and has been shown to be effective in a variety of fields.

[0157] Figure 15 is a graph comparing the activity of the perceptron that outputs the diffusion information network when the division normalization similarity calculation method, the diffusion learning network, and the noise-addition sensitivity characteristic improvement method are used (the change in output when the number of inputs whose input value is 1 during learning and 0 during similarity judgment is changed) with the raised Tanimoto similarity. Figure 16 is a graph comparing the activity of the perceptron that outputs the diffusion information network when the division normalization similarity calculation method, the diffusion learning network, and the noise-addition sensitivity characteristic improvement method are used (the change in output when the number of inputs whose input value is 0 during learning and 1 during similarity judgment is changed) with the raised Tanimoto similarity.

[0158] In the raised Tanimoto in Figures 15 and 16, the value of C in equation (28) is 0.03. In Figures 15 and 16, the raised Tanimoto similarity is expressed as Raised-Tanimoto. For comparison, the value of the above raised Tanimoto similarity is expressed as the Tanimoto similarity S T The coefficient (1-C) is calculated as (DC). Here, D is the activity of the perceptron that outputs the diffusion learning network when the horizontal axis is 0.

[0159] As can be seen from Figures 15 and 16, the gradient of the activity of the perceptron that outputs the diffusion-based learning network is always negative, which solves (Point 1). Also, even when there are different values ​​of N in the training data, the activity of the perceptron that outputs the diffusion-based learning network is close to the value when N = 100 or more, which solves (Point 2). Furthermore, the activity of the perceptron that outputs the diffusion-based learning network is close to the value of the raised Tanimoto similarity, which solves (Point 3).

[0160] FIG. 17 shows the output of a perceptron when a sigmoid function is used as the activation function in division normalization similarity calculation (when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). FIG. 18 shows the output of a perceptron when a sigmoid function is used as the activation function in division normalization similarity calculation (when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed). FIG. 19 shows the expected value of the perceptron output when a noise-addition sensitivity characteristic improvement method is used (when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). FIG. 20 shows the expected value of the perceptron output when a noise-addition sensitivity characteristic improvement method is used (when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed).

[0161] Implementation examples of the fourth embodiment are Example 1 and Example 2, which will be described in order.

[0162] Example 1 Example 1 is an example of the division normalization type similarity determination process of the fourth embodiment, which is realized by combining the division normalization type similarity determination method with the noise addition type sensitivity characteristic improvement method.

[0163] Fig. 21 is a diagram showing a neural network circuit device in which division normalization-type similarity calculation and a noise-addition-type sensitivity characteristic improvement method are combined, where the activation function is a step function for which an arbitrary threshold can be set. In explaining Fig. 21, the same components as those in Fig. 5 are given the same numbers and explanations will be omitted. In <Example 1>, a step function is used as the activation function. Example 1 adds a noise-addition type sensitivity characteristic improvement method to the first embodiment shown in Fig. 5. A neural network circuit device 700 in Fig. 21 further adds a random number generation circuit 711 and an addition circuit 712 to the neural network circuit device 500 in Fig. 5. The neural network circuit device 700 is a circuit that combines division normalization-type similarity calculation and noise addition-type sensitivity characteristic improvement method.

[0164] The processes of the demultiplexer 501, registers 502 and 510, Bitwise-AND circuit 504, T counter 503 (first counter), T counter 505 (second counter), T counter 506 (third counter), adder circuit 507, shift register 508, and divider circuit 509 in Fig. 21 are the same as those in the first embodiment shown in Fig. 5. A division-normalized similarity is output from the divider circuit 509.

[0165] The random number generator circuit 711 outputs a randomly selected number. The randomly selected number can be a random number that follows a Gaussian probability density function. However, the distribution is not limited to Gaussian, and other distributions such as normal, Poisson, Weibull, or other distributions may be used. The random number generated by the random number generator circuit 711 is input to the adder circuit 712 along with the division normalization similarity output from the divider circuit 509. The adder circuit 712 outputs the sum of the division normalization similarity and the random number. The subsequent processing by the comparator circuit 511 and the register 510 is the same as that of the neural network circuit device 500 in FIG. 5, and the overall output is determined.

[0166] FIG. 22 shows a parallel circuit in which a plurality of neural network circuit devices 700 are connected, which combine the division normalization type similarity calculation and the noise addition type sensitivity characteristic improvement method shown in FIG. FIG. 22 shows the neural network circuit device 700 shown in FIG. 21 as noise-added similarity calculation circuits (721, 722, 723, 724 in the figure). 21 is transmitted to all noise-added similarity calculation circuits 721, 722, 723, and 724. The noise-added similarity calculation circuits 721, 722, 723, and 724 then independently and in parallel perform the processing of FIG. 21 described in the fourth embodiment. The outputs of all noise-added similarity calculation circuits (721, 722, 723, and 724 in the figure) are input to a T counter 705. The T counter 705 calculates the number of times the input is 1, and outputs the result to the averaging circuit 706. The averaging circuit 706 outputs an average value obtained by dividing the input value by the number of noise-added similarity calculation circuits.

[0167] <Example 2> Example 2 is an example of the division normalization type similarity determination process of the fourth embodiment, which is realized by combining the division normalization type similarity determination method with the noise addition type sensitivity characteristic improvement method.

[0168] Fig. 23 is a diagram showing a neural network circuit device in which the division normalization type similarity calculation and the noise addition type sensitivity characteristic improvement method are combined, when the activation function is a linear function for which an arbitrary threshold can be set. In explaining Fig. 23, the same components as those in Fig. 8 are given the same numbers and explanations will be omitted. In <Example 2>, a linear function is used as the activation function. Example 2 adds a noise-addition type sensitivity characteristic improvement method to the first embodiment shown in Fig. 8. A neural network circuit device 800 in Fig. 23 further adds a random number generation circuit 711 and an addition circuit 712 to the neural network circuit device 600 in Fig. 8. The neural network circuit device 800 is a circuit that combines division normalization-type similarity calculation and a noise addition-type sensitivity characteristic improvement method.

[0169] The processes of the demultiplexer 501, register 502, Bitwise-AND circuit 504, T counter 503 (first counter), T counter 505 (second counter), T counter 506 (third counter), shift register 507, adder circuit 508, and divider circuit 509 in Fig. 23 are the same as those in the first embodiment shown in Fig. 8. A division-normalized similarity is output from the divider circuit 509.

[0170] The random number generator 711 outputs a randomly selected number. As the randomly selected number, a random number that follows a probability density function of a Gaussian distribution can be used. However, the distribution is not limited to Gaussian distribution, and normal distribution, Poisson distribution, Weibull distribution, or other distributions may also be used. The random number generated by the random number generator circuit 711 is input to the adder circuit 712 together with the division normalization similarity output from the divider circuit 509. The adder circuit 712 outputs the sum of the division normalization similarity and the random number. The subsequent processing by the comparator circuit 511, registers 510 and 601, subtractor circuit 603, and multiplexer 602 is the same as that of the neural network circuit device 600 in Figure 8, and the overall output is determined.

[0171] FIG. 24 shows a parallel circuit in which a plurality of neural network circuit devices 800 are connected, which combine the division normalization type similarity calculation and the noise addition type sensitivity characteristic improvement method shown in FIG. FIG. 24 shows the neural network circuit device 800 shown in FIG. 23 as noise-added similarity calculation circuits (801, 802, 803, 804 in the figure). An input to the neural network circuit device 800 shown in Fig. 23 is transmitted to all noise-added similarity calculation circuits 801, 802, 803, and 804. Then, each of the noise-added similarity calculation circuits 801, 802, 803, and 804 independently and in parallel performs the processing of Fig. 23 described in the fourth embodiment. The outputs of all the noise-added similarity calculation circuits (801, 802, 803, and 804 in the figure) are input to an adder circuit 805. The adder circuit 805 calculates the sum of the outputs of all the noise-added similarity calculation circuits and outputs the sum to an averaging circuit 806. The averaging circuit 806 outputs an average value by dividing the input value by the number of noise-added similarity calculation circuits.

[0172] Fig. 25 is a diagram showing a comparison between the expected value of the perceptron output (change in output when the number of inputs whose input value is 1 during learning and 0 during similarity judgment is changed) and raised Tanimoto similarity when using the division normalization similarity calculation method and the noise addition sensitivity characteristic improvement method. Fig. 26 is a diagram showing a comparison between the expected value of the perceptron output (change in output when the number of inputs whose input value is 0 during learning and 1 during similarity judgment is changed) and raised Tanimoto similarity when using the division normalization similarity calculation method and the noise addition sensitivity characteristic improvement method. As shown in FIGS. 25 and 26, it can be seen that the output of the division normalization type similarity calculation unit (neural network circuit device 700, 800) can be approximated by the Raised_Tanimoto similarity.

[0173] [Effects of the fourth embodiment] The neural network circuit devices 700, 800 (Figures 21, 23) of this embodiment further include a random number generation circuit 711 (Figures 21, 23) that randomly generates random numbers, and a second adder circuit (adder circuit 712) (Figures 21, 23) that adds the random number generated by the random number generation circuit 711 as noise to the output of the division circuit 509 (Figures 21, 23), and a comparison circuit (comparison circuit 511) (Figures 21, 23) compares the output vector of the second adder circuit with a threshold vector.

[0174] In this way, the neural network circuit devices 700 and 800 are realized as circuits that calculate a similarity by adding a predetermined noise to the similarity calculated in the similarity determination methods (FIGS. 1 to 10) according to the first to third embodiments, and then perform calculations using the similarity to which the noise has been added. That is, in the fourth embodiment, a circuit is realized that calculates a similarity Sd represented by the processing of (1) the division normalization similarity calculation method and (2) the diffusion-based learning network method, then calculates a similarity Sg to which noise has been added, and then performs calculations using Sg instead of Sd.

[0175] When only (1) the division normalization type similarity calculation method and (2) the diffusion type learning network method of the first to third embodiments are used, there are some problems that the sensitivity for measuring similarity is poor in some areas (Note 1), it is difficult to compare the similarity between different learning data having different values ​​of N (number of inputs) (Note 2), and the mathematically defined characteristics become unclear when the above processes (1) and (2) are performed (Note 3).

[0176] In the fourth embodiment, by performing calculations using the similarity Sg to which noise has been added, as can be seen by comparing FIGS. 11 and 13 and FIGS. 12 and 14, the poor sensitivity of measuring similarity is partially eliminated (solution to point 1). Furthermore, as shown in FIGS. 15 and 16, the activity of the perceptron that outputs the diffusion-based learning network is close to the value of the raised Tanimoto similarity (solution to point 3).

[0177] As a result, in the fourth embodiment, the similarity between the information stored in the learning phase and the information input to the similarity determination phase can be accurately measured using the division normalization similarity calculation method and the diffusion learning network.Furthermore, the difference in information and the discrepancy in the calculated degree of similarity in the prior art are eliminated, making it possible to calculate the similarity based on the degree of similarity.

[0178] (Fifth embodiment) The fifth embodiment is an example of application of a division normalization type similarity calculation method using fuzzy logic. In the first to third embodiments, a vector w=(w1, w2, w3, . . . ) representing synaptic weights set by inputs in the learning phase is used. T and the vector y=(y1,y2,y3,…) representing the input of the similarity judgment phase. T In order to calculate the similarity of , the above-mentioned formula (6) and formula (7) have been used. In formula (6) and formula (7), it has been explained that each component of vectors w and y takes only the value of 0 or 1, and the following formula (30) is used.

[0179]

number

[0180] Here, (y·w) in equation (30) represents the inner product, and Σ i w i y i When using this formula (30), the input value can only take on the values ​​of 0 or 1. Therefore, it cannot be applied to applications where multi-level values, such as image brightness, are handled instead of two levels of light and dark, or where stepless values ​​such as real numbers are handled. To solve this problem, we will use the fuzzy logic described in Non-Patent Document 9, as in Non-Patent Document 8, so that the input value can be any real number between 0 and 1. By doing so, for example, if the input value x i When is in the range from the minimum value L to the maximum value H, the value x i (x i By replacing it with (-L) / (HL), it becomes possible to convert it into a real number between 0 and 1, so the above problem can be solved using fuzzy logic.

[0181] In the fifth embodiment, the minimum value selection circuit 904 (FIGS. 28 to 31) selects the minimum value for each component of the learning phase vector and the inference phase vector. Specifically, the minimum value selection circuit 904 performs a fuzzy AND operation to extract the minimum value for each component of the vector. The vector output as a result of selecting the minimum value for each component through the fuzzy AND operation is a logical product vector.

[0182] This replacement will be explained. 0≦w i ≦1,0≦y i ≦1, and the learning input when w is determined is x=(x1,x2,x3,…) T For the components of 0≦x i ≦1, and then Σ i w i y i Also Σ i w i ∧ F y i Here, w i ∧ F y i ∧ in F is the operator, and p∧ F The value of q is the smaller of p and q. More specifically, when p≧q, p ∧ F The value of q becomes q. By this substitution, equation (30) becomes equation (31).

[0183]

number

[0184] In equation (31), z i =w i ∧ F y i is. Regarding the characteristics of equation (31), we will explain the range of values ​​that equation (31) can take, the conditions under which the value of equation (31) becomes the maximum value, and how the value of equation (31) changes when it deviates from the conditions under which it becomes the maximum value. First, the range of values ​​that can be taken by equation (31) will be explained. The range of values ​​that the variables used in equation (31) can take is 0 ≤ w i ≦1, 0≦y i ≦1 and 0≦z i Since z is ≦1, equation (31) never takes a negative value. i When =0, the value of equation (31) is 0, so it can be seen that the value of equation (31) is 0 or more. Next, using equation (31), the maximum value becomes 1 according to equation (32).

[0185]

number

[0186] From the above discussion, it can be seen that the value of equation (31) is equal to or greater than 0 and equal to or less than 1. Secondly, the condition under which the value of formula (31) becomes the maximum value will be explained. Since the maximum value of formula (31) is 1, the following conditional formula (33) is obtained.

[0187]

number

[0188] This can be transformed into equation (34).

[0189]

number

[0190] Further transformation gives the following equation (35).

[0191]

number

[0192] In equation (35), w i -z i ≧0 and y i -z i≧0, the condition to satisfy equation (35) is that for any i, w i =z i , and y i =z i is. Therefore, w i =y i =z i Therefore, the condition for the value of equation (31) to be maximum is that for any i, w i =y i This is when... Thirdly, we will explain how the value of equation (31) changes when the condition for the value of equation (31) to be maximum deviates. In equation (31), w i is determined in the learning phase and is a constant in the similarity determination phase. Therefore, we can rewrite equation (31) as equation (36) to y k Partial differentiation is performed by

[0193]

number

[0194] First, w k <y k Considering the time, z k w k Then, equation (36) becomes the following equation (37).

[0195]

number

[0196] where w i , and ,y i If all of the values ​​of w are not 0, the denominator of the above equation is obviously a positive value, and the numerator of equation (37) is obviously a negative value. k <y k In the range of y k It can be seen that the rate decreases monotonically with increasing Next, w k ≧yk Considering the time, z k =y k Then, equation (37) becomes the following equation (38).

[0197]

number

[0198] where w k , and ,y k If all of the above are not 0, then the denominator of equation (38) is clearly a positive value, and the numerator of equation (38) is also clearly a positive value. w k ≧y k In the range of y k From the above discussion, we can see that when the value of equation (31) moves away from the condition where it is at its maximum value, the value of equation (31) decreases monotonically as the value moves away.

[0199] Fig. 27 is a diagram illustrating an example of similarity calculated by a division normalization similarity calculation method using fuzzy logic. Fig. 27 shows the change in similarity when y = (y1, y2) is changed when w = (w1, w2) = (0.5, 0.5). In other words, this is the calculation result of similarity when replaced with fuzzy logic when w = (w1, w2) = (0.5, 0.5). In Fig. 27, y = (y1, y2) is changed. The similarity in Fig. 27 is calculated based on equation (31). As can be seen from Fig. 27, the further y = (y1, y2) is from y = (0.5, 0.5), the lower the similarity becomes.

[0200] Here, when fuzzy logic is not used, it has been explained using equations (9) and (10) that the similarity expressed by equation (7) decreases as the change in vector y from vector w increases. In the above explanation, the change in vector y from vector w is calculated by the following equations: i No wi This means that the elements change from 0 to 1 and from 1 to 0, which causes n 10 , and n 01 This was explained as the change in similarity when y increases. When using fuzzy logic, each element changes continuously, so partial differentiation is used to calculate the change in similarity when y increases. i No w i The change in the calculated similarity with respect to the change from is explained using equations (37) and (38), and the change in the numerical similarity is explained using FIG. From the above, it can be seen that what has been explained here has the same characteristics as when similarity is calculated using equations (6) and (7), and therefore the equation for calculating similarity can be replaced with equation (31).

[0201] Implementation examples of the fifth embodiment are <Example 3>, <Example 4>, <Example 5>, and <Example 6>, which will be described in order.

[0202] Example 3 An implementation method using a logic circuit for the division normalization type similarity calculation method of the fifth embodiment will be described below. Example 3 is an example in which fuzzy logic is used.

[0203] Fig. 28 is a diagram showing a neural network circuit device that realizes division normalization-type similarity calculation using fuzzy logic when the activation function is a step function for which an arbitrary threshold can be set. Fig. 28 shows a case where the activation function is a step function for which an arbitrary threshold can be set in a division normalization-type similarity calculation method using fuzzy logic. As shown in FIG. 28, the neural network circuit device 900 includes a demultiplexer (DEMUX) 901, registers (Register) 902, 910, 911, 913, addition circuits 903, 905, 906, 907, a minimum value selection circuit 904, a doubling circuit 908, a division circuit 909, a comparison circuit 912, and a multiplexer (MUX) 914.

[0204] [Operation] The operation of the neural network circuit device 900 configured as described above will now be described. <Learning Phase> First, the input vector x=(x1,x2,…,x N ) T The demultiplexer 901 receives the input signal and outputs A1 to A N , and B1~B N The output to either of these is specified by a phase switching signal S input to the demultiplexer 901. The phase switching signal S is a signal that distinguishes between the learning phase and the similarity determination phase. When this signal has a value indicating the learning phase, the input vector x is output to one of the outputs B1 to B2. N At this time, the register 902 stores the value of the input vector x, and N Output from

[0205] In the third embodiment, synaptic weights are determined as w=x, so the x stored in the register 902 is expressed as w=(w1, w2, . . . , w N ) T The output of the register 902 is transmitted to a minimum value selection circuit 904 and an adder circuit 906. The minimum value selection circuit 904 selects A1 to A N , and B1~B N For two inputs, for all i, A i and B i The adder circuits 903, 905, and 906 compare the inputs IN1 to IN N Calculate the sum of the values ​​input to OUT1~OUT M Output from The adder circuit 906 calculates the sum of the synaptic weights w by the sum of the sums of the Σ i w i The following is calculated:

[0206] <Similarity determination phase> Next, in the similarity determination phase, the input vector y is input to the demultiplexer 901, and the input vector y is divided into A1 to A2 based on the phase switching signal. N This output is sent to an adder circuit 903 and a minimum value selection circuit 904. The adder circuit 903 operates in the same way as the adder circuit 906 to obtain the minimum value Σ i y i The following is calculated: The minimum value selection circuit 904 receives the synaptic weight w and the input vector y of the similarity determination phase. i ∧ F y i This result is input to the adder circuit 905. The adder circuit 905 calculates Σ i w i ∧ F y i The result of the addition circuit 905 is further sent to a doubling circuit 907, which outputs a value double the result of the addition circuit 905. This value is the numerator value 2Σ i w i ∧ F y i becomes.

[0207] The outputs of the adder circuit 903 and the adder circuit 906 are respectively Σ i y i , and ,Σ i w i and is sent to the adder circuit 907. The adder circuit 907 calculates the sum of the denominator of equation (31), Σ i w i +Σ i y i The divider circuit 909 calculates and outputs the input value 2Σ from the doubling circuit 908 and the adder circuit 907, respectively. i w i ∧ F y i , and ,Σ i w i +Σ i y i Then, the division circuit 909 receives 2Σ i w i ∧ F yi Σ i w i +Σ i y i Perform a division operation by dividing by. Through the above processing, the division circuit 909 calculates the similarity and outputs the result.

[0208] The register 911 stores the threshold value of the activation function by inputting it in advance. As a result, the comparison circuit 912 stores the calculated similarity and the threshold value in the inputs IN-A1 to IN-A2. M , and IN-B1~IN-B M The comparison results are sent to IN-A1 to IN-A M The value entered in IN-B1~IN-B M If the value is greater than the value entered in , the output A>B will be 1, if they are equal, the output A=B will be 1, and if they are smaller, the output A <Bが1になる。

[0209] Registers 910 and 913 are previously stored with the output value when the similarity exceeds the threshold of the activation function and the output value when it does not exceed the threshold. Depending on the result of the comparison circuit 912, when the output value of the division circuit 909 exceeds the value stored in the register 911, the value stored in the register 910 becomes the output of the multiplexer 914; otherwise, the value stored in the register 913 becomes the output of the multiplexer 914.

[0210] Example 4 An implementation method using a logic circuit for the division normalization type similarity calculation method of the fifth embodiment will be described below. Example 4 is an example in which fuzzy logic is used.

[0211] Fig. 29 is a diagram showing a neural network circuit device that realizes division normalization-type similarity calculation using fuzzy logic when the activation function is a linear function for which an arbitrary threshold can be set. Fig. 29 shows a case where the activation function is a linear function for which an arbitrary threshold can be set in the division normalization-type similarity calculation method using fuzzy logic. As shown in FIG. 29, the neural network circuit device 1000 includes a demultiplexer (DEMUX) 901, registers (Register) 902, 911, 913, addition circuits 903, 905, 906, 907, a minimum value selection circuit 904, a doubling circuit 908, a division circuit 909, a subtraction circuit 1001, a comparison circuit 912, and a multiplexer (MUX) 914. That is, the neural network circuit device 1000 includes a subtraction circuit 1001 that subtracts the value of register 911 (input threshold value) from the output of division circuit 909, instead of register 910 of the neural network circuit device 900 shown in FIG.

[0212] [Operation] The operation of the neural network circuit device 1000 configured as described above will now be described. The input is input to a demultiplexer 901. The operation from the input to the output of the division circuit 909 is the same as the operation from the demultiplexer 901 to the output of the division circuit 909 in the neural network circuit device 900 of Fig. 28. The output of the division circuit 909 represents the degree of similarity. The output of the division circuit 909 is sent to a subtraction circuit 1001 and a comparison circuit 912. In addition to the subtraction circuit 1001, the comparison circuit 912 also receives input from a register 911. The register 911 stores the threshold value of the activation function. The threshold value is input in advance, and the register 911 outputs the stored threshold value.

[0213] The inputs from the subtraction circuit 1001 and the register 911 are received by IN-A1 to IN-AM and IN-B1 to IN-BM of the comparison circuit 912, respectively. The comparison circuit 912 compares the inputs from the division circuit 909 with the threshold value of the activation function (stored in the register 911). The comparison circuit 912 then outputs IN-A1 to IN-A M The value entered in IN-B1~IN-B M If the value is greater than the value entered in , the output A>B will be 1, if they are equal, the output A=B will be 1, and if they are smaller, the output A <Bが1になる。

[0214] The output A>B of the comparison circuit 912 is connected to a multiplexer 914, which has two inputs A1 to A M , and B1~B M Either one of the following is output: OUT1~OUT M The output from the two systems is determined by the value of output A>B input to the multiplexer. When this value is 1, A1~A M OUT1~OUT M When this value is 0, B1~B M OUT1~OUT M The inputs A1 to A of the multiplexer 914 are M The output of the subtraction circuit 1001 is connected to the divider circuit 909. The divider circuit 909 multiplies the output of the doubling circuit 907 by 2Σ i w i ∧ F y i Σ of the adder circuit 908 output i w i +Σ i y i The similarity is calculated by dividing by .

[0215] The value of the division circuit 909 is input to the inputs IN-A1 to IN-A1 of the subtraction circuit 1001. M The inputs IN-B1 to IN-B1 of the subtraction circuit 1001 are also sent to MThe threshold value stored in the register 911 is input to the multiplexer 914. As a result, the output of the subtraction circuit 1001 becomes a value obtained by subtracting the threshold value from the similarity, and this value is sent to the multiplexer 914. M The value stored in register 913 is sent to register 913. The output value when the similarity is equal to or less than the threshold is stored in advance in register 913. When a linear function is used as the activation function, a value of 0 is stored in register 913.

[0216] <Example 5> An implementation method using a logic circuit for the division normalization similarity calculation method of the fifth embodiment will be described. <Example 5> is an example in which the division normalization similarity calculation method and the noise addition sensitivity characteristic improvement method are combined, and the division normalization similarity calculation method using fuzzy logic is used for the former.

[0217] 30 is a diagram showing a neural network circuit device in which a division normalization type similarity calculation using fuzzy logic and a noise addition type sensitivity characteristic improvement method are combined, in the case where the activation function is a step function for which an arbitrary threshold can be set. In Example 5, a step function is used as the activation function. <Example 5> is obtained by adding a noise-addition type sensitivity characteristic improvement method to <Example 3>, and the circuit configuration (implementation example) is shown in FIG. As shown in FIG. 30, a neural network circuit device 1100 includes a random number generating circuit 1101 and an adding circuit 1102 in addition to the neural network circuit device 900 of FIG.

[0218] [Operation] The operation of the neural network circuit device 1100 configured as described above will now be described. The processing from the demultiplexer (DEMUX) 901 to the division circuit 909 of the neural network circuit device 1100 shown in Fig. 30 is as described in Fig. 28. As a result of the processing from the demultiplexer (DEMUX) 901 to the division circuit 909 in Fig. 30, the division circuit 909 outputs a division-normalized similarity using fuzzy logic.

[0219] The random number generator 1101 outputs a randomly selected number. The randomly selected number can be a random number that follows a Gaussian probability density function. However, the distribution is not limited to Gaussian, and other distributions such as normal, Poisson, Weibull, or other distributions are also possible. The random number generated by the random number generator 1101 is input to an adder 1102 along with the division normalization similarity output from the divider 909. The adder 1102 outputs the sum of the division normalization similarity calculated by fuzzy logic and the random number. The output of the adder 1102 is input to a comparator 912. The subsequent processing by the comparator 912, registers 910, 911, and 913, and multiplexer 914 is the same as in FIG. 28, and the overall output is determined.

[0220] Here, in Fig. 24 of <Embodiment 2>, an example in which a plurality of circuits shown in Fig. 23 are connected has been described. In <Embodiment 5>, as in <Embodiment 2>, Fig. 30 can also be used as the noise-added similarity calculation circuits (801, 802, 803, 804) of Fig. 24. In this case, <Embodiment 5> can obtain an output obtained by averaging the outputs of a circuit that combines a plurality of division normalization-type similarities using fuzzy logic and a noise-added sensitivity characteristic improvement method, by the same processing as that described in <Embodiment 4>.

[0221] Example 6 An implementation method using a logic circuit for the division normalization similarity calculation method of the fifth embodiment will be described. Example 6 is an example in which the division normalization similarity calculation method and the noise addition sensitivity characteristic improvement method are combined, and the division normalization similarity calculation method using fuzzy logic is used for the former.

[0222] 31 is a diagram showing a neural network circuit device in which a division normalization type similarity calculation using fuzzy logic and a noise addition type sensitivity characteristic improvement method are combined, in the case where the activation function is a linear function for which an arbitrary threshold can be set. In Example 6, a linear function is used as the activation function. In the sixth embodiment, a noise-addition type sensitivity characteristic improvement method is added to the fourth embodiment, and the circuit configuration (implementation example) is shown in FIG. As shown in FIG. 31, a neural network circuit device 1200 includes a random number generating circuit 1101 and an adding circuit 1102 in addition to the neural network circuit device 1000 of FIG. 29, and the subtracting circuit 1001 of the neural network circuit device 1000 of FIG. 29 is replaced with a subtracting circuit 1201 of FIG. 31.

[0223] [Operation] The operation of the neural network circuit device 1200 configured as described above will now be described. The processing from the demultiplexer (DEMUX) 901 to the division circuit 909 of the neural network circuit device 1200 shown in Fig. 31 is as described in Fig. 28. As a result of the processing from the demultiplexer (DEMUX) 901 to the division circuit 909 in Fig. 31, the division circuit 909 outputs a division-normalized similarity using fuzzy logic.

[0224] The random number generator circuit 1101 outputs a randomly selected number. As the randomly selected number, a random number that follows a Gaussian distribution probability density function can be used. However, the distribution is not limited to Gaussian distribution, and normal distribution, Poisson distribution, Weibull distribution, or other distributions may be used. The random number generated by the random number generator circuit 1101 is input to the adder circuit 1102 together with the division normalization similarity output from the divider circuit 909. The adder circuit 1102 outputs the sum of the division normalization similarity calculated by fuzzy logic and the random number.

[0225] The value of the adder circuit 1102 is input to the inputs IN-A1 to IN-A1 of the subtracter circuit 1201. M The inputs IN-B1 to IN-B2 of the subtraction circuit 1201 are also sent to M The threshold value stored in the register 911 is input to the multiplexer 914. As a result, the output of the subtraction circuit 1201 becomes a value obtained by subtracting the threshold value from the similarity, and this value is sent to the multiplexer 914. M The value stored in register 913 is sent to register 913. The output value when the similarity is equal to or less than the threshold is stored in advance in register 913. When a linear function is used as the activation function, a value of 0 is stored in register 913.

[0226] Here, in Fig. 24 of <Embodiment 2>, an example in which a plurality of circuits shown in Fig. 23 are connected has been described. In <Embodiment 6>, as in <Embodiment 2>, Fig. 31 can also be used as the noise-added similarity calculation circuits (801, 802, 803, 804) of Fig. 24. In this case, <Embodiment 6> can obtain an output obtained by averaging the outputs of a circuit that combines a plurality of division normalization-type similarities using fuzzy logic and a noise-added sensitivity characteristic improvement method, by the same processing as that described in <Embodiment 4>.

[0227] [Effects of the fifth embodiment] As explained above, the neural network circuit devices 900, 1000, 1100, 1200 (Figs. 28 to 31) calculate the degree of similarity between the input of the learning phase and the input of the inference phase using a perceptron modeled on a nerve cell, and include a second adder circuit (adder circuit 903) that adds up each component of the vector of the inference phase, a minimum value selection circuit 904 (Figs. 28 to 31) that selects the minimum value for each component of the vector of the learning phase and the vector of the inference phase, and a fuzzy AND operation that extracts the minimum value component for each component of the vector in the minimum value selection circuit 904, and the fuzzy AND operation is performed on the fuzzy The device is equipped with a third adder circuit (adder circuit 905) (Figures 28 to 31) that adds each component of the AND vector, a fourth adder circuit (adder circuit 906) (Figures 28 to 31) that adds the components of the input vector during the learning phase, a fifth adder circuit (adder circuit 907) (Figures 28 to 31) that adds the outputs of the second adder circuit (adder circuit 903) and the fourth adder circuit (adder circuit 906), a doubling circuit 908 (Figures 28 to 31) that doubles the output of the third adder circuit (adder circuit 905), and a division circuit 909 (Figures 28 to 31) that divides the output value of the doubling circuit 908 by the output value of the fifth adder circuit (adder circuit 907).

[0228] In this way, the neural network circuit devices 900, 1000, 1100, and 1200 are realized as circuits that use fuzzy logic to replace values ​​with values ​​that can take any real number between 0 and 1 in the similarity determination methods (FIGS. 1 to 10) according to the first to third embodiments. This makes it possible to apply the invention to applications where the input values ​​are not limited to 0 or 1, for example, when dealing with multi-level values ​​rather than just two levels of light and dark, such as image brightness, or when dealing with stepless values ​​such as real numbers.

[0229] The present invention is not limited to the above-described embodiments, but includes other modifications and applications within the scope of the claims.

[0230] Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, the embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims.

[0231] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0232] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented by software that causes a processor to interpret and execute programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions may be stored in a memory, a recording device such as a hard disk or a solid-state drive (SSD), or a recording medium such as an integrated circuit (IC) card, a secure digital (SD) card, or an optical disc.

[0233] Furthermore, in the above embodiments, the term "neural network circuit device" is used, but this is for the sake of convenience of explanation, and the term "division normalization type similarity calculation unit," "similarity calculation unit circuit device," etc. may also be used. [Explanation of symbols]

[0234] 100 Division normalization type similarity calculation unit (similarity calculation unit) 500,600,700,800,900,1000,1100,1200 Neural network circuit device (logic circuit) 501 Demultiplexer (DEMUX) 501,510 registers 504 Bitwise-AND circuit (logical product operation circuit) 503 T counter (first counter) 505 T counter (second counter) 506 T Counter (3rd Counter) 507,908 Addition circuit 508 Shift Register 509,909 division circuit 511 Comparison circuit 602 Multiplexer (MUX) 603 Subtraction Circuit 701 Memory (storage unit) 702 Multiplication Circuit 711,1101 Random number generator 712,1102 Addition circuit (6th addition circuit) 903 Addition circuit (second addition circuit) 904 Minimum value selection circuit 905 Addition circuit (third addition circuit) 906 Addition circuit (4th addition circuit) 907 Addition Circuit (5th Addition Circuit) 908 double circuit 909 Division circuit

Claims

1. A neural network circuit device that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron that models a nerve cell, comprising: a logical product calculation circuit for calculating a logical product of the learning phase vector and the inference phase vector; a first counter that counts the number of input vectors whose value is 1 during the inference phase; a second counter that counts the number of inputs having a value of 1 among the logical product vectors obtained by the logical product operation circuit; a third counter that counts the number of inputs in the vector during the learning phase that have a value of 1; an adder circuit that adds the output of the first counter and the output of the third counter; a shift register that shifts the result of the second counter one bit to the most significant side; a division circuit that divides the output vector of the shift register by the output vector of the adder circuit. A neural network circuit device comprising:

2. A neural network circuit device that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron that models a nerve cell, comprising: a second summing circuit for summing each component of the vectors of the inference phase; a minimum value selection circuit that selects the minimum value for each component of the vectors in the learning phase and the vectors in the inference phase; a third adding circuit that uses a fuzzy AND operation to extract the minimum value component for each component of the vector in the minimum value selecting circuit, and adds up each component of the fuzzy AND vector that has been subjected to the fuzzy AND operation; a fourth summing circuit that sums the components of the input vector during the learning phase; a fifth adder circuit that adds the outputs of the second adder circuit and the fourth adder circuit; a doubling circuit that doubles the output of the third adder circuit; a division circuit that divides the output value of the doubler circuit by the output value of the fifth adder circuit. A neural network circuit device comprising:

3. a demultiplexer that receives an input vector in the learning phase and outputs the input signal to either a first output or a second output specified by a phase switching signal; 3. The neural network circuit device according to claim 1 or 2.

4. the division circuit includes a storage unit for storing the reciprocal of a divisor; a multiplication circuit that multiplies a dividend by a reciprocal of the divisor that is read when the divisor is given to the storage unit; 3. The neural network circuit device according to claim 1 or 2.

5. a comparison circuit for comparing the output vector of the division circuit with a threshold vector; 3. The neural network circuit device according to claim 1, further comprising:

6. a subtraction circuit that subtracts a threshold value from the output vector of the division circuit; a multiplexer that switches between the output vector of the subtraction circuit and a predetermined value according to the output of the comparison circuit and outputs the selected vector.

3. The neural network circuit device according to claim 1 or 2.

7. A random number generating circuit that randomly generates random numbers; a sixth adder circuit that adds a random number generated by a random number generating circuit as noise to the output of the division circuit; The comparator circuit compares the output vector of the sixth adder circuit with a threshold vector.

6. The neural network circuit device according to claim 5, wherein:

Images