Similarity determination method, learning inference method, and neural network execution program

Abstract

According to the present invention, when one or more input values are received, each of the input values is one of a value L and a value H, an i-th input value in a training phase is denoted as xi, and an i-th input value in an inference phase is denoted as yi, the i-th input value is assigned with wi, the value wi is set with one of the value L and the value H, a weight wi assigned to the i-th input value in the training phase is set to the value of xi, the number of inputs in which the value of xi is H, the number of inputs in which the values of wi and yi are all H, and the number of inputs in which the value of yi is H are calculated in the inference phase, and then a value is calculated as a similarity that indicates a level of similarity, wherein the value is obtained by dividing the number of inputs in which wi and yi are all H by a value obtained by adding the number of inputs in which the value of wi is H and the number of inputs in which the value of yi is H.

Description

[Technical Field]

[0001] The present invention relates to a similarity determination method, a learning inference method, and a neural network execution program. [Background technology]

[0002] In recent years, artificial intelligence technology using artificial neural networks has advanced, and various industrial applications are being developed. Such neural networks are characterized by the use of a network of connected perceptrons, which model nerve cells. A neural network performs calculations based on inputs to the entire network and outputs the calculation results.

[0003] Perceptrons, used in artificial neural networks, are an extension of earlier models of neurons.

[0004] FIG. 56 illustrates the operation of a perceptron 200 with a variable constant input. As shown in Figure 56, the N+1 input values ​​are b, x1, x2, ... x N are input to the perceptron 200. Of these, there are N external inputs to the entire neural network, and input i has the input value x i is input. b is a constant value stored inside the neural network. Also, one output y is output from the perceptron as the output of the neural network. For input i (i=1, 2, ...N), a value called a weight w i are assigned (hereinafter referred to as synaptic weights). In this case, the output y is expressed by equation (1).

[0005]

number

[0006] Here, f(·) represents the activation function. Commonly used activation functions include nonlinear functions such as the sigmoid function and tanh function, and the ReLU (Rectified Linear Unit function). In equation (1), w i x i To eliminate the difference in notation of a and b and to make the equation easier to read, a circuit like that shown in Figure 57, where the constant input is 1 and the corresponding synaptic weight w0 is b, and the following equation (2) are often used. Figure 57 shows the operation of a perceptron 200, which generalizes the representation of input and synaptic weights.

[0007]

number

[0008] As shown in equation (2), the value passed to the activation function is calculated based on the input value, and the output value is calculated by the activation function. In the following explanation, the value passed to the activation function will be called the activation. When the activation function is expressed as f(a), a is the activation. Typically, when performing machine learning using an artificial neural network, a network in which one or more perceptrons 200 are connected hierarchically is used, as shown in Figure 58. Figure 58 is a diagram showing a multi-layered artificial neural network.

[0009] The artificial neural network is i There are multiple combinations (i=1,2,…,N). Each combination is represented by j, and the input value x of combination j is i When each of (i=1,2,…,N) is considered as a component of a vector, x i (i=1,2,…,N) is a vector x j Here, x j component of x j =(x j1 ,x j2 ,…,x jN ) T Let (x j =(x j1,x j2 ,…,x jN ) T The T in (means that the vector is converted to a column vector) will be expressed as follows.

[0010] Then, for each x j For the target value l j We prepare multiple sets of data with w i The value of is determined. This value is determined by minimizing the error for the entire training data, using the difference between the value calculated by the neural network and the target value as the error.

[0011] In machine learning methods using this type of artificial neural network, the training data itself is not stored within the neural network. On the other hand, there is a machine learning method called k-nearest neighbor method, which stores training data, calculates the similarity between the input and memory patterns, and outputs labels using the k most similar memories. This k-nearest neighbor method is known to enable relatively stable learning even when there is little training data, and has advantages in some applications.

[0012] Furthermore, as described in Non-Patent Document 4, the brain is thought to have a function called pattern completion, which, when multiple external inputs are received, completely recalls a close memory that has already been established in the brain, even if an input pattern that perfectly matches the combination of those inputs is not stored in memory. Finding memories that are close to an external input pattern is one of the functions of human intelligence, and calculating the similarity between an input and a memory pattern provides basic information for finding the most similar memory. Therefore, technology for calculating the similarity between an input and a memory pattern is also important as a component technology for realizing this pattern completion.

[0013] As described above, neural networks are a fundamental technology for artificially realizing intelligent functions such as machine learning and the recall of similar memories, which are thought to be inherent in humans.

[0014] In the neurons and neural networks that form the basis of perceptrons and artificial neural networks, the associative network is a technology that learns and stores information that was previously input, and compares that memory with the current input to determine similarity. Examples of the associative network are described in Non-Patent Documents 1, 2, and 3. Examples of neurons used in the associative network and the associative network are shown in Figures 59 and 60, respectively.

[0015] Figure 59 is a diagram showing an example of a simple associative network. In Figure 59, neuron 300 is represented by a combination of an arrow and a black triangle. The upper side of the triangle (the side without the arrowhead) corresponds to the input part of this neuron, and the lower side of the triangle (the side with the arrowhead) corresponds to the output part of this neuron.

[0016] Suppose a neural network has neuron 300 that changes to a firing state (indicating that the neuron's membrane potential rises and exceeds a threshold) when input A is applied. If input B is repeatedly applied at the same time as input A, a phenomenon occurs in which neuron 300 changes to a firing state simply by input B. This phenomenon is explained by Hebb's law, which states that the simultaneous firing of neuron 300 with the neuron that generates input B strengthens the synaptic connection between input B and neuron 300. The phenomenon in which neuron 300 changes to a firing state simply by input B is called classical conditioning, and inputs A and B are called the unconditioned stimulus and the conditioned stimulus, respectively.

[0017] FIG. 60 is a diagram showing an example of an associative network including multiple unconditioned stimuli. Figure 60 shows a case where different unconditioned stimuli P, Q, and R are related to one conditioned stimulus C by classical conditioning. Unconditioned stimulus P and conditioned stimulus C are input to neuron 301. Unconditioned stimulus Q and conditioned stimulus C are input to neuron 302. Unconditioned stimulus R and conditioned stimulus C are input to neuron 303.

[0018] Next, a technique for determining similarity using an associative network will be described. Fig. 61 is a diagram illustrating a neuron 300, which is a component of a technology for determining similarity using an associative network. Fig. 61 shows the setting of synaptic weights in a simple associative network. Four input values ​​x1, x2, x3, and x4 are input to the neuron 300 in Figure 61. Here, if the input i has the input value x i are input. These input values ​​are either 0 or 1. This is related to the state of the preceding neuron that generates each input, with 0 corresponding to the non-firing state of the preceding neuron (a state in which the membrane potential of the nerve cell has not reached the threshold membrane potential state), and 1 corresponding to the firing state of the preceding neuron. In a non-firing state, neurotransmitters do not reach the connected neuron, and in a firing state, neurotransmitters do reach the neuron. Since the combination of input values ​​to a neuron can be considered as a vector with each of these as components, let x be the vector with components x1, x2, x3, and x4, and let x=(x1,x2,x3,x4). T Hereafter, this x will be called the input vector.

[0019] Synaptic weights are assigned to synapses, which are the parts where inputs connect to neurons. Inputs 1, 2, 3, and 4 are assigned w1, w2, w3, and w4, respectively. Since the combination of these synaptic weights can also be considered as a vector, we will use the same notation as the inputs and define the synaptic weight vector w as w=(w1, w2, w3, w4). T It is expressed as:

[0020] 62A to 62F are diagrams for explaining similarity calculation in the prior art. Figure 62A shows the state of the associative network during learning. Six inputs are connected to a neuron 300 in Figure 62A. In Figure 62A, the input vector x l is x l =(1,0,0,1,0,1) T By this learning, the synaptic weight vectors are set as shown in FIG. 62B. This is because when the neuron 300 shown in FIG. 62A is in the firing state, the input vector x l =(1,0,0,1,0,1) T is added, and for the components of this input vector whose value is 1, the weight of the corresponding synapse is set to 1 based on the Hebbian law. That is, w=x l This becomes:

[0021] As an example of the first similarity determination, as shown in FIG. 62C, the input vector x1 is x1=(1,0,0,1,0,1). T In other words, the same input vector as during training is also added during similarity judgment. In the Associative Network, x1 and the input x l The similarity of x is calculated as the dot product of both vectors. l ·x1. w=x l Therefore, the dot product can be rewritten as w·x1. The degree of similarity calculated in this way (hereafter referred to as dot product similarity) is 3. In this case, the activity of the neuron in Figure 62C, i.e., the value passed to the neuron's activation function to determine the output, is considered to be equal to the dot product similarity. If neuron 300 in Figure 62C has a step function with a threshold of 3 as its activation function, this neuron 300 will output 1.

[0022] As a second example of similarity determination, as shown in FIG. 62D, the input vector x2 is x2=(1,0,0,1,1,0). TIn this case, the dot product similarity is 2, and the input vector x l This indicates that there is one less input with a value of 1. If neuron 300 in FIG. 62D has the same activation function as when the above input vector x2 is input, this dot product similarity does not reach the threshold value of 3, so it will output 0.

[0023] As a third example of similarity determination, as shown in FIG. 62E, an input vector x3 is x3=(1,0,0,1,0,0). T In this case, the dot product similarity is also 2, and the input vector x l This means that there is one less input whose value is 1. In this case, too, 0 will be output, just like in Figure 62D.

[0024] Now, looking at the difference between input vectors x2 and x3, in x2 there is one input where the learning input is 0 and the similarity judgment input is 1, and one input where the learning input is 1 and the similarity judgment input is 0. In other words, there are two inputs where a difference occurs. In contrast, in x3 there is only one input where the learning input is 1 and the similarity judgment input is 0. In other words, there is only one input where a difference occurs. Therefore, in reality, x3 is closer to x l However, the dot product similarity is the same value.

[0025] As a fourth example of similarity determination, as shown in FIG. 62F, an input vector x4 is x4=(1,1,1,1,0,1). T In this case, the dot product similarity is 3, and the input vector x l is the same as the first similarity judgment example that is input as is. However, x1 is x l In contrast to the above, in the case of x4, even though there are two inputs where the learning input is 0 and the similarity judgment input is 1, the result is the same as in the case of x1. [Prior art documents] [Non-patent literature]

[0026] [Non-Patent Document 1] BL McNaughton, RGM Morris, "Hippocampal synaptic enhancement and information storage within a distributed memory system," Trends in Neuroscience, volume 10, Issue 10, pp. 408-415, 1987. [Non-patent document 2] Thomas Trappenberg, Fundamentals of Computational Neuroscience, Oxford University Press, 2010. [Non-patent document 3] Edmund T. Roll, Cerebral Cortex: Principles of Operation, Oxford University Press, 2016. [Non-patent document 4] Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, and AJ Hudspeth, "PRINCIPLES OF NEURAL SCIENCE: Fifth Edition," McGraw-Hill Education, 2012. [Non-patent document 5] David J. Heeger, "Normalization of cell responses in cat striate cortex," Visual Neuroscience, vol. 9, pp. 181-197, 1992. [Non-patent document 6] S. Grossberg, "Contour enhancement, short term memory, and constancies in reverberating neural networks," Studies in Applied Mathematics, vol. 52, no. 3, pp. 213-257, 1973. [Non-Patent Document 7] E. Majani, Ruth Erlanson, Yaser Abu-Mostafa, "On the K-Winners-Take-All Network," Advances in Neural Information Processing Systems 1, pp. 634-642, 1989. [Non-patent document 8] Michael A. Arbib, "The Handbook of Brain Theory and Neural Networks: second edition", Bradford Book, 2002. [Non-Patent Document 9] Chang Xiao, Peilin Zhong, Changxi Zheng, “Enhancing Adversarial Defense by k-Winners-Take-All,” CoRR, abs / 1905.10510, 2019, URL. https: / / arxiv.org / abs / 1905.10510. [Non-Patent Document 10] T. Tanimoto, "An elementary mathematical theory of classification and prediction.", Technical report, International Business Machines Corporation, New York, 1958. [Non-Patent Document 11] P. Jaccard, “The distribution of the flora in the alpine zone”, Phytologist, 1912;11(2):37-50. https: / / doi.org / 10.1111 / j.1469-8137.1912.tb05611.x. [Non-Patent Document 12] GA Carpenter, S. Grossberg, N. Markuzon, JH Reynolds, DB Rosen, "Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analog Multidimensional Maps," IEEE Transactions of Neural Networks, Vol. 3, No. 5, pp. 698-713, 1992. https: / / ieeexplore.ieee.org / stamp / stamp.jsp?tp=&arnumber=159059 [Non-Patent Document 13] L. Zadeh, "Fuzzy sets", Information and Control, Vol. 8, No. 3, pp. 338-353, 1965. Summary of the Invention [Problem to be solved by the invention]

[0027] In an associative network, the input to the neural network is a vector (input vector), and similarity is determined by calculating the dot product of the input vector used during training and the input vector used to determine similarity. In reality, the dot product similarity may be the same for two input vectors whose similarity is being determined, even if there is a difference in the distance from the input vector used during training. For example, in the third similarity judgment example shown in FIG. 62E, x3 is actually closer to x lAlthough the result is close to x1, the dot product similarity may end up being the same value. Also, as in the fourth similarity determination example shown in Figure 62F, in the case of x4, even though there are two inputs where the learning input is 0 and the similarity determination input is 1, the result may be the same as in the case of x1. As described above, the similarity calculation in the prior art has had a problem in that the dot product similarity may not be able to accurately determine the difference between the input vector at the time of learning and the input vector at the time of similarity determination.

[0028] The present invention has been made in view of the above circumstances, and an object of the present invention is to be able to accurately determine the difference between an input vector at the time of learning and an input vector at the time of similarity determination when determining inner product similarity. [Means for solving the problem]

[0029] In order to solve the above-mentioned problems, a similarity determination method is provided that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron modeled on a neuron, and receives one or more input values, each of which is input with a value L or a value H, and calculates the i-th input value in the learning phase as x i and the i-th input value of the inference phase is expressed as y i When expressed as i is assigned and the value w i is set to either a value L or a value H, and the weight value w assigned to the i-th input value in the learning phase is i x i In the inference phase, x i The number of inputs with a value of H, w i and y i The number of inputs for which both are H, y i Calculate the number of inputs for which the value of w is H. i and y i The number of inputs for which both have the value H is i y to the input number where H is the value i The similarity determination method is characterized in that the value obtained by dividing the value of the input by the sum of the number of inputs for which the value is H is calculated as the similarity indicating the degree of similarity. [Effects of the Invention]

[0030] According to the present invention, when determining the dot product similarity, it is possible to accurately determine the difference between the input vector at the time of learning and the input vector at the time of similarity determination. [Brief explanation of the drawings]

[0031] [Figure 1] 1 shows an example of a neural circuit that performs division normalization calculations in a division normalization-type similarity determination method according to a first embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a circuit for performing a division normalization type similarity determination method according to a first embodiment of the present invention. FIG. [Figure 3] FIG. 2 is a diagram illustrating the setting of synapse weights in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a similarity determination phase in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 5] FIG. 2 is a diagram showing an example of a diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing a diffusion-type learning network obtained by excluding a perceptron that adds up the outputs of each perceptron from the diffusion-type learning network of FIG. 5. [Figure 7] FIG. 7 is a diagram illustrating the <learning phase> of operation example 1 (step function) of the diffusion-type learning network shown in FIG. 6. [Figure 8] FIG. 7 is a diagram illustrating Example 1 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-type learning network shown in FIG. 6. [Figure 9] FIG. 7 is a diagram illustrating Example 2 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-based learning network shown in FIG. 6. [Figure 10] FIG. 7 is a diagram illustrating Example 3 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-based learning network shown in FIG. 6. [Figure 11]FIG. 7 is a diagram illustrating the <learning phase> of an operation example 2 (linear function) of the diffusion-type learning network shown in FIG. 6. [Figure 12] FIG. 7 is a diagram illustrating Example 1 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. 6. [Figure 13] FIG. 7 is a diagram illustrating Example 2 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. 6. [Figure 14] FIG. 7 is a diagram illustrating Example 3 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. 6. [Figure 15] 10 is a flowchart showing processing in a learning phase of a division normalization type similarity calculation unit in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 16] 10 is a flowchart showing processing in a similarity determination phase of a division normalization similarity calculation unit of the division normalization similarity determination method according to the first embodiment of the present invention. [Figure 17] 10 is a flowchart showing processing in a learning phase of a division normalization type similarity calculation unit in the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 18] 10 is a flowchart showing processing in a similarity determination phase of a division normalization similarity calculation unit of the division normalization similarity determination method according to the first embodiment of the present invention. [Figure 19] FIG. 1 is a diagram showing a neural network obtained by combining a division normalization-based similarity determination method and a diffusion-based learning network according to a first embodiment of the present invention. [Figure 20] 10 is a flowchart showing processing in a learning phase of Example 3 of the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 21] 10 is a flowchart showing processing in a similarity determination phase of Example 3 of the division normalization similarity determination method according to the first embodiment of the present invention. [Figure 22]10 is a flowchart showing processing in a learning phase of Example 4 of the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 23] 10 is a flowchart showing processing in a similarity determination phase of Example 4 of the division normalization type similarity determination method according to the first embodiment of the present invention. [Figure 24] FIG. 1 is a diagram showing a diffusion-type learning network having a perceptron in a division normalization-type similarity determination method according to a first embodiment of the present invention. [Figure 25] FIG. 10 is a diagram showing an information association network of Example 5 in which inference is performed by combining the division normalization similarity calculation method, the diffusion learning network, and the separate memory inference method according to the first embodiment of the present invention. [Figure 26] 10 is a flowchart showing the processing in the learning phase of Example 5 of the separated memory inference method according to the first embodiment of the present invention. [Figure 27] 10 is a flowchart showing processing in an inference phase of Example 5 of the separated-storage inference method according to the first embodiment of the present invention. [Figure 28] FIG. 10 is a diagram showing an information association network of Example 6 in which inference is performed by combining the division normalization similarity calculation method, the diffusion learning network, and the separate memory inference method according to the first embodiment of the present invention. [Figure 29] FIG. 10 is a diagram showing the effect of the diffusion information network when m is changed, with the activation function of the perceptron in the division normalization similarity calculation unit of the division normalization similarity determination method according to the first embodiment of the present invention set to a step function, N=100, p=0.05, and k=0. [Figure 30] FIG. 30 is a diagram showing the effect of a diffusion-type information network when p=1.0 in comparison with FIG. 29. [Figure 31] FIG. 29 is a diagram showing the effect of the diffusion-based learning network when the value of k is changed with m=0 in FIG. [Figure 32] FIG. 31 is a diagram showing the effect of the diffusion-based learning network when the value of k is changed with m=0 in FIG. 30. [Figure 33] FIG. 29 is a diagram showing the effect of the diffusion-based learning network when the values ​​of m and k are changed simultaneously, with m=k in FIG. [Figure 34] FIG. 31 is a diagram showing the effect of the diffusion-based learning network when the values ​​of m and k are changed simultaneously, with m=k in FIG. 30. [Figure 35] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=0.05, and k=0). [Figure 36] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=1.0 and k=0). [Figure 37] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=0.05, and m=0). [Figure 38] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=1.0 and m=0). [Figure 39] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=0.05, and m=k). [Figure 40] FIG. 10 is a diagram showing the effect of the diffusion-type learning network in the division normalization-type similarity determination method according to the first embodiment of the present invention (when a linear function, p=1.0, and m=k). [Figure 41] FIG. 10 is a diagram showing the activity (N=100) of a perceptron that outputs a diffusion information network when using a division normalization type similarity calculation method according to a second embodiment of the present invention and only a diffusion type learning network. [Figure 42] FIG. 10 is a diagram showing the activity (N=1000) of a perceptron that outputs a diffusion information network when using a division normalization type similarity calculation method according to a second embodiment of the present invention and only a diffusion type learning network. [Figure 43]FIG. 10 is a diagram showing the activity (change in output when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed) of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method according to the second embodiment of the present invention are used. [Figure 44] FIG. 10 is a diagram showing the activity (change in output when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed) of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method according to the second embodiment of the present invention are used. [Figure 45] FIG. 10 is a graph comparing the activity of a perceptron that outputs a diffusion information network when a division normalization type similarity calculation method, a diffusion type learning network, and a noise addition type sensitivity characteristic improvement method according to the second embodiment of the present invention are used (changes in output when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed) with raised Tanimoto similarity. [Figure 46] FIG. 10 is a graph comparing the activity (change in output when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed) of the perceptron that outputs the diffusion information network when the division normalization type similarity calculation method, the diffusion type learning network, and the noise addition type sensitivity characteristic improvement method according to the second embodiment of the present invention are used, with the raised Tanimoto similarity. [Figure 47] 10 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit according to a second embodiment of the present invention. [Figure 48] 10 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit according to a second embodiment of the present invention. [Figure 49] 10 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit according to a second embodiment of the present invention. [Figure 50] 10 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit according to a second embodiment of the present invention. [Figure 51] FIG. 11 is a diagram illustrating an example of similarity obtained by a division normalization type similarity calculation method using fuzzy logic according to the third embodiment of the present invention. [Figure 52] 10 is a flowchart showing the process of the learning phase in the division normalization type similarity calculation method using fuzzy logic according to the third embodiment of the present invention. [Figure 53] 11 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit when a noise-addition type sensitivity characteristic improvement method according to a third embodiment of the present invention is not used. [Figure 54] 11 is a flowchart showing processing in an inference phase of a division normalization type similarity calculation unit when a noise-addition type sensitivity characteristic improving method according to a third embodiment of the present invention is used. [Figure 55] 1 is a hardware configuration diagram showing an example of a computer that realizes the functions of a division normalization type similarity calculation unit in the division normalization type similarity determination method according to the first embodiment of the present invention. FIG. [Figure 56] FIG. 1 illustrates the operation of a perceptron with variable constant inputs. [Figure 57] This figure shows the operation of a perceptron with a generalized representation of inputs and synaptic weights. [Figure 58] Figure 1 shows a multi-layered artificial neural network. [Figure 59] FIG. 1 is a diagram illustrating an example of a simple associative network. [Figure 60] FIG. 1 is a diagram showing an example of an associative network including multiple unconditioned stimuli. [Figure 61] FIG. 1 is a diagram illustrating neurons that are components of a technology for determining similarity using an associative network. [Figure 62A] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 62B] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 62C]FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 62D] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 62E] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. [Figure 62F] FIG. 1 is a diagram illustrating a similarity calculation in the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, a similarity determination method, a similarity calculation unit, a diffusion-based learning network, and an execution program for a neural network in an embodiment for implementing the present invention (hereinafter referred to as the "first embodiment") will be described with reference to the drawings. (First embodiment) The present invention is realized by combining the division normalization type similarity determination method and the diffusion type learning network method. [Division normalization type similarity determination method] First, the division normalization type similarity determination method (similarity determination method) will be described. In the similarity determination using the Associative Network, which was explained as an existing technology, similarity is calculated by taking the dot product of the input vector during learning and the input vector during similarity determination. Therefore, each neuron has the ability to calculate the product (i.e., multiplication as an operation) of the input value and the synaptic weight value for each input, and to add the product values ​​for all inputs. Generally speaking, if the input value can take any real value, and the input value and synaptic weight value can also be negative, it actually has the ability to perform multiplication, addition, and subtraction.

[0033] In contrast, the division normalization similarity determination method incorporates into the perceptron model not only multiplication, addition, and subtraction, but also operations caused by a phenomenon called the shunt effect of nerve cells (neurons) (Non-Patent Document 4). The shunt effect occurs in nerve cells due to inhibitory synapses formed near the cell body. The shunt effect is an effect in which the entire summed signal transmitted to a neuron is divided by the signal transmitted via an inhibitory synapse formed near the cell body. This division caused by the shunt effect is also used in a model called division normalization that explains the adjustment of visual sensitivity, as described in Non-Patent Document 5.

[0034] Figure 1 shows an example of a division normalization-type similarity calculation unit for division normalization, representing an example of a neural circuit that performs division normalization calculations. In Figure 1, neurons 001, 002, and 003, which are represented by black triangles, form excitatory synapses with neurons 005, 006, and 007, respectively. Neuron 004, which is represented by a white triangle (△), forms inhibitory synapses 008, 009, and 010. Here, an excitatory synapse is a synapse that has the effect of shifting the activation state of the neuron receiving the synapse toward firing. Conversely, an inhibitory synapse is a synapse that has the effect of shifting the activation state toward quiescence. In Figure 1, the inhibitory synapses 008, 009, and 010 formed by neuron 004 are connected to the black triangles, which indicates that the inhibitory synapses 008, 009, and 010 exhibit a shunt effect.

[0035] Neurons 001, 002, and 003 in Figure 1 receive inputs 1 and 2, 3 and 4, and 5 and 6, respectively, and input values ​​x1 and x2, x3 and x4, and x5 and x6, respectively. Assume that these inputs cause the output values ​​of neurons 001, 002, and 003 to become e1, e2, and e3, respectively. The output values ​​e1, e2, and e3 are sent to neurons 005, 006, and 007, respectively. Assume that these output values ​​are transmitted as is to neurons 005, 006, and 007, and become their respective activity values. Neuron 004 also receives e1, e2, and e3 as is, and sets its activity value to Σ 3 j=1 e j Then, the activity of neuron 004 is output as is and sent to neurons 005, 006, and 007, causing a shunt effect at synapses 008, 009, and 010. In this case, the effect of division normalization is expressed by the following equation, and neurons 005, 006, and 007 have activity expressed by this equation (3). Here, k is 1, 2, or 3.

[0036]

number

[0037] In this case, the activity of neurons 005, 006, and 007 is the value when the numerators are e1, e2, and e3, respectively, in equation (3). In this way, in division normalization, the activity of a neuron is divided by the sum of the outputs of multiple neurons called the neuron pool (neurons 001, 002, and 003 in the example in Figure 1). This effect explains the adjustment of visual sensitivity. The division normalization model does not take into account changes in synaptic weights due to learning. Furthermore, the value of C is determined experimentally to prevent saturation of the current visual input, so there is no clear method for determining it based on the input during learning, etc.

[0038] The division normalization similarity determination method of the present invention is realized by (A) a method for determining synaptic weights, (B) a method for determining the division normalization constant C, and (C) a method for determining a perceptron set (hereinafter referred to as a perceptron pool) that corresponds to a neuron pool in division normalization, which are described below.

[0039] 2 is a diagram showing an example of a division normalization type similarity calculation unit (similarity calculation unit) that performs the division normalization type similarity determination method, and illustrates the learning phase in the example of the division normalization type similarity determination method. Hereinafter, the module that executes the processing of the division normalization type similarity determination method will be referred to as the division normalization type similarity calculation unit 100 (similarity calculation unit). The input values ​​x1, x2, x3, x4, x5, and x6 to inputs 1, 2, 3, 4, 5, and 6 shown in FIG. 2 represent the input values ​​to the division normalization type similarity calculation unit 100. These are input equally to perceptrons 001 and 002. In this way, in the division normalization type similarity determination method, only the inputs to the division normalization type similarity calculation unit are used as the perceptron pool in division normalization (C). Each input takes two types of values ​​when the previous perceptron is in a resting state and when it is in a firing state, and in this specification these are represented by 0 and 1, respectively. That is, x i ∈{0,1}(i=1,2,3,4,5,6).

[0040] Fig. 3 shows the setting of synaptic weights in the division normalization type similarity determination method. Fig. 3 shows that as a result of the learning phase in Fig. 2, the synaptic weights formed in perceptron 001 by input values ​​x1, x2, x3, x4, x5, and x6 are w1, w2, w3, w4, w5, and w6.

[0041] In the division normalization type similarity judgment method (A), the synaptic weight determination method is i =x iIn other words, the weight of a synapse that receives an input signal corresponding to the firing state during the learning phase is 1, and the weight of a synapse that receives an input signal corresponding to the quiet state is 0.

[0042] Figure 4 shows the similarity determination phase in the division normalization type similarity determination method. Figure 4 shows the similarity determination phase when input values ​​y1, y2, y3, y4, y5, and y6 arrive. At this time, the input to the perceptron 001 is Σ 6 j=1 y j ·w j On the other hand, there is no change in the synaptic weights of Perceptron 002, and Σ 6 j=1 y j The output of the perceptron 002 generates a shunt effect on the perceptron 001 through a synapse 003 formed between the perceptron 002 and the perceptron 001, and calculates the following operation.

[0043]

number

[0044] Furthermore, as a method for determining the constant C for division normalization (B), the constant C is set to a value calculated as follows in the learning phase:

[0045]

number

[0046] where x=(x1,x2,x3,x4,x5,x6) T where ||x|| represents the norm of vector x. When equation (5) is substituted into equation (4), equation (4) is transformed into the following equation (6).

[0047]

number

[0048] However, y=(y1,y2,y3,y4,y5,y6) T and w=(w1,w2,w3,w4,w5,w6) T is. Equation (6) includes the square of the norm and the dot product of two vectors as vector operations. In general, a vector v = (v1, v2, ..., v N ) T , and vector u=(u1,u2,…,u N ) T When there was ||u|| 2 =u1 2 +u2 2 +…+u N 2 and u·v=u1v1+u2v2+…+u N v N is.

[0049] Now, u i ∈{0,1} and v i ∈{0,1}, then ||u|| 2 =u1 2 +u2 2 +…+u N 2 =u1+u2+…+u N and u v =u1v1+u2v2+…+u N v N =Σ N i=1 u i v i =Σ N i=1 (u i ANDv i ) can also be calculated as u i ANDv i u i and v i represents the logical AND operation of

[0050] where n 11 , n 10 , n 01 , and n 00 respectively, x i =1 and y i= 1, x i =1 and y i = 0, x i =0 and y i = 1, and the number of inputs x i =0 and y i = 0. Also, N = n 11 +n 10 +n 01 +n 00 is assumed to be constant since it represents the total number of inputs. The above equation (6) can be transformed as follows:

[0051]

number

[0052] In the calculation of formula (7), if the denominator is 0, n 11 , n 10 , n 01 Since all of these are 0, the numerator is also n 11 , the value is also 0. In this case, the calculation result of equation (7) is calculated as 0 because there is no similarity between the two vectors. Now, when the same input is used in the learning phase and the similarity judgment phase, n 10 =n 01 =0, so we get equation (8).

[0053]

number

[0054] Next, consider the case where the input differs between the learning phase and the similarity judgment phase. f =n 11 +n 10 is the number of inputs of 1 during learning, and remains constant in the similarity judgment phase after the learning phase. f Using this, equation (7) can be transformed as follows:

[0055]

number

[0056] From this equation (9), the value calculated by equation (9) is n 10 and n 01 From this, we can see that it changes only depending on n 10 and n 01 We explain how the value of equation (9) changes with the change of

[0057] <n 10 Changes in First, n 10 Consider the change in equation (9) with respect to the change in . Equation (9) is transformed into the following equation (10).

[0058]

number

[0059] In equation (10), n 01 If we keep constant, n 10 It can be seen that the value of the above equation decreases monotonically as increases.

[0060] <n 01 Changes in Second, n 01 Consider the change in equation (9) with respect to the change in n 10 If we keep constant, n 01 It can be seen that the value of equation (9) decreases monotonically with increasing From the above, equation (7) is 10 =n 01 =0 gives the value 1, and n 10 and n 01 It monotonically decreases with the increase of n, which represents the degree of similarity. 10 and n 01 It can be seen that this solves the problem that the degree of similarity does not change even if the value of

[0061] <The exact meaning of the value calculated by the division normalization similarity calculation method> Next, the exact meaning of the value calculated by the division normalization type similarity calculation method will be explained. Two equations, S d , and ,S c Think about it.

[0062]

number

[0063] Equation (11) is 11 +n 10 This is the equation that becomes the division normalization type similarity calculation method of the present invention when

[0064]

number

[0065] Equation (12) is the case where c2 is n 11 +n 10 When x is a vector, it represents the cosine similarity between vectors x and y. Cosine similarity represents the degree of similarity between two vectors. Specifically, it is the cosine value of the angle between two vectors in vector space. This value is calculated by dividing the dot product of two vectors (an operation in which the products of corresponding components of two vectors are added for all components) by the product of the magnitudes (norms) of the two vectors.

[0066] First, let u and v be 11 , and n 01 Substituting these into the above formulas (11) and (12), we get S d , and ,S c is expressed as a function of u and v as follows:

[0067]

number

[0068]

number

[0069] Now, in general, as a Taylor expansion of a function f(u,v) around (u,v), if we consider up to the first-order terms, we get the Taylor series f (1) (u+h,v+k) is expressed as follows:

[0070]

number

[0071] Using this, S d (u,v), and S c Taylor series S of (u,v) around (u,v) up to first order d (1) (u+h,v+k), and S c (1) Calculating (u+h,v+k) gives us the following:

[0072]

number

[0073]

number

[0074] In the above equations (16) and (17), c1 = c2 = n 11 +n 10 =N f , u=N f , and substituting v=0, we get the following.

[0075]

number

[0076]

number

[0077] Therefore, c1=c2=n 11 +n 10 =N f , u=N f , and when v=0, the following equality holds:

[0078]

number

[0079] From the above, it can be seen that the value calculated by the division normalization type similarity determination method of the present invention is an approximation of the cosine similarity, and therefore the similarity calculated by the division normalization type similarity determination method can calculate the recognized similarity more accurately than the existing technology.

[0080] [Diffusion learning network method] Next, the diffusion-type learning network method will be described. FIG. 5 is a diagram illustrating an example of a diffusion-type learning network. As shown in FIG. 5, in diffusion-based learning network 1000, multiple division normalization type similarity calculation units 100 having some or all of the inputs (the portions in FIG. 5 where input values ​​x1, x2, x3, x4, x5, x6, etc. are input) are connected to the inputs. Furthermore, the outputs of each division normalization type similarity calculation unit 100 output output values ​​z1, z2, z3, z4, z5, z6, which are input to perceptron 013.

[0081] As a result, in diffusion-type learning network 1000, output values ​​z1, z2, z3, z4, z5, and z6 are added in perceptron 013, and then an output value according to the activation function of perceptron 013 is output from z7.

[0082] The operation of the components other than the perceptron 013 will be described below with reference to FIG. 6, which shows diffusion-based learning network 1000 with perceptron 013 removed. Figure 6 is a diagram showing a diffusion-type learning network in which the perceptron that adds the outputs of each perceptron is removed from the diffusion-type learning network in Figure 5. For ease of explanation, diffusion-type learning network 1000 in Figure 6 in which perceptron 013 is removed from diffusion-type learning network 1000 is denoted by the same reference numeral.

[0083] Examples of the operation of the diffusion-type learning network include Operation Example 1 when using a (step function) (Figs. 7 to 10), and Operation Example 2 when using a (linear function) (Figs. 11 to 14). Each of Operation Examples 1 and 2 can be further divided into a <learning phase> (Figs. 7 and 11), a (step function) <similarity determination phase> (Figs. 8 to 10), and a (linear function) <similarity determination phase> (Figs. 12 to 14). These will be explained in order below.

[0084] <Example 1 (Step function)> First, an operation example 1 (step function) of the diffusion-type learning network will be described. FIG. 7 is a diagram illustrating the "learning phase" of the first operational example (step function) of the diffusion-type learning network shown in FIG. In Figure 7, the learning phase is x=(x1,x2,x3,x4,x5,x6). T =(1,0,1,1,0,1) T This shows the state when you enter In this case, the activation functions of perceptrons 001, 002, 003, 004, 005, and 006 are step functions with a threshold of 0.6.

[0085] During this learning phase, the synaptic weights of perceptrons 001, 002, 003, 004, 005, and 006 change in the same way as during the learning phase of the division normalization similarity determination method. That is, when the learning input is 1, the synaptic weight associated with that input is set to 1, and when the input is 0, the synaptic weight is set to 0. As a result, perceptrons 001, 002, 003, 004, 005, and 006 have two, one, one, one, one, and two synapses with a weight of 1, respectively.

[0086] FIG. 8 is a diagram illustrating Example 1 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-type learning network shown in FIG. In Example 1 of the <Similarity Determination Phase> in Figure 8, (y1, y2, y3, y4, y5, y6) T =(1,0,1,1,0,1) T This figure shows the state when input is made. This input is the same as the input in the <Learning Phase> in Figure 7. At this time, perceptrons 001 to 006 calculate the similarity as follows, depending on the synaptic weights that have changed depending on the input value in the <Learning Phase> and the input value in the similarity judgment phase.

[0087] (1) Perceptron 001 and 006 The value calculated by the division normalization type similarity determination method is as follows: In the following formula, the last value is compared to 0.6 because 0.6 is set as the threshold value of the activation function of the perceptron.

[0088]

number

[0089] (2) Perceptrons 002, 003, 004, and 005 The value calculated by the division normalization type similarity determination method is as follows:

[0090]

number

[0091] As described above, all perceptrons have inputs that exceed the threshold, and since their activation functions are step functions, the output is 1. Therefore, all perceptrons 001, 002, 003, 004, 005, and 006 output 1. As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013, and when the activity of this perceptron is represented by the sum of the input values ​​and the activation function is represented by a linear function with the threshold value of 0, perceptron 013 outputs 6.

[0092] FIG. 9 is a diagram illustrating Example 2 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-type learning network shown in FIG. In Example 2 of the <Similarity Determination Phase> in Figure 9, the similarity determination phase has (y1, y2, y3, y4, y5, y6) T =(1,1,0,0,0,1) T Given the input:

[0093] (1) Perceptron 001 The value calculated by the division normalization type similarity determination method is as follows:

[0094]

number

[0095] (2) Perceptron 002 The value calculated by the division normalization type similarity determination method is as follows:

[0096]

number

[0097] (3) Perceptron 003 and 005 The value calculated by the division normalization type similarity determination method is as follows:

[0098]

number

[0099] (4) Perceptron 004 The value calculated by the division normalization type similarity determination method is as follows:

[0100]

number

[0101] (5) Perceptron 006 The value calculated by the division normalization type similarity determination method is as follows:

[0102]

number

[0103] As a result of the above, the output of the three perceptrons, perceptrons 001, 002, and 006, is 1. As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013, and when the activity of this perceptron is expressed as the sum of the input values ​​and the activation function is expressed as a linear function with a threshold of 0, perceptron 013 outputs 3. Here, if we consider the case where all inputs are connected to one perceptron, the value calculated by the division normalization type similarity determination method will be as follows:

[0104]

number

[0105] In this case, similarity could not be calculated without the diffusion learning network. On the other hand, in the example in Figure 9, the effect of the diffusion learning network causes a bias in the situation where the input to some perceptrons is 1 both during learning and when determining similarity, which causes three perceptrons to fire, making it possible to determine similarity.

[0106] FIG. 10 is a diagram illustrating Example 3 of the <similarity determination phase> of Operation Example 1 (step function) of the diffusion-type learning network shown in FIG. In Example 3 of the <Similarity Determination Phase> in Figure 10, the similarity determination phase includes (y1, y2, y3, y4, y5, y6) T =(1,0,1,1,1,0) T Given the input:

[0107] (1) Perceptron 001 The value calculated by the division normalization type similarity determination method is as follows:

[0108]

number

[0109] (2) Perceptron 002 The value calculated by the division normalization type similarity determination method is as follows:

[0110]

number

[0111] (3) Perceptron 003 The value calculated by the division normalization type similarity determination method is as follows:

[0112]

number

[0113] (4) Perceptron 004 The value calculated by the division normalization type similarity determination method is as follows:

[0114]

number

[0115] (5) Perceptron 005 The value calculated by the division normalization type similarity determination method is as follows:

[0116]

number

[0117] (6) Perceptron 006 The value calculated by the division normalization type similarity determination method is as follows:

[0118]

number

[0119] As a result of the above, the output of the five perceptrons, perceptrons 001, 003, 004, 005, and 006, is 1. As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013, and when the activity of this perceptron is expressed as the sum of the input values ​​and the activation function is expressed as a linear function with a threshold of 0, perceptron 013 outputs 5. Here, if we consider the case where all inputs are connected to one perceptron, the value calculated by the division normalization type similarity determination method will be as follows:

[0120]

number

[0121] In this case, similarity can be determined without a diffusion-based learning network. On the other hand, in this example, the output is 5, while in the previous example, it is 3. This is because, due to the so-called sparse distributed learning network, only a portion of the total input is input to the division normalization similarity determination method, and the output changes depending on the degree of bias. Therefore, the higher the similarity, the more likely it is that the input to the division normalization similarity determination method will have an activation level that exceeds the activation function threshold, even if the bias is small. Therefore, the output in this example is large. This shows that the diffusion-based learning network makes it possible to determine similarity for a wide range of inputs.

[0122] The above explains the operation when the activation function is a step function with a threshold of 0.6. From here on, we will explain the operation when the activation function is a linear function with a threshold of 0.6, using Figure 11.

[0123] FIG. 11 is a diagram illustrating the learning phase of the second operation example (linear function) of the diffusion-type learning network shown in FIG. In Figure 11, the learning phase is x=(x1,x2,x3,x4,x5,x6). T =(1,0,1,1,0,1) T This shows the state when you enter In this case, the activation functions of perceptrons 001, 002, 003, 004, 005, and 006 are linear functions with a threshold of 0.6 and a slope of 1.

[0124] During this learning phase, the synaptic weights of perceptrons 001, 002, 003, 004, 005, and 006 change in the same way as during the learning phase of the division normalization similarity determination method. That is, when the learning input is 1, the synaptic weight associated with that input changes to 1, and when the input is 0, the synaptic weight is 0. As a result, perceptrons 001, 002, 003, 004, 005, and 006 have two, one, one, one, one, and two synapses with a weight of 1, respectively.

[0125] FIG. 12 is a diagram illustrating Example 1 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. In Example 1 of the <Similarity Determination Phase> in Figure 12, (y1, y2, y3, y4, y5, y6) T =(1,0,1,1,0,1) T This shows the state when input is made. This input is the same as the input in the learning phase.

[0126] At this time, perceptrons 001 to 006 calculate the similarity and output as follows, depending on the synaptic weights that have changed depending on the input values ​​in the learning phase and the input values ​​in the similarity judgment phase. In the following, a linear function with a threshold of 0.6 and a slope of 1 is defined as f l This will be expressed as (a).

[0127] (1) Perceptron 001 and 006 The value calculated by the division normalization type similarity determination method is as follows:

[0128]

number

[0129] Therefore, f l (S d )=f l The output is (1)=0.4.

[0130] (2) Perceptrons 002, 003, 004, and 005 The value calculated by the division normalization type similarity determination method is as follows:

[0131]

number

[0132] Therefore, f l (S d )=f l The output is (1)=0.4.

[0133] As described above, all perceptrons have inputs that exceed the threshold, and generate outputs proportional to the similarity. As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013. When the activity of this perceptron is expressed as the sum of the input values ​​and the activation function is expressed as a linear function with a threshold of 0, perceptron 013 outputs 2.4.

[0134] FIG. 13 is a diagram illustrating Example 2 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. In Example 2 of the <Similarity Determination Phase> in Figure 13, (y1, y2, y3, y4, y5, y6) T =(1,1,0,0,0,1) T This shows the state when

[0135] (1) Perceptron 001 The value calculated by the division normalization type similarity determination method is as follows:

[0136]

number

[0137] Therefore, f l (S d )=f l The output is (2 / 3)=2 / 3-0.6.

[0138] (2) Perceptron 002 The value calculated by the division normalization type similarity determination method is as follows:

[0139]

number

[0140] Therefore, f l (S d )=f lThe output is (2 / 3)=2 / 3-0.6.

[0141] (3) Perceptron 003 and 005 The value calculated by the division normalization type similarity determination method is as follows:

[0142]

number

[0143] Therefore, f l (S d )=f l (0)=0 is the output.

[0144] (4) Perceptron 004 The value calculated by the division normalization type similarity determination method is as follows:

[0145]

number

[0146] Therefore, f l (S d )=f l (0)=0 is the output.

[0147] (5) Perceptron 006 The value calculated by the division normalization type similarity determination method is as follows:

[0148]

number

[0149] Therefore, f l (S d )=f l The output is (1)=0.4.

[0150] As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013, and when the activity of this perceptron is represented by the sum of the input values ​​and the activation function is represented by a linear function with a threshold of 0, perceptron 013 outputs 4 / 3-0.8 ≒ 0.53.

[0151]

number

[0152] FIG. 14 is a diagram illustrating Example 3 of the <similarity determination phase> of Operation Example 2 (linear function) of the diffusion-type learning network shown in FIG. In Example 3 of the <Similarity Determination Phase> in Figure 14, (y1, y2, y3, y4, y5, y6) T =(1,0,1,1,1,0) T This shows the state when

[0153] (1) Perceptron 001 The value calculated by the division normalization type similarity determination method is as follows:

[0154]

number

[0155] Therefore, f l (S d )=f l The output is (1)=0.4.

[0156] (2) Perceptron 002 The value calculated by the division normalization type similarity determination method is as follows:

[0157]

number

[0158] Therefore, f l(S d )=f l (0)=0 is the output.

[0159] (3) Perceptron 003 The value calculated by the division normalization type similarity determination method is as follows:

[0160]

number

[0161] Therefore, f l (S d )=f l The output is (2 / 3)=2 / 3-0.6.

[0162] (4) Perceptron 004 The value calculated by the division normalization type similarity determination method is as follows:

[0163]

number

[0164] Therefore, f l (S d )=f l The output is (1)=0.4.

[0165] (5) Perceptron 005 The value calculated by the division normalization type similarity determination method is as follows:

[0166]

number

[0167] Therefore, f l (S d )=f l The output is (2 / 3)=2 / 3-0.6.

[0168] (6) Perceptron 006 The value calculated by the division normalization type similarity determination method is as follows:

[0169]

number

[0170] Therefore, f l (S d )=f l The output is (2 / 3)=2 / 3-0.6.

[0171] As shown in Figure 5, the outputs of perceptrons 001, 002, 003, 004, 005, and 006 are input to perceptron 013, and when the activity of this perceptron is expressed as the sum of the input values ​​and the activation function is expressed as a linear function with a threshold of 0, perceptron 013 becomes as follows.

[0172]

number

[0173] The division normalization similarity determination method and the diffusion learning network method have been described above. Below, the division normalization similarity calculation unit of the diffusion learning network will be described.

[0174] [Division normalization-based similarity calculation unit for diffusion-based learning networks] The diffusion-based learning network has one or more division normalization-type similarity calculation units. The following description will explain how the inputs to the diffusion-based learning network are connected to the division normalization-type similarity calculation units and the resulting average output values ​​of the division normalization-type similarity calculation units.

[0175] First, the input to the diffusion learning network (in the example in Figure 5, input i is given an input value x i is entered), or the following six sets whose elements are part of it, I N, I k , I m , I n , I d , I l Think about it.

[0176]

number

[0177]

number

[0178]

number

[0179]

number

[0180]

number

[0181]

number

[0182] I N is the set of inputs whose value is 1 during the learning phase. k is a set of inputs whose input values ​​in the learning phase and similarity judgment phase are 0 and 1, respectively. I m is a set of inputs whose input values ​​in the learning phase and similarity judgment phase are 1 and 0, respectively. I n is the set of inputs connected to the division normalization type similarity calculation unit. I d is the set I n and I m is the set of inputs that are contained in both I l is the set In and I k is the set of inputs that are contained in both

[0183] Now, let,N,k,m,n,d,and,l,be,set,I,, respectively. N , I k , I m , I n , I d , and I l In this case, the number of inputs whose input value is 1 in at least one of the learning phase or the similarity determination phase is N+k. In the division normalization similarity determination method, as can be seen from equation (7), only these N+k inputs affect the similarity. Therefore, focusing on these N+k inputs, we analyze the connection status of the inputs to the division normalization similarity calculation unit. Since the number of inputs connected to the division normalization similarity calculation unit is n, the number of patterns when n of the N+k inputs are connected is expressed by the following equation.

[0184]

number

[0185] Second, the number of inputs whose input values ​​are both 1 in the learning phase and the similarity determination phase is Nm. Of these, ndl are input to the division normalization type similarity calculation unit. Therefore, the number of patterns is expressed by the following formula:

[0186]

number

[0187] Third, there are m inputs whose input values ​​in the learning phase and the similarity determination phase are 1 and 0, respectively, and d of them are input to the division normalization type similarity calculation unit. Therefore, the number of patterns is expressed by the following formula:

[0188]

number

[0189] Fourth, there are k inputs whose input values ​​in the learning phase and the similarity judgment phase are 0 and 1, respectively, and l of them are input to the division normalization type similarity calculation unit. Therefore, the number of patterns is expressed by the following formula:

[0190]

number

[0191] Then, among the input patterns connected to the division normalization type similarity calculation unit, set I m , I n , and I d The probability that the number of elements of is m, n, and d, respectively, is given by the following formula:

[0192]

number

[0193] In this case, the similarity calculated by the division normalization type similarity determination method is as follows:

[0194]

number

[0195] If this value is expressed as the activity S(n,d,l) and the activation function is f(a), the output can be calculated as f(S(n,d,l)). From the above, the output of the division normalization type similarity calculation unit is expressed by the following formula.

[0196]

number

[0197] Here, C (C representing the summation range written below the symbol Σ) is a set of combinations of n, d, and l that simultaneously satisfy the following conditions, with τ being the threshold of the activation function: The number of inputs whose value is 1 in the learning phase is N, and some of them become 0 in the similarity determination phase. Since this number is m, the following inequality holds:

[0198]

number

[0199] The total number of inputs whose value in the learning phase is 1 and whose value in the similarity judgment phase is 0 is m. Some of them are connected to the division normalization type similarity calculation unit, and the number of such units is d, so the following inequality holds:

[0200]

number

[0201] The total number of inputs whose value in the learning phase is 0 and whose value in the similarity judgment phase is 1 is k. Some of them are connected to the division normalization type similarity calculation unit, and the number of such units is l, so the following inequality holds:

[0202]

number

[0203] The number of inputs connected to the division normalization type similarity calculation unit is n. Some of them are d, l, and d+l, so the following three inequalities hold:

[0204]

number

[0205]

number

[0206]

number

[0207] The total number of inputs for which the value in the learning phase is 1 and the value in the similarity determination phase is 1 is Nm. Some of these are connected to the division normalization type similarity calculation unit, and the number of such inputs is ndl, so the following inequality holds:

[0208]

number

[0209] In order for the division normalization type similarity calculation unit to be in a firing state and output a value greater than 0, its activity must exceed a threshold τ. Therefore, the following inequality holds:

[0210]

number

[0211] In the above discussion of the expected value of the output calculated by the division normalization type similarity calculation unit, the expected value was calculated with n as a constant. Now, we will calculate the expected value of the output when each input is connected to the division normalization type similarity calculation unit with a certain probability p. The inputs focused on in the discussion up to this point are inputs whose value is 1 in at least one of the learning phase and the similarity judgment phase, and the total number of such inputs is N+k. Of these, the probability that n inputs will be connected to the division normalization type similarity calculation unit is expressed by the following formula.

[0212]

number

[0213] Therefore, from equations (63) and (72), the expected value of the output of the division normalization type similarity calculation unit is expressed by the following equation.

[0214]

number

[0215] Since equation (73) represents the expected value of the output of the division normalization type similarity calculation unit, the activity of the perceptron (013 in Figure 5) that outputs the diffusion information network is the sum of the outputs of the division normalization type similarity calculation unit, and is therefore proportional to equation (73). The effect of this diffusion information network will be described later using Figures 29 to 40.

[0216] [Processing of the learning phase and similarity judgment phase of the diffusion-based learning network] The processes in the learning phase and similarity determination phase of the diffusion-based learning network will be described below with reference to FIGS.

[0217] Example 1 In Example 1, a first example of the division normalization type similarity determination method will be described. First, the learning phase process of the diffusion learning network will be described. FIG. 15 is a flowchart showing the processing in the learning phase of the division normalization type similarity calculation unit. In step S1, the division normalization type similarity calculation unit 100 (FIGS. 2 to 14) calculates the input vector x=(x1, x2, ..., x N ) T Receive.

[0218] In step S2, the division normalization type similarity calculation unit 100 calculates the synaptic weight vector w=(w1, w2, ..., w N ) T W i =x i Set as (i=1,2,…,N).

[0219] In step S3, the division normalization type similarity calculation unit 100 calculates a parameter C used in the similarity determination phase as C=||x|| 2 Calculate and set as follows. After the learning phase of FIG. 15, the operation of the similarity determination phase shown in FIG. 16 is performed.

[0220] Next, the similarity determination phase process of the diffusion-based learning network will be described. FIG. 16 is a flowchart showing the processing of the division normalization type similarity calculation unit in the similarity determination phase.

[0221] In step S11, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive.

[0222] In step S12, the division normalization type similarity calculation unit 100 calculates Y=||y|| required to calculate the similarity. 2 Calculate.

[0223] In step S13, the division normalization type similarity calculation unit 100 calculates Z=w·y required to calculate the similarity.

[0224] In step S14, the division normalization type similarity calculation unit 100 uses the calculated Y and Z as well as the parameter C calculated in step S3 of FIG. 15 to calculate the similarity s according to the following equation (74).

[0225]

number

[0226] In step S15, the division normalization type similarity calculation unit 100 inputs the calculated similarity s to the activation function f(a) to obtain an output value f(s). This output value f(s) becomes the output of the division normalization type similarity calculation unit 100. Here, the activation function may be the commonly used ReLU or a step function, or may be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial basis as described in Non-Patent Document 2. Furthermore, in these functions, functions with a threshold value of 0 may be used in place of functions with a threshold value of any other value.

[0227] <Example 2> In Example 2, a second example of the division normalization type similarity determination method will be described. In Example 2, the inner product between vectors ((w·y) in Equation (74)) and the square of the norm (C=||x|| in Equation (74)) in Example 1 are used. 2 , and ||y|| 2 An embodiment for efficiently calculating ) will now be described.

[0228] Now, the vector v=(v1,v2,…,v N ) T , and u=(u1,u2,…,u N ) T Suppose there is v i ∈{0,1}, and u i If ∈{0,1}, the inner product (u·v) is (u·v)=v1u1+v2u2+…+u N v N v i ∈{0,1}, and u i ∈{0,1}, so v i u i is v i and u i Therefore, (u·v) is equal to the logical product of v i and u i The logical product of these is added up over all i. Also, the vector v=(v1,v2,…,v N ) T The square of the norm of ||v|| 2 =v1v1+v2v2+…+v N v N and v i ∈{0,1}, so ||v|| 2 =v1+v2+…+v N Therefore, ||v|| 2 is v i is the sum of all i. <Example 2> is an example in which the above-mentioned methods of calculating the inner product between vectors and the square of the norm of a vector are applied.

[0229] First, the learning phase process of the diffusion learning network will be described. 17 is a flowchart showing the processing in the learning phase of the division normalization type similarity calculation unit. Steps that perform the same processing as in FIG. 15 are assigned the same reference numerals and their explanations will be omitted.

[0230] In step S21, the division normalization type similarity calculation unit 100 calculates the input vector x=(x1, x2, ..., x N ) T Receive.

[0231] In step S22, the division normalization type similarity calculation unit 100 calculates the synaptic weight vector w=(w1, w2, ..., w N ) T W i =x i Set as (i=1,2,…,N).

[0232] In step S23, the division normalization type similarity calculation unit 100 calculates a parameter C=||x|| used in the similarity determination phase. 2 , C=Σ N i=1 x i Calculate as follows. After the learning phase of FIG. 17, the operation of the similarity determination phase shown in FIG. 18 is performed.

[0233] Next, the similarity determination phase process of the diffusion-based learning network will be described. FIG. 18 is a flowchart showing the processing of the division normalization type similarity calculation unit in the similarity determination phase.

[0234] In step S31, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive.

[0235] In step S32, the division normalization type similarity calculation unit 100 calculates Y=||y|| required to calculate the similarity. 2 Calculate Y=Σ N i=1 y i Calculate as follows.

[0236] In step S33, the division normalization type similarity calculation unit 100 calculates Z=w·y, which is necessary to calculate the similarity. N i=1 w i ANDy i Here, w i ANDy i Ha, w i and y i represents the logical AND operation of

[0237] In step S34, the division normalization type similarity calculation unit 100 uses the calculated Y and Z as well as the parameter C calculated in step S23 of FIG. 17 to calculate the similarity s according to equation (74).

[0238] In step S35, the division normalization type similarity calculation unit 100 inputs the calculated similarity s to the activation function f(a) to obtain an output value f(s). This output value f(s) becomes the output of the division normalization type similarity calculation unit 100.

[0239] Here, the activation function may be the commonly used ReLU or a step function, or may be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial basis as described in Non-Patent Document 2. Furthermore, in these functions, functions with a threshold value of 0 may be used in place of functions with a threshold value of any other value.

[0240] Example 3 <Example 3> describes Example 3 of the division normalization type similarity determination method. In the third embodiment, a method for realizing a combination of the division normalization type similarity calculation method and a diffusion type learning network will be described. Figure 19 shows a neural network that combines the division normalization similarity calculation method and a diffusion-based learning network. The diffusion-type learning network includes one or more division normalization similarity calculation units. First, it is determined whether or not to connect inputs to each division normalization similarity calculation unit. The determination of whether or not to connect inputs is performed so that the combination of inputs to each division normalization similarity calculation unit is as different as possible. For example, for each combination of input and division normalization similarity calculation unit, the connection may be determined with a certain probability. In the case of FIG. 17, six division normalization similarity calculation units 101 to 106 (hereinafter referred to as units) are included. All or part of all inputs are connected to each of the units 101 to 106. Therefore, generally, each of the units 101 to 106 receives a different combination of inputs as inputs.

[0241] Therefore, in Example 3, the learning phase processing (FIGS. 15 and 17) of Example 1 and Example 2 is performed on only the connected components of the input vectors, synapse weight vectors, and similarity determination phase input vectors of each unit in the learning phase. This processing will be explained using FIG. 20.

[0242] FIG. 20 is a flowchart showing the process in the learning phase of the third embodiment. Of the inputs 1, 2, 3, 4, 5, and 6, only inputs 1 and 3 are connected to the unit 101 shown in FIG. In step S41, the division normalization type similarity calculation unit executes the learning phase process of the division normalization type similarity determination method. Specifically, the process is as follows. In the learning phase, the entire input vector is x=(x1,x2,x3,x4,x5,x6) T Then, the input vector x1 to unit 101 in the learning phase is x1=(x1,x3) TAs a result, the synaptic weight vector w1 becomes w1=(w1, w3) T =x1. If the constant C of the unit 101 is C1, then C1=||x1|| as in the first and second embodiments. 2 Thereafter, for units 102 to 106, synaptic weight vectors w2, w3, w4, w5, and w6 and constants C2, C3, C4, C5, and C6 are calculated in the same manner. After the learning phase of FIG. 20, the operation of the similarity determination phase shown in FIG. 21 is performed.

[0243] Next, the similarity determination phase process of the third embodiment will be described. FIG. 21 is a flowchart showing the process in the similarity determination phase of the third embodiment. In step S51, for each division normalization type similarity calculation unit, the process of the similarity determination phase of each division normalization type similarity determination method is executed, and the output value of each division normalization type similarity calculation unit i is calculated as f(s i ) is set as follows. The unit 101 will be used as a representative example. In the similarity determination phase, the entire input vector is expressed as y=(y1, y2, y3, y4, y5, y6). T Then, the input vector y1 to the unit 101 in the similarity determination phase is y1=(y1, y3) T Using these vectors, the similarity s1 of the unit 101 is calculated as in the first and second embodiments using the following formula:

[0244]

number

[0245] Thereafter, the similarity is calculated for units 102 to 106 in the same manner. Next, the output value of the unit is calculated as f(s i ) is calculated as Here, f(x) represents an activation function. The activation function may be the commonly used ReLU or a step function. Alternatively, it may be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial basis, as described in Non-Patent Document 2. Furthermore, in these functions, functions with a threshold value of 0 may be used in place of functions with a threshold value of any other value.

[0246] In step S52, the total output of all division normalization type similarity calculation units (the total value of the output calculated by each unit) S is calculated as follows:

[0247]

number

[0248] In step S53, the calculated S is input to the activation function g(·) to calculate the output value V=g(S) of the diffusion-based learning network. Here, the activation function may be the commonly used ReLU or a step function. It may also be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial-basis, as described in Non-Patent Document 2. Other activation functions may also be k-Winner-Take-All (kWTA) or Winner-Take-All (WTA), as described in Non-Patent Document 3 and Non-Patent Document 8. Furthermore, for these functions with a threshold of 0, functions with any other value as the threshold may also be used.

[0249] Example 4 <Example 4> describes Example 4 of the division normalization type similarity determination method. In the fourth embodiment, a method for realizing a combination of the division normalization type similarity calculation method and the diffusion type learning network will be described. In <Example 4>, instead of individually creating an input vector for the learning phase, a synaptic weight vector, and an input vector for the similarity judgment phase for each unit and calculating similarity, as described in <Example 3>, similarity is calculated using the input vector for the learning phase, the synaptic weight vector, and the input vector for the similarity judgment phase for the entire input.

[0250] First, it is determined whether or not to connect the inputs to each division normalization similarity calculation unit. The determination of whether or not to connect the inputs is performed so that the combinations of inputs to each division normalization similarity calculation unit are as different as possible. For example, for each combination of input and division normalization similarity calculation unit, the presence or absence of connection may be determined with a certain probability.

[0251] Second, we create a matrix that indicates which input is connected to which division normalization type similarity calculation unit. This matrix will be called the connection matrix hereafter. The element in the i-th row and j-th column of the connection matrix is ​​X ij Let X be the input signal, and let this component represent whether input i is connected to unit j or not. ij = 1, and X ij = 0, it means that input i is connected to unit j, and input i is not connected to unit j, respectively. The incidence matrix X is expressed as follows:

[0252]

number

[0253] For the following explanation, let X be the vector consisting of the elements of the jth column of the incidence matrix. j It is expressed as: Third, the input vector for the learning phase is x=(x1,x2,x3,x4,x5,x6). T In this case, let the synaptic weight vector be w and set w=x. w=(w1,w2,w3,w4,w5,w6) T is. Here, in general, two vectors v=(v1,v2,…,v N )T , and u=(u1,u2,…,u N ) T Let us denote the Hadamard product of vectors v and u as v○u, then v○u=(v1u1,v2u2,…,v N u N ) T Now, when each component of vector v and u is expressed as a binary value of 0 and 1, if we focus on each component i, the Hadamard product v i u i is v i , and u i can be considered as a logical product when considering as logical variables. Therefore, the Hadamard product process described below may be calculated as a logical product for each component. Using this Hadamard product representation, w1·y1, C1, and ||y1|| in equation (75) 2 are (w○X1)·y and C1=||x1||, respectively. 2 =||x○X1|| 2 , and ||y1|| 2 =||y○X1|| 2 Therefore, fourthly, in the similarity determination phase, the similarity s calculated by unit i is i can be calculated by:

[0254]

number

[0255] The learning phase and similarity determination phase processes based on the above are shown in FIGS. 22 and 23, respectively.

[0256] FIG. 22 is a flowchart showing the process in the learning phase of the fourth embodiment. In the learning phase, as described above, the input vector x in the learning phase is used, and the synapse weight vector w is set as w=x (step S61).

[0257] In step S62, the parameter Ci About C i =||x i || 2 =||x○X i || 2 Calculate and set as follows.

[0258] Next, the similarity determination phase process of the fourth embodiment will be described. FIG. 23 is a flowchart showing the process in the similarity determination phase of the fourth embodiment. In step S71, for each division normalization type similarity calculation unit i, the similarity s i is calculated using equation (78).

[0259] In step S72, the total sum S of the outputs of all division normalization type similarity calculation units is calculated as shown in equation (76).

[0260] In step S73, the output value V=g(S) of the diffusion-type learning network is calculated by inputting the activation function g(·) based on the obtained S. Note that steps S72 and S73 in FIG. 23 are the same as steps S52 and S53 in FIG. 21 of the third embodiment.

[0261] 22 and 23, the relationship between the "learning phase" and the "similarity determination phase" in the division normalization type similarity calculation unit i will be described. In the diffusion-type learning network 1000 of the first embodiment, a plurality of division normalization type similarity calculation units i having some or all of the inputs are connected to a plurality of inputs of the diffusion-type learning network, and further, the output of each division normalization type similarity calculation unit i is input to a perceptron. The division normalization type similarity calculation unit i receives one or more input values, and either a value L or a value H is input to each input, and the value of the i-th input in the learning phase is calculated by x i and the value of the i-th input in the similarity judgment phase is expressed as y i When the i-th input is expressed as i is assigned and the value w iis set to one of two values, L or H, and the weight value w assigned to the i-th input in the learning phase i x i In the similarity determination phase, i The number of inputs whose value is H, w i and y i The number of inputs whose values ​​are both H, y i Calculate the number of inputs whose value is H, and i and y i The number of inputs for which both have the value H is i y to the input number where H is the value i The similarity calculation is performed by dividing the value of the input by the sum of the number of inputs for which the value is H, and calculating the value as the similarity representing the degree of similarity.

[0262] In addition, in this division normalization type similarity assessment method, in the similarity assessment phase, the division normalization type similarity is calculated using the above-mentioned formula (6), which incorporates the calculation caused by a phenomenon called the shunt effect of neurons into the perceptron model.

[0263] Here, the "learning phase" corresponds to steps S1 and S2 in Fig. 15, and the "similarity determination phase" corresponds to step S3 in Fig. 15 and steps S11 to S15 in Fig. 16. That is, the "learning phase" is calculated in steps S1 and S2 in Fig. 15, and the "similarity determination phase" is calculated in step S3 in Fig. 15 and steps S11 to S15 in Fig. 16.

[0264] When formula (6) is transformed for different cases, formulas (7) to (10) are obtained. By analyzing these formulas, it can be seen that the values ​​calculated by the division normalization similarity calculation method are approximate values ​​of cosine similarity. In other words, the similarity calculated by the division normalization similarity calculation method can calculate a more accurate confirmation similarity than existing techniques. Thus, by accurately measuring the similarity between the information stored in the learning phase and the information input to the similarity determination phase using the division normalization similarity calculation method, it is possible to eliminate discrepancies in the information and the calculated degree of similarity in the prior art, and to calculate a similarity based on the degree of similarity.

[0265] [Separate memory reasoning method] The separate memory type inference method (learning inference method) will be explained. The separate memory type inference method uses multiple diffusion type learning networks and an information association network. Generally, inference in learning involves associating two pieces of information, E and F. The association between the input to the neural network, expressed as a vector, and the target value corresponds to the association between these two pieces of information, E and F, respectively. The information association network is a network for associating this information E and information F.

[0266] Fig. 24 is a diagram showing a diffusion-type learning network having a perceptron. The same components as those in Figs. 5 to 14 are assigned the same reference numerals. One diffusion-type learning network 1000 shown in Fig. 24 is called a diffusion-type learning network unit (learning network unit).

[0267] Figure 25 shows an information association network that performs inference by combining the division normalization similarity calculation method, the diffusion learning network, and the separate memory inference method. Figure 25 shows an example of a neural network that performs separate memory inference, having five diffusion learning network units and an information association network. The information association network 2000 includes a plurality of diffusion-type learning network units 1001 to 1005 (learning network units), a kWTA (k-Winner-Take-All) / WTA (Winner-Take-All) 1100, and a kWTA / WTA 1200.

[0268] The diffusion-based learning network units 1001 to 1005 each calculate a division-normalized similarity and output the similarity. kWTA / WTA 1100, 1200 are k-Winner-Take-All (k-WTA) or Winner-Take-All (WTA) described in Non-Patent Document 8. kWTA / WTA 1100 receives the similarity outputs of diffusion-based learning network units 1001-1005, and outputs the top k largest values ​​from diffusion-based learning network units 1001-1005 to perceptrons 007, 008, and 009. kWTA / WTA 1200 is also connected to perceptrons 007, 008, and 009, which include black triangles. For example, suppose that diffusion-based learning network units 1001 and 1002 output the similarity of the number "1" in an image to perceptron 007, diffusion-based learning network units 1003 and 1004 output the similarity of the number "2" in an image to perceptron 008, and diffusion-based learning network unit 1005 output the similarity of the number "3" in an image to perceptron 009. kWTA / WTA 1200 determines which of the outputs to perceptrons 007, 008, and 009 is most strongly stimulated, and determines that it is, for example, the number "2."

[0269] In the learning phase, one diffusion-based learning network unit 1001-1005 is assigned to each piece of learning data. Each piece of learning data consists of a feature vector, which represents the input value as a vector, and a label assigned to it. Of these, the feature vector is set as a synaptic weight in the assigned diffusion-based learning network unit 1001-1005 as part of the learning phase processing. This setting is the processing described as part of the learning phase processing of the diffusion-based learning network.

[0270] The labels in the training data are set as synaptic weights connecting the outputs of diffusion-based learning network units 1001-1005 in Figure 25 to perceptrons 007, 008, and 009. The network consisting of the outputs of diffusion-based learning network units 1001-1005 and perceptrons 007, 008, and 009 is the network in which synaptic weights responsible for associating information within the information association network are set. Each of perceptrons 007, 008, and 009 is associated with a single label, and the output of that perceptron represents the strength with which the associated label is inferred. These perceptrons are hereafter referred to as label strength calculation perceptrons. The information association network makes it possible to associate information represented by a single label with information represented by multiple feature vectors.

[0271] The output of the diffusion learning network units 1001 to 1005 is the output of the perceptron 013 in FIG. 24, and its activation is the sum of the outputs z1, z2, z3, z4, z5, and z6 of the previous perceptron that are input thereto. The value converted by an activation function is output from the perceptron 013. The activation function of the perceptron 013 is k-Winner-Take-All (k-WTA) or Winner-Take-All (WTA) described in Non-Patent Document 3, Non-Patent Document 6, Non-Patent Document 7, and Non-Patent Document 8. These are the outputs of the top k activations or the top activation, respectively, as V maxotherwise, the output is V min The activation function is where V max , and ,V min is a constant, and V max >V min Alternatively, as described in Non-Patent Document 9, k-WTA may be used in which the top k activity values ​​are used as output values.

[0272] As shown in Figure 25, the outputs of diffusion-based learning network units 1001 to 1005 are connected to label strength calculation perceptrons. Assume that label strength calculation perceptrons 007, 008, and 009 represent labels 1, 2, and 3, respectively. During the learning phase, the outputs of diffusion-based learning network units 1001 to 1005, whose synaptic weights are set based on the feature vectors of certain training data, form synapses with label strength calculation perceptrons 007, 008, and 009. Of these synapses, only the synaptic weights with the label strength calculation perceptron corresponding to the label of the training data are set to 1, and the other synaptic weights are set to 0. For example, learning data with labels 1 and 2 are set in diffusion-based learning network units 1001 and 1003 in Figure 25, and as a result, of the synapses that the outputs of diffusion-based learning network units 1001 and 1003 make with label strength calculation perceptrons 007, 008, and 009, the synapse weights of the synapses with 007 and 008 are set to 1, and the other synapse weights are set to 0.

[0273] Next, the operation of the inference phase will be described. The input to the information association network 2000 in Figure 25 is sent to all diffusion-based learning network units 1001-1005. Each diffusion-based learning network unit 1001-1005 calculates activation based on the similarity to the feature vector of the training data set therein. The activation function of the perceptron related to the output of the diffusion-based learning network units 1001-1005 is k-WTA or WTA, as described above. Due to this activation function, only the output values ​​of diffusion-based learning network units 1001-1005 with high activation selected by k-WTA or WTA are sent to the label strength calculation perceptrons 007, 008, and 009.

[0274] These outputs are transmitted to the label strength calculation perceptrons 007, 008, and 009 via synapses with a synaptic weight of 1, and not via synapses with a synaptic weight of 0. The transmitted outputs are added in the label strength calculation perceptrons 007, 008, and 009, and the resulting value becomes the activity of the label strength calculation perceptron. The activation function of the label strength calculation perceptron is k-WTA or WTA, as described above. This activation function outputs only the output values ​​of label strength calculation perceptrons with large activity selected by k-WTA or WTA.

[0275] <Example 5> In <Example 5>, a method for realizing learning and inference that combines a division normalization type similarity calculation method, a diffusion type learning network, and a separate memory type inference method will be described. In the learning phase, one diffusion-based learning network unit 1001 to 1005 (Fig. 25) is assigned to each learning data. Each learning data consists of a feature vector, which represents the input value as a vector, and a label assigned to it. The feature vector and label of the i-th learning data are denoted by x i , and ,l iHere, each label is identified by using integers equal to or greater than 1, in order from smallest to largest. That is, if there are five labels, the labels are identified as 1, 2, 3, 4, and 5.

[0276] Among these, the feature vectors are set as synaptic weights in the diffusion-based learning network units 1001 to 1005 in the learning phase (diffusion-based learning network unit 1001 in FIG. 25 ) as described in the learning phase processing of the diffusion-based learning network in <Example 3> or <Example 4>. That is, the diffusion-based learning network units 1001 to 1005 set synaptic weights based on input data for one of the training data.

[0277] In this case, a simple method for allocating which training data to which diffusion learning network unit 1001-1005 can be to allocate them sequentially. That is, the i-th training data can be allocated to diffusion learning network unit i. Alternatively, when new training data arrives, a random integer greater than or equal to 1 can be generated, and the training data can be allocated to diffusion learning network unit 1001-1005 having that value. This means that when the random number is i, the training data is allocated to diffusion learning network unit i. In this case, a sufficient number of diffusion learning network units i should be prepared to reduce the probability that multiple training data are assigned to one diffusion learning network unit i.

[0278] Now, in FIG. 25, we define a matrix L that represents the degree to which the outputs of the diffusion learning network units 1001 to 1005 are transmitted to the label strength calculation perceptron. The matrix L is called the label strength calculation perceptron transfer matrix. The element of the i-th row and j-th column of the label strength calculation perceptron transfer matrix L is defined as L. ij It is expressed as L ijrepresents the synaptic weight for the input connected to the label strength calculation perceptron. i and j are used to identify the label strength calculation perceptron (007, 008, 009 in Figure 25) and the diffusion learning network units 1001 to 1005, respectively. The component L ij In the learning phase, the feature vector of the jth training data is stored as a synaptic weight in the jth diffusion learning network unit. Therefore, when the label of the jth training data is i, L ij is set to 1, and for all k ≠ i, L kj is set to 0 (step S82 in FIG. 26, which will be described later).

[0279] After setting the synaptic weights in the learning phase described above, the inference phase operates as follows. When a feature vector y is input in the inference phase, each diffusion learning network unit i receives the feature vector x from the learning phase, which is the basis of the synaptic weights set there. i The output values ​​calculated based on the similarity between y and y (the output values ​​of perceptrons 001, 002, 003, 004, 005, and 006 in Figure 25) are added by the perceptron (perceptron 013 in Figure 25) that handles the output of diffusion learning network unit i to calculate the activity. This activity of diffusion learning network unit i is called u i and the vector whose components are the activations of all the diffusion learning network units is denoted as u=(u1,u2,…). T This vector is called the diffusion learning network unit activation vector.

[0280] The activation function of the perceptron (perceptron 013 in Figure 25) that handles the output of the diffusion learning network unit i is k-WTA or WTA, as described above. This activation function allows some components of the diffusion learning network unit activation vector u to pass, while the rest do not. The values ​​of the components that pass are V max The value of the component that is not passed is V min Here, V max = 1, and V min = 0. Now, let us sort u1, u2, ... from largest to smallest and call them u1 (o) ,u2 (o) , .... As the activation function of the perceptron (perceptron 013 in Figure 25) that handles the output of the diffusion learning network unit i, three sets O are used as a set of components of the diffusion learning network unit activation vector u to determine the components to pass. c , O r , and , O w is defined as follows:

[0281]

number

[0282] r in equation (79) i u i When the numbers are arranged from largest to smallest, i This is the ranking.

[0283]

number

[0284]

number

[0285] Also, O t the component value u iis the largest of all components, and if there are multiple such components, then the set whose element is the smallest i is O. c is a set of the k elements of u. r is the ratio R from the largest element among the components of u b (r) O is a set whose elements are in the range w is the sum of all components of uΣ j u j Find Σ j 1u j (o) / Σ j u j But the ratio R b (w) These sets are used to select the components of the diffusion learning network unit activation vector u whose feature vectors in the training data are close to the feature vectors input in the inference phase.

[0286] O c , O r , and , O w When using, the value k used in k-WTA is |O c |, |O r |, and |O w | Also, O t When using the WTA, the activation function becomes WTA. Any of these sets can be used in the activation function of the perceptron (Perceptron 013 in Figure 25) that handles the output of diffusion learning network unit i, or any set that can select labels of training data whose feature vectors are close to the feature vectors input in the inference phase. Hereinafter, the set used in the activation function of the perceptron (Perceptron 013 in Figure 25) that handles the output of the diffusion learning network unit, i.e., the set used to select elements of the diffusion learning network unit activation vector u, will be called the similarity top-selection set.

[0287] Diffusion learning network unit activation vector u=(u1,u2,…)T Given each element u i When i is included in the similarity upper selection set, its element u i is replaced by 1, otherwise u i is replaced with 0. The vector after this replacement is u'=(u1 ’ ,u2 ’ ,…) T and is called the diffusion learning network unit output vector. Here, when i is included in the similarity upper selection set, its element u i is replaced by 1, otherwise u i is replaced by 0, but when i is included in the similarity upper selection set, its element u i otherwise, u i may be replaced with 0.

[0288] The diffusion learning network unit output vector is u ’ =(u1 ’ ,u2 ’ ,…) T When this is the case, the activity of each label strength calculation perceptron is q=Lu ’ =(q1,q2,...). Let q be called the label strength calculation perceptron activation vector. The i-th component of this vector is the activation of label i. The activation function of the label strength calculation perceptron is k-WTA or WTA as described above. Therefore, the activation function of the label strength calculation perceptron performs the same operation as the activation function of the perceptron (perceptron 013 in Figure 25) that handles the output of the diffusion learning network unit i. However, O c , O r , and , O w In, k, R, respectively b (r) , and ,R b (w) may use different values.

[0289] Using these similarity top selection sets, the activation function is k-WTA or WTA and each element of q is processed. That is, the elements that represent the activation of the label strength calculation perceptron included in the similarity top selection set are set to 1, and the elements that represent the activation of other label strength calculation perceptrons are set to 0. The vector representation of the elements generated by this process is q ’ and called the label strength calculation perceptron output vector. In this case, the similarity top selection set is O t Then, only the output of the label strength calculation perceptron corresponding to the label with the highest activation level will be 1, and the other outputs will be 0. In this case, the label assigned to the label strength calculation perceptron with an output of 1 will be the inference result.

[0290] The above operations will be explained using a flowchart, divided into a learning phase and an inference phase. First, the processing in the learning phase will be described. 26 is a flowchart showing the processing in the learning phase of Example 5. This flowchart is an example of setting the i-th learning data using the synaptic weight of the i-th diffusion learning network unit. In the learning phase, in step S81, for each learning data i, the input feature vector x i As described in the learning phase of the diffusion learning network described in Example 3 or Example 4, a synaptic weight vector is set as the synaptic weight for the diffusion learning network unit i using the vector i. This is performed for all i.

[0291] In step S82, the label of each training data j is represented by i, and the component L of the label strength calculation perceptron transfer matrix is ij is set to 1, and for all k ≠ i, L kj is set to 0. The learning data j is assigned to the diffusion learning network unit j. Therefore, when the label of the learning data j is i, L ij is set to 1, and for all k ≠ i, Lkj is set to 0. This is done for all j.

[0292] Next, the processing in the inference phase will be described. FIG. 27 is a flowchart showing the processing in the inference phase of the fifth embodiment. When the feature vector y of the inference phase is input in step S91, y is input to all diffusion learning network units i. The diffusion learning network unit i obtains the activity u by performing the processing up to step S72 in Fig. 21 or Fig. 23 according to the processing of the similarity determination phase described in <Example 3> or <Example 4>. i In step S72 of FIG. 21 and FIG. 23, S is the activity u i This is done for all i. Here, the processing in step S73 in FIG. 21 or FIG. 23 is the processing of the activation function, and this processing corresponds to step S92 in FIG.

[0293] In step S92, the diffusion-based learning network unit activation vector u = (u1, u2, ...) T , and the similarity top selection set of the perceptron (perceptron 013 in Figure 25) that handles the output of the diffusion learning network unit, the diffusion learning network unit output vector u ’ =(u1 ’ ,u2 ’ ,…).

[0294] In step S93, the diffusion learning network unit output vector u ’ =(u1 ’ ,u2 ’ , ...), and the label strength calculation perceptron transfer matrix L are used to calculate the label strength calculation perceptron activity vector as q = Lu = (q1, q2, ...).

[0295] In step S94, the label strength calculation perceptron activation vector q and the label strength calculation perceptron similarity top selection set are used to generate the label strength calculation perceptron output vector q ’ =(q1 ’ ,q2 ’ ,…) is calculated. Here, the similarity top selection set is O t Then, only the output of the label strength calculation perceptron corresponding to the label with the highest activation level will be 1, and all other outputs will be 0. The label assigned to the label strength calculation perceptron with an output of 1 will be the inference result.

[0296] As described above, in Example 5, the separate memory inference method (learning inference method) (FIGS. 24 to 28) uses multiple diffusion-type learning networks and an information association network. In the learning phase, one diffusion-type learning network unit 1001 to 1005 (learning network unit) is assigned to each piece of training data. The network consisting of the outputs of the diffusion-type learning network units 1001 to 1005 and perceptrons forms the information association network 2000 (FIG. 25). Each perceptron is associated with one label, and the output of the perceptron represents the strength with which the associated label is inferred; it is called a label strength calculation perceptron. The information association network 2000 makes it possible to associate information represented by a single label with information represented by multiple feature vectors.

[0297] The output of diffusion learning network units 1001 to 1005 is a value obtained by adding the output of the previous perceptron and converting it using an activation function. This output is connected to the label strength calculation perceptron. During training, only the synaptic weight with the label strength calculation perceptron corresponding to the label of the training data is set to 1, and all other synaptic weights are set to 0. During inference, the input is sent to all diffusion learning networks 1000, and activation is calculated based on the similarity with the feature vector of the set training data. Only output values ​​of diffusion learning networks with high activation are sent to the label strength calculation perceptron, where they are added, and the resulting value becomes the activation of the label strength calculation perceptron. Only output values ​​of label strength calculation perceptrons with high activation are output.

[0298] This allows the division normalization-based similarity calculation method and the diffusion-based learning network to accurately measure the similarity between the information stored in the learning phase and the information input to the similarity judgment phase, and also allows accurate inference by storing the information of each piece of learning data using the separate memory-based inference method and associating multiple feature vectors for each label using the information association network. This solves the problems of similarity judgment in the prior art, such as the deterioration of similarity judgment due to the association of multiple feature vectors for each label, and the loss of learning data memory.

[0299] Example 6 In the same manner as in the <Example 5>, the <Example 6> will explain a method for realizing learning and inference by combining a division normalization type similarity calculation method, a diffusion type learning network, and a separate memory type inference method. Example 6 is an example in which labels included in two label sets are associated with feature vectors.

[0300] 28 is a diagram showing an information association network 2000A that performs inference by combining the division normalization similarity calculation method, the diffusion learning network, and the separate memory inference method. The same components as in FIG. 25 are assigned the same reference numerals. The information association network 2000A includes a plurality of diffusion-type learning network units 1001 to 1005, a k-Winner-Take-All (kWTA) / Winner-Take-All (WTA) 1100, a kWTA / WTA 1200, and a kWTA / WTA 1300.

[0301] Inter-information association network 2000A is obtained by adding label strength calculation perceptrons 011, 012, and 013, and kWTA / WTA 1300 that calculates the activation function of the label strength calculation perceptron, to diffusion-type learning network 1000 in FIG. In information association network 2000A, label strength calculation perceptrons 007, 008, and 009 each correspond to one label included in the first label set in added kWTA / WTA 1200. Also, label strength calculation perceptrons 011, 012, and 013 each correspond to one label included in the second label set in added kWTA / WTA 1300.

[0302] The operations of the label strength calculation perceptrons 007, 008, and 009 are the same as those in Example 5. The operations of the label strength calculation perceptrons 011, 012, and 013 are also the same as those of the label strength calculation perceptrons 007, 008, and 009 in Example 5. The activation functions of the label strength calculation perceptrons 007, 008, and 009 and the label strength calculation perceptrons 011, 012, and 013 are separate k-WTA or WTA.

[0303] This activation function outputs, for each label set, only the output values ​​of the label strength calculation perceptrons with high activation levels selected by k-WTA or WTA among the label strength calculation perceptrons corresponding to the labels contained therein.

[0304] As a result, when there is training data in which multiple label sets can be assigned to a common feature vector, conventional training methods using gradient descent or backpropagation methods require a training phase for each combination of feature vector and one label set. However, by standardizing weight determination, training can be performed efficiently.

[0305] [Effects of diffusion-based learning networks] The effects of the diffusion-type learning networks of <Example 1> to <Example 4> will be explained. Since the above equation (73) represents the expected value of the output of the division normalization type similarity calculation unit, the activity of the perceptron (013 in Figure 5) that outputs the diffusion information network is the sum of the outputs of the division normalization type similarity calculation unit, and therefore is proportional to equation (73). The effect of this diffusion information network will be explained using Figures 29 to 40.

[0306] Figure 29 shows the effect of the diffusion-based learning network (step function, p=0.05 and k=0), Figure 30 shows the effect of the diffusion-based learning network (step function, p=1.0 and k=0), Figure 31 shows the effect of the diffusion-based learning network (step function, p=0.05 and m=0), Figure 32 shows the effect of the diffusion-based learning network (step function, p=1.0 and m=0), Figure 33 shows the effect of the diffusion-based learning network (step function, p=0.05 and m=k), and Figure 34 shows the effect of the diffusion-based learning network (step function, p=1.0 and m=k). Figure 35 shows the effect of the diffusion-based learning network (linear function, when p=0.05 and k=0), Figure 36 shows the effect of the diffusion-based learning network (linear function, when p=1.0 and k=0), Figure 37 shows the effect of the diffusion-based learning network (linear function, when p=0.05 and m=0), Figure 38 shows the effect of the diffusion-based learning network (linear function, when p=1.0 and m=0), Figure 39 shows the effect of the diffusion-based learning network (linear function, when p=0.05 and m=k), and Figure 40 shows the effect of the diffusion-based learning network (linear function, when p=1.0 and m=k).

[0307] Figure 29 shows the effect of the diffusion-type information network when m is changed, with the activation function of the perceptron in the division normalization type similarity calculation unit 100 set to a step function, N=100, p=0.05, and k=0 in the above explanation. The activation function thresholds shown are 0.9, 0.8, and 0.7. The vertical axis in FIG. 29 represents the normalized activity of the perceptron that outputs the diffusion-based learning network (the activity of the perceptron that outputs the diffusion-based learning network divided by the number of division-normalization similarity calculation units 100, which is the value calculated using equation (73) above). The horizontal axis in FIG. 29 represents the number of inputs whose value is 1 during learning and 0 during similarity determination (the value of m). In other words, when the horizontal axis is 0, it indicates that the same input as during learning is received during similarity determination, and as the value on the horizontal axis increases, it indicates that the difference between the input during learning and the input during similarity determination increases.

[0308] As can be seen from Figure 29, as the difference in input between training and similarity judgment increases, the activity of the perceptron that outputs the diffusion-based learning network gradually decreases, indicating that the similarity between the inputs during training and similarity judgment can be accurately determined.

[0309] Figure 30 shows the effect of the diffusion-based information network when p = 1.0, as compared to Figure 29. In this case, all inputs are connected to all division normalization similarity calculation units in the same way, resulting in a situation similar to when the diffusion-based learning network is not used. The vertical and horizontal axes in Figure 30 are the same as those in Figure 29. As can be seen from Figure 30, the vertical axis is 1 from 0 to the value determined by the threshold of the activation function of the perceptron in the division normalization similarity calculation unit, and is 0 thereafter. Therefore, compared to when the diffusion-based information network is used with p < 1.0, the range in which the similarity of inputs can be determined during learning and similarity judgment is narrower, and the degree of similarity can only be determined using two values, 1 and 0, resulting in a rough judgment.

[0310] Figures 31 and 32 show the results of Figures 29 and 30, respectively, when m = 0 and the value of k is varied. The horizontal axis therefore represents the number of inputs (value of k) whose value is 0 during learning and 1 during similarity assessment. In other words, when the horizontal axis is 0, it indicates that the same input as during learning is received during similarity assessment, and as the value on the horizontal axis increases, it indicates that the difference between the input during learning and the input during similarity assessment increases. In Figures 31 and 32, as in the comparison between Figures 29 and 30, it can be seen that when a diffusion-based information network is used with p<1.0, the similarity between the inputs during learning and the input during similarity assessment can be accurately determined.

[0311] Figures 33 and 34 show the results when m = k in Figures 29 and 30 and the values ​​of m and k are changed simultaneously. When the horizontal axis is 0, it indicates that the same input as during training is received during similarity assessment, and as the value on the horizontal axis increases, the difference between the input during training and the input during similarity assessment increases. In Figures 60 and 34, as in the comparison between Figures 29 and 30, it can be seen that when a diffusion-based information network is used with p<1.0, the similarity between the inputs during training and the input during similarity assessment can be determined with high accuracy.

[0312] Figures 35 to 40 show the results of Figures 29 to 34, respectively, when the activation function of the perceptron in the division normalization similarity calculation unit is a linear function. When the activation function is a linear function, the biggest difference from the step function is the effect of the diffusion information network when p = 1.0. In other words, there is a big difference in the situation, which is essentially the same as when the diffusion information network is not used. The step function outputs 0 below the threshold and 1 above the threshold. On the other hand, a linear function outputs a value proportional to the activation level above the threshold. Therefore, as shown in Figures 30, 32, 34, 36, 38, and 40, similarity can be accurately determined above the threshold. On the other hand, similarity cannot be determined below the threshold. From the above, it can be seen that even when the activation function is a linear function, if a diffusion-type information network is used with p<1.0, the similarity of the inputs during learning and similarity judgment can be accurately determined.

[0313] [Effects of the first embodiment] As described above, the separated memory type inference method (learning and inference method) according to the first embodiment (FIGS. 24 to 28) is a similarity determination method that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron modeled on a neuron. One or more input values ​​are received, and each input value is either a value L or a value H. The i-th input value in the learning phase is calculated as x i and the i-th input value of the inference phase is expressed as y i When expressed as iis assigned and the value w i is set to either value L or value H, and the weight value w assigned to the i-th input value in the learning phase is i x i and in the inference phase, x i The number of inputs with a value of H, w i and y i The number of inputs for which both are H, y i Calculate the value of the input number for which the value of is H, and i and y i The number of inputs for which both have the value H is i y to the input number where H is the value i The value obtained by dividing the sum of the number of inputs for which the value is H is calculated as the similarity indicating the degree of similarity.

[0314] In this way, the similarity between the information stored in the learning phase and the information input to the inference phase can be measured with high accuracy using the division normalization similarity calculation method (Figs. 15-18) and the diffusion learning network 1000 (Figs. 5-14). Furthermore, accurate inference becomes possible by storing the information of individual learning data using the separate memory inference method (learning inference method) (Figs. 24-28) and associating multiple feature vectors with each label using the information association network 2000 (Fig. 25). This solves the problems of the prior art, such as the problem of similarity judgment, the problem of similarity judgment being worsened by associating multiple feature vectors with each label, and the problem of learning data being lost.

[0315] The value calculated by the division normalization similarity calculation method is an approximation of the cosine similarity. As a result, the similarity calculated by the division normalization similarity calculation method can calculate the recognition similarity more accurately than existing technologies, as described in Figures 31 to 40. This allows the division normalization similarity calculation method to accurately measure the similarity between the information memorized in the learning phase and the information input to the similarity judgment phase. As a result, it is possible to eliminate the discrepancies in the information and calculated degree of similarity of the prior art, and to calculate the similarity based on the degree of similarity. In an artificial neural network composed of perceptrons modeled after neurons, it is possible to accurately determine the similarity between the information memorized in the network and the information newly input to the network.

[0316] In the separate memory type inference method (learning inference method) according to the first embodiment (FIGS. 24 to 28), the input value L is set to 0, the value H is set to 1, and in the inference phase, x i The number of inputs for which H is the value of x i Calculate as the sum of w i and y i The number of inputs for which both are the value H is i and y i The sum of the products of, or, w i and y i Calculate the sum of the logical products of y i The number of inputs for which H is the value of y for all i is i Calculate as the sum of

[0317] In this way, accurate inference becomes possible by storing information about individual learning data using a separate storage type inference method (learning inference method) and associating multiple feature vectors with each label using the information association network 2000 (Figure 25).

[0318] In the similarity determination method (division normalization type similarity calculation method) according to the first embodiment (FIGS. 15 to 18), a plurality of similarity calculation units (division normalization type similarity calculation units 100, 101 to 106) (FIG. 19) that perform similarity calculation processing are combined, one or more of the entire input is input to each similarity calculation unit, each similarity calculation unit calculates the similarity, and the sum of the similarities calculated by all the similarity calculation units is output as the final similarity.

[0319] In this way, it is possible to realize a diffusion-type learning network that can calculate similarity more accurately than existing techniques.

[0320] In addition, in the similarity determination method (division normalization type similarity calculation method) according to the first embodiment (FIGS. 15 to 18), the calculated similarity is used as an input value to an activation function for defining the behavior of a perceptron and a neuron, and the resulting value calculated by the activation function is output as a value indicating the degree of similarity.

[0321] By doing so, the value calculated by the division normalization similarity calculation method is an approximation of the cosine similarity. Note that the value of the activation function that uses similarity as input is not the cosine similarity. This allows the division normalization similarity calculation method to accurately measure the similarity between the information stored in the learning phase and the information input to the inference phase.

[0322] In the separate memory type inference method (learning inference method) (Figs. 24 to 28) according to the first embodiment, a learning network unit (diffusion type learning network units 1001 to 1005) (Figs. 25 and 28) is provided, which is made up of a plurality of connected similarity calculation units (division normalization type similarity calculation units 100, 101 to 106) (Fig. 19) that determine similarity by a similarity determination method and perform similarity calculation processing, and the learning network unit has a number of learning data or more. A vector having inputs to the learning network unit as components is called a feature vector, and the learning data is a combination of the feature vector and a label associated with the feature vector. One learning data is stored in one learning network. When assigning a similarity to a network unit, in the learning phase, the feature vector of the training data is used to determine the weight value included in the similarity calculation unit, and in the inference phase, the similarity calculated by the training network unit based on the feature vector is used as the input value to the perceptron and the activation function that defines the operation of the neuron, the value calculated by the activation function is used as the output value of the training network unit, the output values ​​are tallied for each label included in the training data assigned to the training network unit that calculated the similarity that was the basis for the output value, and the tallied value for each label is used as the inference result.

[0323] By doing this, the similarity between the information stored in the learning phase and the information input to the similarity judgment phase can be accurately measured using the division normalization similarity calculation method and the diffusion learning network, and accurate inference is possible by storing information about individual learning data using the separate memory inference method and associating multiple feature vectors for each label using the information association network.This solves the problems of similarity judgment in the prior art, such as the problem of similarity judgment becoming worse due to the association of multiple feature vectors for each label, and the problem of memory loss of learning data.

[0324] In the inference phase, the learning network units (diffusion learning network units 1001 to 1005) (Figs. 25 and 28) use an activation function to calculate output values, which selectively outputs a relatively large similarity among the similarities calculated by multiple learning network units. For example, k-Winner-Take-All or Winner-Take-All calculations are used as the calculation to selectively output a relatively large similarity.

[0325] By doing this, only the output value of the label strength calculation perceptron with high activation selected by k-WTA or WTA is output from the label strength calculation perceptrons using the activation function. This makes it possible to associate multiple feature vectors with each label using the information association network 2000 (Figure 25), thereby achieving accurate inference.

[0326] In the inference phase, the output values ​​of the learning network units (diffusion learning network units 1001 to 1005) (Figures 25 and 28) are aggregated for each label, and a calculation is performed to selectively output relatively large aggregate values ​​among aggregate values ​​that function on multiple labels.

[0327] In this way, information represented by one label can be associated with information represented by multiple feature vectors. The information association network 2000 (FIG. 25) enables association of multiple feature vectors with each label, enabling accurate inference.

[0328] When the training data is a combination of feature vectors and labels associated with the feature vectors, and each training data is associated with labels included in multiple label sets, in the training phase, the weight values ​​included in the training network units (diffusion type training network units 1001 to 1005) (Figures 25 and 28) are determined, and in the inference phase, for each label set, the similarity calculated by the training network unit based on the feature vector is used as the input value to the activation function that defines the behavior of the perceptron and neuron, the value calculated by the activation function is used as the output value of the training network unit, the output values ​​are aggregated for each label included in the training data assigned to the training network unit that calculated the similarity that was the basis for the output value, and the aggregated value for each label is used as the inference result, thereby simultaneously learning for training data in which labels included in multiple label sets are associated with a common feature vector.

[0329] By doing this, when there is training data in which multiple label sets can be assigned to a common feature vector, conventional training methods using gradient descent, backpropagation, etc. require a training phase for each combination of a feature vector and a label set. However, by standardizing the weight determination, training can be performed efficiently.

[0330] (Second embodiment) In the second embodiment, the division normalization similarity determination method and the diffusion learning network of the first embodiment are further combined with a noise addition sensitivity characteristic improvement method. First, a noise-addition type sensitivity characteristic improvement method will be described. Generally, the sensitivity of a measuring device is expressed as the ratio of the measuring device's indicated amount to the observed value. On the other hand, the division normalization similarity determination method and the diffusion learning network described in the first embodiment can be regarded as measuring devices for measuring the similarity of data in the learning phase and the inference phase. In order to explain the characteristics of these as a measuring instrument, Figures 41 and 42 are used.

[0331] Fig. 41 shows the activity (N=100) of a perceptron that outputs a diffusion information network when using only a division normalization similarity calculation method and a diffusion learning network. Fig. 42 shows the activity (N=1000) of a perceptron that outputs a diffusion information network when using only a division normalization similarity calculation method and a diffusion learning network. In Figures 41 and 42, the difference between the learning phase and inference data increases as the horizontal axis moves to the right. The vertical axis represents the similarity calculated when using the division normalization similarity determination method and the diffusion learning network, and is the value calculated using equation (73). The activation function included in equation (73) used in Figures 41 and 42 is a sigmoid function. The sigmoid function is expressed by the following equation (82). In this equation, β and τ are a parameter representing the slope and a threshold, respectively.

[0332]

number

[0333] The parameters included in equations (73) and (82) are p = 0.05 and β = 1.0 × 10 4 , τ=0.9. The values ​​of N are 100 and 1000 in Figures 41 and 42, respectively. As shown in dashed box a in Figure 41 and dashed box b and c in Figure 42, where the perceptron activity is close to 0.0 and where the perceptron activity is close to 1.0, the slope of the curve is almost zero and is nearly horizontal.

[0334] The fact that the slope of the curve shown in Figure 41 is horizontal means that the calculated similarity does not change depending on the difference in data between the learning phase and the inference phase, resulting in poor sensitivity. Thus, when only the division normalization type similarity determination method and the diffusion type learning network of the first embodiment are used, a problem occurs in that there are some areas where the sensitivity for measuring similarity is poor (Note 1).

[0335] Furthermore, comparing Figures 41 and 42, we see that the curves differ depending on the value of N, which represents the square of the norm of the training data. For example, when the value on the horizontal axis is 0.3, the values ​​on the vertical axis are 0.302 and 0.0287 in Figures 41 and 42, respectively. Therefore, when various training data have different values ​​of N, different similarities will be output even if the difference in proportion to the training data is the same. This creates a problem in that it becomes difficult to compare the similarities between different training data with different values ​​of N (Note 2).

[0336] Furthermore, Equation (7) used in the division normalization similarity determination method of the first embodiment is an approximation of cosine similarity, which is mathematically defined, its characteristics are thoroughly analyzed, and its effectiveness has been demonstrated. However, after calculating the activity using Equation (7), the conversion is performed using an activation function, and then the result is processed using the diffusion-based learning network method of the first embodiment. This causes a problem in that the mathematically defined characteristics become unclear (Note 3).

[0337] The noise-addition type sensitivity characteristic improving method described below in the second embodiment is a technology that solves these points 1 to 3. The noise-addition type sensitivity characteristic improvement method calculates the similarity Sd expressed by equation (7) used in the division normalization type similarity determination method of the first embodiment and the diffusion type learning network, and then calculates the similarity Sg by adding noise to Sd as shown in equation (83) below.

[0338]

number

[0339] Here, when the probability density function that generates the random variable X is expressed as P(X), G is the value of the random variable randomly generated according to this probability density function. This value is generated anew each time Sg is calculated. After Sg is calculated, Sg is used instead of Sd when performing the [Division Normalization Similarity Judgment Method] and [Diffusion Learning Network] processes.

[0340] In this way, consider the expected value of the output of the division normalization similarity calculation unit when Sg is used instead of Sd. The probability that random variable X occurs in a certain division normalization similarity calculation unit is P(X)dX. If the activity and activation function when no noise is added are S(n,d,l) and f(·), respectively, as expressed in equation (73), when S is used, the output of this division normalization similarity calculation unit is f(S(n,d,l)+X). In this equation, the value G of the randomly generated random variable mentioned above is represented by X.

[0341] Now, if there are a sufficiently large number of division normalization type similarity calculation units, it can be considered that there are also a sufficient number of division normalization type similarity calculation units that have the same activity S(n,d,l). Therefore, the expected value of the output of a division normalization type similarity calculation unit whose activity is S(n,d,l) is given by equation (84).

[0342]

number

[0343] Furthermore, since the probability that the activity becomes S(n, d, l) is calculated when calculating equation (73), the expected value of the output of the division normalization type similarity calculation unit can be expressed as equation (85) using the probability that the activity becomes S(n, d, l) as follows.

[0344]

number

[0345] The characteristics of the similarity actually calculated by the division normalization type similarity calculation unit using equation (85) will be explained with reference to FIGS.

[0346] Fig. 43 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization similarity calculation method, a diffusion learning network, and a noise-addition sensitivity characteristic improvement method are used (the output changes when the number of inputs whose input value is 1 during learning and 0 during similarity determination is changed). Fig. 44 is a diagram showing the activity of a perceptron that outputs a diffusion information network when a division normalization similarity calculation method, a diffusion learning network, and a noise-addition sensitivity characteristic improvement method are used (the output changes when the number of inputs whose input value is 0 during learning and 1 during similarity determination is changed). In Figures 43 and 44, the vertical axis represents the activity of the perceptron that outputs the diffusion-type learning network, and the horizontal axis represents the rate at which the data in the inference phase differs from the data in the learning phase.

[0347] In Figures 43 and 44, a sigmoid function is used as the activation function, and the parameters included in equation (85) and equation (82) representing f(·) included in equation (85) are p = 0.05, β = 1.0 × 10 4 , τ=0.9. The values ​​of N are 25, 50, 100, and 1000. Furthermore, the probability density function P(X) in equation (85) is a Gaussian probability density function with a mean value and standard deviation of 0.01 and 0.5, respectively.

[0348] 43 and 44, the difference between the data in the learning phase and the inference phase increases as the horizontal axis moves to the right. The vertical axis represents the activity of the perceptron that outputs the diffusion-type learning network, calculated using equation (85).

[0349] As can be seen from Figures 43 and 44, the activity of the perceptron that outputs the diffusion-based learning network always has a negative slope as the value on the horizontal axis increases. This shows that by using the activity of the perceptron that outputs the diffusion-based learning network as the similarity, the problem of insufficient sensitivity for measuring similarity (Note 1) can be solved. Furthermore, Figures 43 and 44 show that when N = 100 or more, it is almost independent of N, solving the problem of difficulty in comparing similarities between different training data with different values ​​of N (Note 2).

[0350] To explain that (Note 3) is resolved, we will explain a method for expressing the degree of similarity between two sets, called Tanimoto similarity or Jaccard similarity, which is described in Non-Patent Document 10 and Non-Patent Document 11. In this specification, these similarities, which have equivalent definitions, will be abbreviated as Tanimoto similarity. Consider two sets A and B. The Tanimoto similarity S T is expressed by the following equation (86).

[0351]

number

[0352] In equation (86), |A| represents the number of elements contained in set A. Here, using the symbols used in equation (7), the Tanimoto similarity S T In this case, if we consider the two sets as the set of components whose value is 1 in the input vector w in the learning phase and the set of components whose value is 1 in the input vector y in the inference phase, then, using the notation used in equation (7), we can write |A∩B|=n 11 , |A|=n 11 +n 10 , |B|=n 11 +n 01 Substituting these into equation (86), we get the following equation (87).

[0353]

number

[0354] The number of components in w that have a value of 1, N, is N=n 11 +n 10 Therefore, this formula is transformed into n 11 =Nn 10 Substituting this into equation (87), the Tanimoto similarity S T is expressed as the following equation (88).

[0355]

number

[0356] Here, by introducing a constant C, we obtain S RT Define

[0357]

number

[0358] S in equation (89) RT Hereafter, we call the Tanimoto similarity the raised Tanimoto similarity. Now, let us define the Tanimoto similarity included in two raised Tanimoto similarities as S T (1) and S T (2) In this case, the difference in the raised Tanimoto similarity calculated from these is given by the following equation (90).

[0359]

number

[0360] From the above, we can see that the difference in raised Tanimoto similarity is a constant multiple of the difference in Tanimoto similarity. This means that when comparing the magnitude of the difference between two sets, we can use either Tanimoto similarity or raised Tanimoto similarity. Tanimoto similarity is a mathematically defined and widely applied similarity measure, and has been shown to be effective in a variety of fields.

[0361] Figure 45 is a graph comparing the activity of the perceptron that outputs the diffusion information network when the division normalization similarity calculation method, the diffusion learning network, and the noise-addition sensitivity characteristic improvement method are used (the change in output when the number of inputs whose input value is 1 during learning and 0 during similarity judgment is changed) with the raised Tanimoto similarity. Figure 46 is a graph comparing the activity of the perceptron that outputs the diffusion information network when the division normalization similarity calculation method, the diffusion learning network, and the noise-addition sensitivity characteristic improvement method are used (the change in output when the number of inputs whose input value is 0 during learning and 1 during similarity judgment is changed) with the raised Tanimoto similarity.

[0362] In the raised Tanimoto in Figures 45 and 46, the value of C in equation (89) is 0.03. In Figures 45 and 46, the raised Tanimoto similarity is expressed as Raised-Tanimoto. For comparison, the value of the above raised Tanimoto similarity is expressed as the Tanimoto similarity S T The coefficient (1-C) is calculated as (DC). Here, D is the activity of the perceptron that outputs the diffusion learning network when the horizontal axis is 0.

[0363] As can be seen from Figures 45 and 46, the gradient of the activity of the perceptron that outputs the diffusion-based learning network is always negative, which solves (Point 1). Also, even when there are different values ​​of N in the learning data, the activity of the perceptron that outputs the diffusion-based learning network is close to the value when N = 100 or more, which solves (Point 2). Furthermore, the activity of the perceptron that outputs the diffusion-based learning network is close to the value of the raised Tanimoto similarity, which solves (Point 3).

[0364] Example 7 In the seventh embodiment, a seventh example of the processing in the inference phase will be described. The learning phase of Example 7 of the second embodiment is the same as Example 1 of the first embodiment, and the process described with reference to FIG. 15 is performed. After the learning phase of FIG. 15, the operation of the inference phase shown in FIG. 47 is performed. 47 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit of the second embodiment. The same reference numerals are used to denote steps that perform the same processing as in FIG.

[0365] In step S11, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive.

[0366] In step S12, the division normalization type similarity calculation unit 100 calculates Y=||y|| required to calculate the similarity. 2 Calculate.

[0367] In step S13, the division normalization type similarity calculation unit 100 calculates Z=w·y required to calculate the similarity.

[0368] In step S14, the division normalization type similarity calculation unit 100 uses the calculated Y and Z as well as the parameter C calculated in step S3 of FIG. 15 to calculate the similarity s according to the equation (74).

[0369] After the processes of steps S11 to S14 are performed, a random variable X according to a probability density function P(X) is randomly generated, and the generated random variable is designated as G (step S81).

[0370] That is, in step S81, the division normalization type similarity calculation unit 100 generates a random variable X according to a probability density function P(X), and defines this as G.

[0371] In step S82, the division normalization type similarity calculation unit 100 inputs the calculated similarity s and G generated from the random variable X to the activation function f(a) to obtain the output value f(s+G). This output value f(s+G) becomes the output of the division normalization type similarity calculation unit 100.

[0372] The probability density function used here is not limited to a specific distribution, and may be a Gaussian distribution, a normal distribution, a Poisson distribution, a Weibull distribution, or any other distribution. Then, using this G, the similarity s calculated in step S14, and the activation function f(?), f(s+G) is calculated and this value is used as the output.

[0373] The activation function may be the commonly used ReLU or a step function. It may also be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial basis, as described in Non-Patent Document 2. Furthermore, for these functions, functions with a threshold of 0 may be replaced with functions with any other value as the threshold.

[0374] Example 8 In Example 8, an example 8 of the processing in the inference phase will be described. The learning phase of Example 8 of the second embodiment is the same as Example 2 of the first embodiment, and the process described with reference to FIG. 17 is performed. After the learning phase of FIG. 17, the operation of the inference phase shown in FIG. 48 is performed. 48 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit of the second embodiment. The same reference numerals are used to denote steps that perform the same processing as in FIG.

[0375] In step S31, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive.

[0376] In step S32, the division normalization type similarity calculation unit 100 calculates Y=||y|| required to calculate the similarity. 2 Calculate Y=Σ N i=1 y i Calculate as follows.

[0377] In step S33, the division normalization type similarity calculation unit 100 calculates Z=w·y, which is necessary to calculate the similarity. N i=1 (w i ANDy i ) where w i ANDy i Ha, w i and y i represents the logical AND operation of

[0378] In step S34, the division normalization type similarity calculation unit 100 uses the calculated Y and Z as well as the parameter C calculated in step S23 of FIG. 17 to calculate the similarity s according to equation (74).

[0379] After the processing of steps S31 to S34 is performed, a random variable according to the probability density function P(X) is calculated. is randomly generated and the generated random variable is called G.

[0380] That is, in step S91, the division normalization type similarity calculation unit 100 generates a random variable X according to a probability density function P(X), and defines this as G.

[0381] In step S92, the division normalization type similarity calculation unit 100 inputs the calculated similarity s and G generated from the random variable X to the activation function f(a) to obtain the output value f(s+G). This output value f(s+G) becomes the output of the division normalization type similarity calculation unit 100.

[0382] The probability density function used here is not limited to a specific distribution, and may be a Gaussian distribution, a normal distribution, a Poisson distribution, a Weibull distribution, or any other distribution. Then, using this G, the similarity s calculated in step S34, and the activation function f(a), f(s+G) is calculated, and this value is used as the output.

[0383] The activation function may be the commonly used ReLU or a step function. It may also be a simple linear function, a linear function with a threshold (threshold-linear), a sigmoid function, or a radial basis, as described in Non-Patent Document 2. Furthermore, for these functions, functions with a threshold of 0 may be replaced with functions with any other value as the threshold.

[0384] Example 9 In Example 9, an example 9 of the processing in the inference phase will be described. The learning phase of Example 9 of the second embodiment is the same as Example 3 of the first embodiment, and the process described with reference to FIG. 20 is performed. After the learning phase of FIG. 20, the operation of the inference phase shown in FIG. 49 is performed. 49 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit and the diffusion type learning network unit of the second embodiment. The same reference numerals are used to denote steps that perform the same processing as in FIG. 21. The division normalization similarity calculation unit performs the processes of steps S101 to S102 below, and the diffusion learning network unit performs the processes of steps S103 and S53 below. That is, steps S103 and S53 add up the outputs of the individual division normalization similarity calculation units, so the diffusion learning network unit one layer above performs the processes. The diffusion learning network unit includes multiple division normalization similarity calculation units.

[0385] In step S101, each division normalization type similarity calculation unit i generates a random variable X according to a probability density function P(X), and calculates it as G i Let's say. In step S102, each division normalization type similarity calculation unit i calculates the similarity s i and the output of each division normalized similarity calculation unit i is expressed as f(s i +G i That is, in step S102, each division normalization type similarity calculation unit i executes the process of the similarity determination phase (inference phase) of each division normalization type similarity calculation method, and calculates the output value of each division normalization type similarity calculation unit i as f(s i +G i )

[0386] In step S103, the diffusion-type learning network unit calculates the sum S=Σ of the outputs of the all-division normalization-type similarity calculation units. i f(S i +G i ) (Equation (86)) is calculated. In step S53, the output value V=g(S) of the diffusion-type learning network is calculated by inputting the obtained S into the activation function g(·).

[0387] Here, the activation function may be the commonly used ReLU or a step function. It may also be a simple linear function, a linear function with a threshold (Threshold-linear), a sigmoid function, or a Radial-basis, as described in Non-Patent Document 2. Other activation functions may also be k-Winner-Take-All (kWTA) or Winner-Take-All (WTA), as described in Non-Patent Document 3 and Non-Patent Document 8. Furthermore, for these functions, functions with a threshold of 0 may be functions with any other value as the threshold.

[0388] The following describes the differences between the similarity determination phase and the inference phase. In the first and second embodiments, there are parts that mainly explain similarity determination and parts that mainly explain inference, and it is difficult to make a strict distinction between them. For this reason, the similarity determination phase and the inference phase are considered to represent the same phase in terms of content. Here, the inference phase is used to mean the same thing as the similarity determination phase.

[0389] Example 10 In Example 10, an example 10 of the processing in the inference phase will be described. The learning phase of Example 10 of the second embodiment is the same as Example 4 of the first embodiment, and the process described with reference to FIG. 22 is performed. After the learning phase of FIG. 22, the operation of the inference phase shown in FIG. 50 is performed. 50 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit and the diffusion type learning network unit of the second embodiment. The same reference numerals are used to denote steps that perform the same processing as in FIG. 23. In step S111, each division normalization type similarity calculation unit i generates a random variable X according to a probability density function P(X), and calculates it as G i Let's say.

[0390] In step S71, each division normalization type similarity calculation unit i calculates the similarity s i is calculated using the above equation (78). In step S112, the diffusion-type learning network unit calculates the sum S=Σ of the outputs of the all-division normalization-type similarity calculation units. i f(S i +G i ) (Equation (86)) is calculated. In step S73, the diffusion-based learning network unit inputs the obtained S into the activation function g(·) to calculate the output value V=g(S) of the diffusion-based learning network.

[0391] [Effects of the second embodiment] In the similarity determination method according to the second embodiment (FIGS. 41 to 46), a predetermined noise is added to the calculated similarity to obtain a similarity, and subsequent calculations are performed using the similarity to which the noise has been added. Incidentally, FIG. 41 shows the problems that arise when no noise is added, and FIG. 43 shows the similarity calculated when noise is added.

[0392] That is, in the second embodiment, after calculating the similarity Sd represented by (1) the division normalization type similarity calculation method and (2) the processing of the [diffusion type learning network], the similarity Sg to which noise has been added is calculated, and thereafter, calculations are performed using Sg instead of Sd.

[0393] When only (1) the division normalization type similarity calculation method and (2) the [diffusion type learning network] of the first embodiment are used, there are some areas where the sensitivity for measuring similarity is poor (Note 1), it is difficult to compare the similarity between different learning data having different values ​​of N (number of inputs) (Note 2), and the mathematically defined characteristics become unclear when the above processes (1) and (2) are performed (Note 3).

[0394] In the second embodiment, by performing calculations using the similarity Sg to which noise has been added, as can be seen by comparing Figures 41 and 43 and Figures 42 and 44, the poor sensitivity for measuring similarity is partially eliminated (solution to point 1). Furthermore, as shown in Figures 45 and 46, the activity of the perceptron that outputs the diffusion-based learning network is close to the value of the raised Tanimoto similarity (solution to point 3).

[0395] As a result, in the second embodiment, the similarity between the information stored in the learning phase and the information input to the inference phase can be accurately measured using the division normalization type similarity calculation method, the diffusion type learning network, and the noise addition type sensitivity characteristic improvement method. Furthermore, the difference in information and the discrepancy in the calculated degree of similarity in the prior art are eliminated, making it possible to calculate the similarity based on the degree of similarity.

[0396] In the similarity determination method according to the second embodiment, a predetermined noise is added to the calculated similarity Sd to obtain a similarity Sg, and the final similarity calculation is performed using the similarity Sg to which the noise has been added. By doing so, the above (Points to Note 1) to (Points to Note 3) can be resolved.

[0397] In the similarity determination method according to the second embodiment, the noise is a random number that is generated randomly.

[0398] In this way, the random numbers to be generated can be easily generated, for example, by a random number generating circuit, and by using these random numbers as noise, the amount of calculation required when calculating similarity can be reduced.

[0399] (Third embodiment) The third embodiment is an example of application of a division normalization type similarity calculation method using fuzzy logic. In the first and second embodiments, a vector w=(w1, w2, w3, . . . ) representing synaptic weights set by inputs in the learning phase is used. T and the vector y=(y1,y2,y3,…) representing the input of the inference phase (similarity judgment phase). T In order to calculate the similarity of , the above-mentioned formula (6) and formula (7) have been used. In formula (6) and formula (7), it has been explained that each component of vectors w and y takes only the value of 0 or 1, and the following formula (91) is used.

[0400]

number

[0401] Here, (y·w) in equation (91) represents the inner product, and Σ i w i y i When using this formula (91), the input value can only take on the values ​​of 0 or 1. Therefore, it cannot be applied to applications where multi-level values, such as image brightness, are handled instead of two levels of light and dark, or where stepless values ​​such as real numbers are handled. To solve this problem, we will use fuzzy logic described in Non-Patent Document 13, as in Non-Patent Document 12, so that the input value can be any real number between 0 and 1. By doing so, for example, if the input value x i When is in the range from the minimum value L to the maximum value H, the value x i (x i By replacing it with (-L) / (HL), it becomes possible to convert it into a real number between 0 and 1, so the above problem can be solved using fuzzy logic.

[0402] This replacement will be explained. 0≦w i ≦1,0≦y i ≦1, and the learning input when w is determined is x=(x1,x2,x3,…) T For the components of 0≦x i ≦1, and then Σi w i y i Also Σ i w i ∧ F y i Here, w i ∧ F y i ∧ in F is the operator, and p∧ F The value of q is the smaller of p and q. More specifically, when p≧q, p ∧ F The value of q becomes q. By this substitution, equation (91) becomes equation (92).

[0403]

number

[0404] In equation (92), z i =w i ∧ F y i is. Regarding the characteristics of equation (92), we will explain the range of values ​​that equation (92) can take, the conditions under which the value of equation (92) becomes the maximum value, and how the value of equation (92) changes when it deviates from the conditions under which it becomes the maximum value. First, we will explain the range of values ​​that Equation (92) can take. The range of values ​​that the variables used in equation (92) can take is 0 ≤ w i ≦1, 0≦y i ≦1 and 0≦z i Since z is ≦1, equation (92) never takes a negative value. i When =0, the value of equation (92) is 0, so it can be seen that the value of equation (92) is 0 or more. Next, using equation (92), the maximum value becomes 1 according to equation (93).

[0405]

number

[0406] From the above discussion, it can be seen that the value of equation (92) is greater than or equal to 0 and less than or equal to 1. Secondly, the condition under which the value of formula (92) becomes the maximum value will be explained. Since the maximum value of formula (92) is 1, the following conditional formula (94) is obtained.

[0407]

number

[0408] Transforming this gives equation (95).

[0409]

number

[0410] Further transformation gives the following equation (96).

[0411]

number

[0412] In equation (96), w i -z i ≧0 and y i -z i ≧0, the condition to satisfy equation (96) is that for any i, w i =z i , and y i =z i is. Therefore, w i =y i =z i Therefore, the condition for the value of equation (92) to be maximum is that for any i, w i =y i This is when... Thirdly, we will explain how the value of equation (92) changes when the condition for the value of equation (92) to be at its maximum value deviates. In equation (92), w iis determined in the learning phase and is a constant in the similarity determination phase. Therefore, we can rewrite equation (92) as equation (97) to y k Partial differentiation is performed by

[0413]

number

[0414] First, w k <y k Considering the time, z k w k Then, equation (97) becomes the following equation (98).

[0415]

number

[0416] where w i , and ,y i If all of the values ​​of w are not zero, the denominator of the above equation is obviously positive, and the numerator of equation (98) is obviously negative. k <y k In the range of y k It can be seen that the rate decreases monotonically with increasing Next, w k ≧y k Considering the time, z k =y k Then, equation (98) becomes the following equation (99).

[0417]

number

[0418] where w k , and ,y k If all of the values ​​of w are not 0, then the denominator of equation (99) is clearly a positive value, and the numerator of equation (99) is also clearly a positive value. k≧y k In the range of y k From the above discussion, we can see that when the value of equation (92) moves away from the condition where it is at its maximum value, the value of equation (92) decreases monotonically as the value moves away.

[0419] Figure 51 is a diagram illustrating an example of similarity calculated by a division normalization type similarity calculation method using fuzzy logic. Figure 51 shows the change in similarity when y = (y1, y2) is changed when w = (w1, w2) = (0.5, 0.5). In other words, this is the calculation result of similarity when replaced with fuzzy logic when w = (w1, w2) = (0.5, 0.5). In FIG. 51, y=(y1, y2) is changed. The similarity in FIG. 51 is calculated based on equation (92). As can be seen from FIG. 51, the further y=(y1, y2) is from y=(0.5, 0.5), the lower the similarity becomes. This is the same characteristic as when similarity is calculated using equations (6) and (7), so it can be seen that the equation for calculating similarity can be replaced with equation (92).

[0420] Here, when fuzzy logic is not used, it has been explained using equations (9) and (10) that the similarity expressed by equation (7) decreases as the change in vector y from vector w increases. In the above explanation, the change in vector y from vector w is calculated by the following equations: i No w i This means a change from 0 to 1. In other words, it means that an element changes from 0 to 1, and from 1 to 0, and this has been explained as a change in similarity when it increases. When fuzzy logic is used, each element changes continuously, so partial differentiation is used to explain the change in similarity calculated in response to a change in each element using equations (98) and (99), and the change in numerical similarity is also explained in Figure 51. From the above, it can be seen that what has been explained here has the same characteristics as when similarity is calculated using equations (6) and (7), and therefore the equation for calculating similarity can be replaced with equation (92).

[0421] Example 11 In <Example 11>, we will explain the learning phase processing using a division normalization type similarity calculation method using fuzzy logic, the inference phase processing when the noise addition type sensitivity characteristic improvement method is not used, and the inference phase processing when the noise addition type sensitivity characteristic improvement method is used.

[0422] 52 is a flowchart showing the learning phase process using the division normalization type similarity calculation method with fuzzy logic. Steps that perform the same processes as those in FIG. 17 are given the same reference numerals and their explanations will be omitted.

[0423] In step S21, the division normalization type similarity calculation unit 100 calculates the input vector x=(x1, x2, ..., xN) in the learning phase. T Receive.

[0424] In step S22, the division normalization type similarity calculation unit 100 calculates the synaptic weight vector w=(w1, w2, ..., wN) T Set wi = xi (i = 1, 2, ..., N).

[0425] In step S121, the division normalization type similarity calculation unit 100 calculates a parameter C used in the similarity determination phase (inference phase) as C=Σ N i=1 x i Calculate and set as follows. After the learning phase of FIG. 52, the operations of the inference phase shown in FIGS. 53 and 54 are performed.

[0426] Next, we will describe the inference phase processing of the diffusion-based learning network. 53 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit when the noise-addition type sensitivity characteristic improvement method is not used. The same reference numerals are used for steps that perform the same processing as in FIG. 18.

[0427] In step S31, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive. In step S131, the division normalization type similarity calculation unit 100 calculates Y=Σ N i=1 y i Calculate.

[0428] In step S132, the division normalization type similarity calculation unit 100 calculates z=Σ N i=1 (w i ∧ F y i ) is calculated. In step S133, the division normalization type similarity calculation unit 100 calculates the similarity s=2Z / (C+Y).

[0429] In step S35, the division normalization type similarity calculation unit 100 inputs the calculated similarity s to the activation function f(a) to obtain the output value f(s). This output value f(s) is the output of the division normalization type similarity calculation unit 100 when the noise-addition type sensitivity characteristic improvement method is not used.

[0430] 54 is a flowchart showing the processing in the inference phase of the division normalization type similarity calculation unit when the noise addition type sensitivity characteristic improvement method is used. The same reference numerals are used to denote steps that perform the same processing as in FIG. 53.

[0431] In step S31, the division normalization type similarity calculation unit 100 calculates the input vector y=(y1, y2, ..., y N ) T Receive. In step S131, the division normalization type similarity calculation unit 100 calculates Y=Σ N i=1 y i Calculate. In step S132, the division normalization type similarity calculation unit 100 calculates z=Σ N i=1 (w i ∧ F y i ) is calculated.

[0432] In step S133, the division normalization type similarity calculation unit 100 calculates the similarity s=2Z / (C+Y). In step S134, the division normalization type similarity calculation unit 100 generates a random variable X according to the probability density function P(X), and defines this as G.

[0433] In step S135, the division normalization type similarity calculation unit 100 calculates f(s+G) as an output value. This output value f(s+G) becomes the output of the division normalization type similarity calculation unit 100 when the noise-addition type sensitivity characteristic improvement method is used.

[0434] [Effects of the third embodiment] The separated memory type inference method (learning inference method) according to the third embodiment (FIGS. 51 to 54) is a similarity determination method that calculates the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron modeled on a neuron. One or more input values ​​are received, and any value between input value L and value H is input. The value of the i-th input in the learning phase is calculated by x i and the value of the i-th input in the similarity judgment phase is expressed as y i When expressed as i is assigned and the value w i is set to any value between L and H, and the weight value w assigned to the i-th input in the learning phase is i x i and in the inference phase, i The sum of the values ​​of w i and y i The sum of the smaller values ​​of y iCalculate the sum of the three values ​​of w i and y i The value that represents the sum of the smaller values ​​of i The value that represents the sum of the values ​​of y i The value obtained by dividing the input value by the sum of the values ​​of the inputs is calculated as the similarity indicating the degree of similarity. That is, the separate memory type inference method (learning inference method) according to the third embodiment (FIGS. 24 to 28) replaces the input value with a value that can take any real number between 0 and 1 using fuzzy logic.

[0435] By doing this, the input values ​​are not limited to 0 or 1, for example, the brightness of an image is not limited to two levels of light and dark, but can be multi-level values, and the input values ​​can be applied to applications that handle continuous values ​​such as real numbers.

[0436] [Hardware configuration] The division normalization type similarity calculation unit 100 (FIGS. 1 to 14) according to the first to third embodiments is realized by a computer 900 having a configuration as shown in FIG. 55, for example. FIG. 55 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the division normalization type similarity calculation unit 100. The computer 900 includes a CPU 901, a RAM 902, a ROM 903, a HDD 904, an accelerator 905, an input / output interface (I / F) 906, a media interface (I / F) 907, and a communication interface (I / F) 908. The accelerator 905 corresponds to the division normalization type similarity calculation unit 100 shown in FIGS.

[0437] The accelerator 905 is a division normalization type similarity calculation unit 100 (FIGS. 1 to 14) that processes at least one of data from the communication I / F 908 and data from the RAM 902 at high speed. Note that the accelerator 905 may be of a type (look-aside type) that returns the execution result to the CPU 901 or RAM 902 after executing processing from the CPU 901 or RAM 902. On the other hand, the accelerator 905 may be of a type (in-line type) that performs processing between the communication I / F 908 and the CPU 901 or RAM 902.

[0438] The accelerator 905 is connected to an external device 915 via a communication I / F 908. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from and to a recording medium 917.

[0439] The CPU 901 operates based on a program stored in the ROM 903 or the HDD 904, and controls each part of the division normalization type similarity calculation unit 100 shown in Figures 1 to 14 by executing a program (also called an application or its abbreviation, "app") read into the RAM 902. This program can also be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM. The ROM 903 stores a boot program executed by the CPU 901 when the computer 900 is started, programs that depend on the hardware of the computer 900, and the like.

[0440] The CPU 901 controls an input / output device 916, which is made up of input units such as a mouse and a keyboard, and output units such as a display and a printer, via an input / output I / F 906. The CPU 901 acquires data from the input / output device 916 via the input / output I / F 906, and outputs generated data to the input / output device 916. Note that a GPU (Graphics Processing Unit) or the like may be used as a processor together with the CPU 901.

[0441] The HDD 904 stores programs executed by the CPU 901 and data used by the programs. The communication I / F 908 receives data from other devices via a communication network (e.g., a network) and outputs the data to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network.

[0442] The media I / F 907 reads a program or data stored in the recording medium 917 and outputs it to the CPU 901 via the RAM 902. The CPU 901 loads a program related to a target process from the recording medium 917 onto the RAM 902 via the media I / F 907, and executes the loaded program. The recording medium 917 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disc), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0443] For example, when the computer 900 functions as the division normalization similarity calculation unit 100 configured as one device according to the first embodiment, the CPU 901 of the computer 900 executes a program loaded onto the RAM 902 to realize the functions of the division normalization similarity calculation unit 100. The HDD 904 also stores data in the RAM 902. The CPU 901 reads and executes a program related to a target process from a recording medium 917. Alternatively, the CPU 901 may read a program related to a target process from another device via a communication network.

[0444] The present invention is not limited to the first to third embodiments described above, and includes other modifications and applications within the scope of the claims. For example, instead of a logic gate as a multiplication circuit, a look-up table (LUT) may be used. The LUT is a basic component of a field programmable gate array (FPGA), which is an accelerator, and is highly compatible with FPGA synthesis and easy to implement using an FPGA. The accelerator may also be a graphics processing unit (GPU) / application specific integrated circuit (ASIC), etc.

[0445] Furthermore, the above-described first to third embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, a portion of the configuration of any of the first to third embodiments may be replaced with the configuration of another of the first to third embodiments, and the configuration of any of the first to third embodiments may be added to the configuration of another of the first embodiments. Furthermore, the first to third embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These first to third embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the claims and their equivalents.

[0446] Furthermore, among the processes described in the first to third embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0447] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented by software that causes a processor to interpret and execute programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions may be stored in a memory, a recording device such as a hard disk or a solid-state drive (SSD), or a recording medium such as an integrated circuit (IC) card, a secure digital (SD) card, or an optical disc.

[0448] In the first to third embodiments, the names "division normalization type similarity determination method" and "learning inference method" are used for the sake of convenience, but these may also be called "similarity calculation method," "inference method," "neural network program," etc. Furthermore, the learning network unit may also be a diffusion type learning network unit circuit device, an information association network, etc. [Explanation of symbols]

[0449] 100, 101 to 106 Division normalization type similarity calculation unit (similarity calculation unit) 1000 Diffusion Learning Network 1001~1005 Diffusion Learning Network Unit (Learning Network Unit) 2000,2000A Information Association Network (Learning Network Unit)

Claims

1. A similarity determination method for calculating the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron modeled on a neuron, comprising: Accepts one or more input values, Each input value is input with either a value L or a value H, The i-th input value of the learning phase is x ｉ and The i-th input value of the inference phase is expressed as y ｉ When expressed as The i-th input value is w ｉ is assigned, Value w ｉ is set to either a value L or a value H, The weight value w assigned to the i-th input value in the learning phase ｉ x ｉ Set it to In the inference phase, x ｉ the number of inputs with a value of H, w ｉ and y ｉ The number of inputs for which both are H, y ｉ Calculate the number of inputs for which the value of w ｉ and y ｉ The number of inputs for which both are the value H is ｉ y to the input number whose value is H ｉ The value obtained by dividing the value of H by the number of inputs for which H is the value is calculated as the similarity indicating the degree of similarity. A similarity determination method comprising:

2. A similarity determination method for calculating the degree of similarity between an input in a learning phase and an input in an inference phase using a perceptron modeled on a neuron, comprising: Accepts one or more input values, Each input value is input with any value between value L and value H. The i-th input value of the learning phase is x ｉ and The i-th input value of the inference phase is expressed as y ｉ When expressed as i-th input value w ｉ is assigned, Value w ｉ is set to an arbitrary value between the value L and the value H, The weight value w assigned to the i-th input value in the learning phase ｉ x ｉ Set it to In the inference phase, w ｉ The sum of the values ​​of w ｉ and y ｉ The sum of the smaller values ​​of y ｉ Calculate the three values ​​of the sum of the values ​​of w ｉ and y ｉ The value representing the sum of the smaller values ​​of ｉ The value representing the sum of the values ​​of y ｉ The value obtained by dividing the sum of the values ​​of the items by the sum of the values ​​of the items is calculated as the similarity. A similarity determination method comprising:

3. Let the input value L be 0 and the value H be 1. In the inference phase, x ｉ The number of inputs for which H is the value of x for all input values ｉ Calculate as the sum of w ｉ and y ｉ The number of inputs for which both are the value H is defined as w ｉ and y ｉ The sum of the products of, or, w ｉ and y ｉ Calculate it as the sum of the logical products of y ｉ The number of inputs for which H is the value of y for all i is ｉ Calculate as the sum of 3. The similarity determination method according to claim 1 or 2.

4. A similarity determination method characterized in that a plurality of similarity calculation units that determine similarity and perform similarity calculation processing using the similarity determination method described in claim 1 or claim 2 are combined, one or more of the entire input is used as input to each of the similarity calculation units, each of the similarity calculation units calculates the similarity, and the sum of the similarities calculated by all of the similarity calculation units is output as the final similarity.

5. A predetermined noise is added to the calculated similarity to obtain a similarity, and the final similarity is calculated using the similarity to which the noise has been added.

3. The similarity determination method according to claim 1 or 2.

6. The noise is a randomly generated number.

6. The similarity determination method according to claim 5.

7. Uses fuzzy logic to replace input values ​​with values ​​that can take any real number between 0 and 1 3. The similarity determination method according to claim 1 or 2.

8. In the substitution of input values ​​using fuzzy logic, In the inference phase, w ｉ The sum of the values ​​of w ｉ and y ｉ The sum of the smaller values ​​of y ｉ Calculate the three values ​​of the sum of the values ​​of w ｉ and y ｉ The value representing the sum of the smaller values ​​of ｉ The value representing the sum of the values ​​of y ｉ The value obtained by dividing the sum of the values ​​of the two is calculated as the similarity, which indicates the degree of similarity, and is output.

8. The similarity determination method according to claim 7.

9. The method for determining similarity according to claim 1 or 2 has a learning network unit, each of which is made up of a plurality of connected similarity calculation units that determine similarity and perform similarity calculation processing, the number of which is equal to or greater than the number of pieces of training data. A vector having inputs to the learning network unit as components is called a feature vector. The training data is a combination of the feature vector and a label associated with the feature vector. When one piece of training data is assigned to one of the learning network units, In the learning phase, a weight value included in the similarity calculation unit is determined using a feature vector of learning data; In the inference phase, the similarity calculated by the similarity calculation unit based on the feature vector is used as an input value to a perceptron and an activation function for defining the operation of a neuron; The value calculated by the activation function is used as the output value of the similarity calculation unit; aggregating the output values ​​for each label included in the learning data assigned to the similarity calculation unit that calculated the similarity that served as the basis for the output values; The aggregated value for each label is used as the inference result. A learning and inference method characterized by:

10. The learning inference method according to claim 9, characterized in that in the inference phase, the learning network unit selectively outputs a relatively large similarity among the similarities calculated by multiple learning network units as an activation function used when calculating an output value by the learning network unit.

11. The learning inference method according to claim 9, characterized in that in the inference phase, a calculation is performed to selectively output relatively large aggregated values ​​obtained by aggregating the output values ​​of the learning network unit for each label, with respect to the aggregated values ​​that function on multiple labels.

12. When the training data is a combination of a feature vector and a label associated with the feature vector, and each of the training data is associated with a label included in a plurality of label sets, During the learning phase, determining the values ​​of the weights included in the learning network units; In the inference phase, for each label set, the similarity calculated by the learning network unit based on the feature vector is used as an input value to a perceptron and an activation function for defining the operation of a neuron; The value calculated by the activation function is set as the output value of the learning network unit; aggregating the output values ​​for each label included in the learning data assigned to the learning network unit that calculated the similarity that served as the basis for the output values; By using the aggregated values ​​for each label as the inference result, learning can be performed simultaneously on training data in which labels included in multiple label sets are associated with a common feature vector. The learning and inference method according to claim 11 .

13. A computer as a similarity calculation unit that accepts a part or all of a plurality of inputs, a step of receiving one or more input values ​​of either the value L or the value H; The i-th input value of the learning phase is x ｉ and The i-th input value of the inference phase is expressed as y ｉ When expressed as The i-th input value is w ｉ is assigned, Value w ｉ a procedure of setting either a value L or a value H to The weight value w assigned to the i-th input value in the learning phase ｉ x ｉ Steps to set up, In the inference phase, x ｉ the number of inputs with a value of H, w ｉ and y ｉ The number of inputs for which both are H, y ｉ a procedure for calculating the number of inputs for which the value of w ｉ and y ｉ The number of inputs for which both are the value H is ｉ y to the input number whose value is H ｉ a procedure of calculating a value obtained by dividing the sum of the number of inputs for which the value is H as a similarity indicating the degree of similarity; A neural network execution program for executing the above.

Images