Information processing device, information processing method and storage medium

Abstract

The embodiment relates to an information processing device, an information processing method, and a storage medium. The information processing device comprises: a sequence prediction unit, which uses a prediction model that outputs a sequence of L classification distributions, i.e., a classification distribution column, for one input data to obtain a first classification distribution column corresponding to a first input data, and a second classification distribution column corresponding to a second input data near the first input data; a distance calculation unit, which calculates the inter-distribution distance between the i-th classification distribution in the first classification distribution column and the i-th classification distribution in the second classification distribution column for i=1 to L respectively; a merging unit, which calculates the sum of the L inter-distribution distances; and an optimization unit, which updates the parameters of the prediction model in a manner that reduces the sum.

Description

Technical Field

[0001] Embodiments relate to an information processing device, an information processing method, and a storage medium. Background Art

[0002] In order to improve the generalization performance of the prediction model and semi-supervised learning, the following method is known: the parameters of the prediction model are updated in a way that minimizes the distance between the distributions of the posterior probability distribution at the learning data and its nearby points, thereby smoothing the posterior probability distribution. The posterior probability distribution refers to the probability distribution of the output with conditions attached to a certain input. The state in which the posterior probability distribution is smooth means that when two data are close in the data space, the posterior probabilities corresponding to them are also similar.

[0003] However, when processing sequence data, the computational load of the distance between distributions becomes high and the processing requires a lot of time. Therefore, for the prediction model that processes sequence data, it is required to be able to smooth the posterior probability distribution with less computational effort, so as to achieve improved generalization performance and semi-supervised learning. Summary of the invention

[0004] The problem to be solved by the present embodiment is to provide an information processing device, an information processing method, and a program that can smooth the posterior probability distribution of a prediction model that processes sequence data with a small amount of calculation.

[0005] The information processing device of the embodiment includes a sequence prediction unit, a distance calculation unit, a merging unit, and an optimization unit. The sequence prediction unit uses a prediction model that outputs a sequence of L (L is a natural number greater than 2) categorical distributions (Categorical distribution) for one input data, that is, a categorical distribution column, to obtain a first categorical distribution column corresponding to the first input data, and a second categorical distribution column corresponding to the second input data near the first input data. The distance calculation unit calculates the inter-distribution distance between the i-th categorical distribution in the first categorical distribution column and the i-th categorical distribution in the second categorical distribution column for i=1~L respectively. The merging unit calculates the sum of the L inter-distribution distances. The optimization unit updates the parameters of the prediction model in a way that reduces the sum. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 This is a diagram illustrating the smoothing of the posterior probability distribution.

[0007] Figure 2 This is a diagram for explaining the effect of smoothing the posterior probability distribution.

[0008] Figure 3This is a diagram for explaining the outline of character string recognition.

[0009] Figure 4 This is a block diagram showing an example of the functional configuration of an information processing device according to an embodiment.

[0010] Figure 5 This is a flowchart showing the procedure for smoothing the posterior probability distribution of the prediction model.

[0011] Figure 6 This is a block diagram showing a hardware configuration example of an information processing device according to an embodiment. DETAILED DESCRIPTION

[0012] Hereinafter, the information processing device, the information processing method, and the program according to the embodiments will be described in detail with reference to the drawings.

[0013] <Overview of implementation>

[0014] The present embodiment relates to a technique for improving the generalization performance of a prediction model and enabling semi-supervised learning by smoothing the posterior probability distribution, and in particular, for smoothing the posterior probability distribution of a prediction model that processes sequence data with less computational effort. A prediction model refers to a set of parameters and a calculation method used to predict a label for input data, and the parameter set is updated in a manner that statistically establishes a relationship between a plurality of data and a set of labels in advance, thereby generating and learning.

[0015] As methods for generating and learning prediction models, supervised learning, semi-supervised learning, and the like are known. Supervised learning is a learning method that assigns category labels to all data used in learning (learning data), and semi-supervised learning is a learning method that mixes data with category labels (labeled data) and data without category labels (unlabeled data) in the learning data. Virtual Adversarial Training described in Non-Patent Document 1 is a method for smoothing the posterior probability distribution by updating the parameters of the prediction model in a way that minimizes the distance between the distributions of the posterior probability distribution at the learning data and its nearby points.

[0016] Reference Figure 1 The smoothing of the posterior probability distribution based on Virtual Adversarial Training described in Non-Patent Document 1 is described. Figure 1In Virtual Adversarial Training, the posterior probability distribution p(y|x) of the prediction model is virtually represented by one dimension in the vertical direction, and the data space (input data space) of the input data x is virtually represented by one dimension in the horizontal direction. n becomes smooth near , and the calculation will learn the data x n The posterior probability distribution p(·|x) of the prediction model when given as input n ) and the learning data x n The nearby data x n The posterior probability distribution p(·|x n +r) between the distribution distance Dist(p(·|x n )||p(·|x n +r)), and make the distance between the distributions Dist(p(·|x n )||p(·|x n +r)) is minimized to update the parameters of the prediction model.

[0017] In Virtual Adversarial Training, in order to smooth the posterior probability distribution of the prediction model, there is no need to learn the data x n That is, Virtual Adversarial Training is able to use unlabeled data as learning data x n Semi-supervised learning.

[0018] Figure 2 This is a diagram that illustrates the effect of smoothing the posterior probability distribution. Here, in order to simplify the explanation, two examples of classification are given: the black □ category and the black △ category. In addition, in the input data space (with Figure 1 Same, expressed in one dimension along the horizontal axis), for example Figure 2 (a) shows the presence of learning data. At this time, if a prediction model is generated by supervised learning using only labeled data, the position where the posterior probability distribution of the prediction model changes (recognition boundary) may be as follows: Figure 2 (b) is a position that is biased toward the labeled data side to which the black □ category label is assigned, or vice versa. Figure 2 As shown in (c), the position is biased toward the labeled data side to which the category label of the black triangle is assigned.

[0019] In contrast, when unlabeled data is included in the learning data and the posterior probability distribution is smoothed based on Virtual Adversarial Training, the posterior probability distribution of the prediction model is smoothed near each unlabeled data, such as Figure 2 As shown in (d), the prediction model is optimized in such a way that the posterior probability distribution in the space where the learning data is sparse in the input data space changes significantly. This makes it possible to obtain a prediction model that is robust to noise and has high generalization performance.

[0020] Virtual Adversarial Training is an effective method for improving the generalization performance of prediction models in this way. However, when using sequence data as the object, the high computational load of the posterior probability distribution and the distance between its distributions becomes a problem. That is, in the problem of predicting a single label, the posterior probability distribution is the number of categories N. c It is not difficult to find the posterior probability distribution and the distance between their distributions. However, in problems such as string recognition and speech recognition that predict label sequences, the posterior probability distribution becomes the categorical distribution of the combination of all label columns. The combination of label columns increases exponentially with respect to the length of the sequence, so the calculation of the posterior probability distribution and the distance between their distributions becomes huge.

[0021] In addition, the classification distribution is a probability distribution represented by the following formula (1).

[0022]

[0023] Where y={y0, y1, ···, y Nc－1} is a one-hot N where only one element is "1" and the rest are "0" c dimensional vector, π i Satisfy Σ i＜Nc π i =1, π i Parameters ≥ 0. N c is the number of categories. Categorical distribution is also called category distribution.

[0024] Figure 3 This is a diagram that explains the outline of string recognition. In string recognition, Figure 3As shown in FIG. 1 , a string image is input as input data x to the prediction model. The prediction model extracts L feature vectors from the input string image, calculates the classification distribution of each of the L symbols including blank symbols and continuous symbols, and outputs the sequence. Then, the sequence of classification distribution of each symbol is converted into the posterior probability distribution of the string by the CTC (Connectionist Temporal Classification) decoder described later, and the string is predicted by maximum likelihood estimation.

[0025] Here, when the prediction model outputs the number of categories N containing blanks c The sequence p of the categorical distribution c 1.···、p c L When the posterior probability distribution of string w is w (w) can be calculated as shown in the following formula (2).

[0026]

[0027] Where S = s1, ···, s L is a label column containing blanks, p c i (k) is the categorical distribution p c i The probability of label k (0≤k<Nc) in the input is, and B is the CTC decoder. The CTC decoder converts the input label sequence of length L into a label sequence of any length less than L, removing blanks in the input label sequence and removing continuous labels. For example, "-" is used as a blank label, B("-aa--b-b") = "abb". In addition, for example, when L = 3, B －1 ("ab")={"aab", "abb", "-ab", "a-b", "ab-"}.

[0028] Here, given two categorical distribution sequences p c 1. ···p c L With q c 1. ···, q c L When, with p c 1. ···p c L The corresponding posterior probability distribution p of the string w and q c 1. ···, q c L The corresponding posterior probability distribution q of the string w The KL divergence D between KL(p w ||q w ) can be obtained as shown in the following formula (3). In addition, KL divergence is an indicator known as a distance between probability distributions.

[0029]

[0030] D w represents the set of label sequences converted from the label sequence of length L including blanks by the CTC decoder (B above). w The number of elements is O(N c L ), so the computational cost is very high.

[0031] Therefore, in this embodiment, the posterior probability distribution of the attention label column is represented by the product and sum of multiple small-scale classification distributions. The sum of the distances between these small-scale classification distributions suppresses the upper limit of the distance between the posterior probability distributions of the label column, so by minimizing this value, the approximate minimization (optimization) of the distance between the distributions of the true posterior probability distribution can be achieved. The distance between the small-scale classification distributions can be calculated at low cost, so as a whole, the posterior probability distribution can be approximately smoothed at low cost.

[0032] That is, in this embodiment, the following formula (4) is minimized.

[0033] ···(4)

[0034] In addition, the following formula (5) holds.

[0035]

[0036] This is because the following formula (6) usually holds true.

[0037]

[0038] In order to find D KL (p w ||q w ), O(N c L ) times of arithmetic and logarithmic operations, for which the following formula (7)

[0039]

[0040] Can be achieved through O(N c ×L) times of arithmetic and logarithmic operations to find the answer.

[0041] As described above, in this embodiment, when the posterior probability distribution of the prediction model of the prediction label column is smoothed, the parameters of the prediction model are not updated in a manner that minimizes the distance between the distributions of the posterior probability distribution of the label column, but are updated in a manner that minimizes the sum of the distances between the distributions of multiple small-scale classification distributions that are components of the posterior probability distribution. In this way, the posterior probability distribution can be smoothed at high speed and low computational complexity, and the generalization performance of the prediction model can be improved and semi-supervised learning can be achieved.

[0042] <First embodiment>

[0043] Figure 4 1 is a block diagram showing an example of the functional configuration of the information processing device of the present embodiment. The information processing device of the present embodiment improves the generalization performance of the prediction model 10 by smoothing the posterior probability distribution of the prediction model 10 near the learning data in a recognition system that uses the prediction model 10 to recognize sequence data. The information processing device is a functional component for smoothing the posterior probability distribution of the prediction model 10, such as Figure 4 As shown, the system includes an input receiving unit 1, a sequence prediction unit 2, a distance calculation unit 3, a merging unit 4, and an optimization unit 5.

[0044] The input accepting unit 1 accepts input of any learning data. The learning data may be unlabeled data. The learning data accepted by the input accepting unit 1 is referred to as "first input data". In the case of character string recognition, the first input data is a character string image, and in the case of speech recognition, the first input data is speech data.

[0045] The sequence prediction unit 2 uses the prediction model 10 to obtain a sequence of classification distributions corresponding to the input data. The prediction model 10 outputs a sequence of L (L is a natural number greater than 2) classification distributions, i.e., a classification distribution column, for one input data. L varies depending on the length of the input data. The length of the input data refers to, for example, the image size (number of pixels) in the direction of the character string in the case of a character string image, and time in the case of speech recognition. The sequence prediction unit 2 receives the first input data from the input acceptance unit 1, inputs the first input data to the prediction model 10, and obtains a first classification distribution column corresponding to the first input data. In addition, the sequence prediction unit 2 generates data near the first input data, that is, data close to the first input data in the input data space (referred to as "second input data"), and inputs the second input data to the prediction model 10 to obtain a second classification distribution column corresponding to the second input data. In addition, the length of the second input data is equal to that of the first input data, and therefore, the number of corresponding classification distributions is L.

[0046] The distance calculation unit 3 calculates the inter-distribution distance between the i-th classification distribution in the first classification distribution sequence and the i-th classification distribution in the second classification distribution sequence for i=1 to L. That is, the distance calculation unit 3 calculates the number of elements in the sequence (sequence length), that is, the inter-distribution distance between L classification distributions that have the same order (position) in the sequence between the first classification distribution sequence and the second classification distribution sequence.

[0047] The merging unit 4 calculates the sum of the L inter-distribution distances calculated by the distance calculation unit 3 , that is, the sum of the inter-distribution distances between the class distributions having the same order (position) in the sequence between the first class distribution sequence and the second class distribution sequence.

[0048] The optimization unit 5 updates the parameters of the prediction model 10 in such a way that the sum of the L distribution distances calculated by the merging unit 4, that is, the sum of the distribution distances between the classification distributions having the same order (position) in the sequence between the first classification distribution column and the second classification distribution column, becomes smaller.

[0049] Hereinafter, the information processing device of the present embodiment is assumed to be applied to an example of a character string recognition system that predicts a corresponding character string using a character string image as input, and the specific situation of the processing for smoothing the posterior probability distribution of the prediction model 10 is further described in detail. In addition, as described later, by changing the feature vector extracted from the input data according to the recognition object, it can also be easily applied to other recognition systems for sequence data such as speech recognition.

[0050] The prediction model 10 extracts L feature vectors corresponding to a specific area from the input image x. L is determined by the length of the input image x in the string direction. In the case of application to speech recognition, these feature vectors can be replaced by acoustic feature vectors extracted from speech in a specific interval. The prediction model 10 of parameter θ regards these L feature vectors as sequence data and uses a recursive neural network to output a sequence of L categorical distributions, namely, a categorical distribution sequence p. c 1(·|X,θ),···,p c L (·|X,θ).

[0051] A recurrent neural network is a neural network that has the function of outputting a predicted value for one input and also has the function of maintaining the internal state for the next prediction. When making predictions for the second and subsequent times, in addition to the original input, the internal state at the time of the previous prediction is further considered to make predictions. In this embodiment, the prediction model 10 is constructed using a recurrent neural network, which includes a classification distribution column p in the output. c 1(·|X,θ),···,p c LThe adjacent classification distributions in (·|X, θ) are recursively combined to influence each other.

[0052] Use the classification distribution column p output by prediction model 10 c 1(·|X,θ),···,p c L (·|X, θ), the likelihood of the character string w corresponding to the input image x can be calculated by the following formula (8).

[0053]

[0054] When updating the parameter θ of the prediction model 10 by supervised learning, for all the learning data (labeled data), the negative log-likelihood Loss of the correct string is used. sl is the loss function, and the parameter θ is updated in sequence in a way that reduces the loss function. When smoothing the posterior probability distribution of the prediction model 10, in addition to the above-mentioned Loss sl In addition, we also use the learning data x n (First input data x1) and its nearby points x n +r (second input data x2) updates the parameter θ of the prediction model 10 in a way that the distance between the distributions of the posterior probability distribution becomes smaller. n and nearby points x n The sum of the distances between the posterior probability distributions at +r is set to Loss smooth , then finally to make Loss sl With Loss smooth The parameters θ of the prediction model 10 are updated in a manner that minimizes these two factors.

[0055] In order to make Loss sl With Loss smooth To minimize these two aspects, the parameter θ can be updated alternately, or the parameter θ can be updated using the following total value as a new loss function.

[0056] Loss=Loss sl +λLoss smooth

[0057] Here, λ is a predetermined hyperparameter.

[0058] Below, it is assumed that Loss sl The update of the parameter θ that minimizes the loss smooth The update of the minimized parameter θ refers to Figure 5 The processing procedure of the information processing device of this embodiment is described. Figure 51 is a flowchart showing the order in which the information processing device of this embodiment smoothes the posterior probability distribution of the prediction model 10. Figure 5 The series of processing shown in the flowchart is started when the input accepting unit 1 accepts input of learning data (first input data).

[0059] When to start Figure 5 In the series of processing shown in the flowchart, first, the sequence prediction unit 2 receives the first input data x1 from the input receiving unit 1 (step S101), and generates the second input data x2 based on the first input data (step S102). Here, the second input data x2 uses a random vector d of the same dimension as the first input data x1 and a predetermined hyperparameter ε, and is determined by

[0060] x2=x1+ε×d / |d|

[0061] Find out.

[0062] Next, the sequence prediction unit 2 inputs the first input data x1 into the prediction model 10, and obtains a sequence of L classification distributions corresponding to the first input data x1, that is, a first classification distribution sequence p c 1(·|x1,θ),···,p c L (·|x1, θ), and input the second input data x2 into the prediction model 10, and obtain the sequence of L classification distributions corresponding to the second input data x2, that is, the second classification distribution sequence p c 1(·|x2,θ),···,p c L (·|x2, θ) (step S103).

[0063] Next, the distance calculation unit 3 calculates the first classification distribution sequence p c 1(·|x1,θ),···,p c L The i-th categorical distribution p in (·|x1,θ) c i (·|x1,θ) and the second classification distribution column p c 1(·|x2,θ),···,p c L The i-th categorical distribution p in (·|x2,θ) c i Here, when the KL divergence is calculated as the inter-distribution distance between the class distributions, the inter-distribution distance D KL (p c i (·|x1,θ)||p ci (·|x2, θ)) can be obtained by the following formula (9).

[0064]

[0065] Where Nc is the number of categories including blanks.

[0066] The distance calculation unit 3 repeats the calculation of step S104 until the inter-distribution distance D is calculated for each of i=1 to L. KL (p c i (·|x1,θ)||p c i (·|x2, θ)) (step S105: No). Then, when the calculation of step S104 is completed for all i (step S105: Yes), the merging unit 4 calculates the L inter-distribution distances D calculated in step S104. KL (p c i (·|x1,θ)||p c i The sum of (·|x2, θ)) (step S106) is set as Loss smooth .

[0067]

[0068] Finally, the optimization unit 5 makes the sum Loss calculated in step S106 smooth Update the parameter θ of the prediction model 10 in a minimizing manner (step S107), Figure 5 The series of processing shown in the flowchart is completed. The update of the parameter θ of the prediction model 10 can be realized by using the probability gradient method, for example. That is, the parameter θ of the prediction model 10 is respectively updated for the sum (Loss smooth ) is differentiated, and the sum (Loss smooth ) is smaller.

[0069] As described in detail with the specific examples cited above, in the present embodiment, when the posterior probability distribution of the prediction model 10 is smoothed near the learning data, the parameters of the prediction model 10 are not updated in a manner that minimizes the distance between the distributions of the posterior probability distribution at the learning data and its nearby points, but are updated in a manner that minimizes the sum of the distances between the distributions of the constituent elements of the posterior probability distribution, i.e., a plurality of small-scale classification distributions. Therefore, according to the present embodiment, the posterior probability distribution of the prediction model 10 that processes sequence data can be smoothed with less computational effort, and the generalization performance of the prediction model 10 can be improved and semi-supervised learning can be achieved.

[0070] <Second embodiment>

[0071] The method of solving the second input data x2 in this embodiment is different from that in the first embodiment. That is, in the first embodiment, the second input data x2 is randomly selected from the data that is away from the first input data x1 by a predetermined distance ε in the input data space, but in this embodiment, the second input data x2 is randomly selected from the data that is away from the first input data x1 by a predetermined distance ε in the input data space. smooth ) is set to the second input data x2.

[0072] In this embodiment, among the data separated by a predetermined distance ε from the first input data x1, the following formula (11) is used

[0073]

[0074] The data with the largest value is selected as the second input data x2. Such second input data x2 can be obtained by the method described in non-patent document 1, for example. That is, a random vector d of the same dimension as the first input data x1 can be set as the initial value, and the following formula (12) can be repeated.

[0075] d←d / ||d||

[0076]

[0077] I p The d obtained by

[0078] x2=x1+ε×d / |d|

[0079] To approximate the second input data x2. In addition, ξ, I p is a predetermined constant.

[0080] As described above, in this embodiment, the above-mentioned sum (Losssmooth ) is used as the second input data x2, so compared with the above-mentioned first embodiment, a high degree of calculation is required, but it can be expected that the parameter θ of the prediction model 10 will efficiently converge to an appropriate value.

[0081] In addition, in the first embodiment and the second embodiment described above, the distance ε between the first input data x1 and the second input data x2 is predetermined as a hyperparameter, but the distance ε may also be mechanically calculated based on the density of the learning data. That is, the distance Dt in the input data space between the learning data closest to the first input data x1 tth (t is a predetermined arbitrary natural number) among the multiple learning data used in the learning of the prediction model 10 and the first input data x1 may be set to ε. In this case, the second input data x2 is data that is Dt away from the first input data x1 in the input data space. For example, the case where the distance ε is calculated by setting t=10 is described in non-patent document 2.

[0082] In this way, by determining the distance ε between the first input data x1 and the second input data x2 based on the density of the learning data, the posterior probability distribution of the prediction model 10 can be smoothed more efficiently.

[0083] <Third embodiment>

[0084] The method for calculating the distance between distributions of the classification distributions of this embodiment is different from that of the first embodiment described above. That is, in the first embodiment described above, the KL divergence is calculated as the distance between distributions of the classification distributions, but in this embodiment, the square error is calculated as the distance between distributions of the classification distributions.

[0085] The distance calculation unit 3 of this embodiment uses the first classification distribution sequence p c 1(·|x1,θ),···,p c L The i-th categorical distribution p in (·|x1,θ) c i (·|x1,θ) and the second classification distribution column p c 1(·|x2,θ),···,p c L The i-th categorical distribution p in (·|x2,θ) c i The distribution distance D between (·|x2, θ) is calculated using the square error for each of i=1 to L. SQ (p c i (·|x1,θ)||p c i (·|x2,θ)). The distance between distributions DSQ (p c i (·|x1,θ)||p c i (·|x2, θ)) can be calculated by the following formula (13).

[0086]

[0087] Where Nc is the number of categories including blanks.

[0088] The merging unit 4 of this embodiment calculates the L inter-distribution distances D calculated by the distance calculation unit 3. SK (p c i (·|x1,θ)||p c i The sum of (·|x2,θ)) is set as Loss smooth .

[0089]

[0090] As described above, in this embodiment, the square error is used instead of the KL divergence as the inter-distribution distance between the class distributions. Therefore, the gradient of the loss function becomes gentle, and the parameter θ of the prediction model 10 may converge to a more appropriate value.

[0091] In addition, whether the generalization performance of the prediction model 10 is improved when the KL divergence or the square error is used as the distribution distance between the classification distributions varies depending on various factors such as the current performance of the prediction model 10 and what kind of learning data is used. Therefore, the performance of the prediction model 10 in which the parameter θ is updated using the KL divergence as the distribution distance between the classification distributions and the performance of the prediction model 10 in which the parameter θ is updated using the square error as the distribution distance between the classification distributions can also be actually evaluated using evaluation data, and the prediction model 10 with the higher evaluation value can be adopted.

[0092] <Supplementary explanation>

[0093] The information processing apparatus of each embodiment described above can be implemented, for example, by using a general-purpose computer as basic hardware. That is, the functions of each part of the information processing apparatus described above can be implemented by causing one or more processors mounted on a general-purpose computer to execute a program. In this case, the information processing apparatus can be implemented by pre-installing the program on the computer, or by storing the program in a storage medium readable by the computer, or distributing the program via a network and installing the program on the computer as appropriate.

[0094] Figure 61 is a block diagram showing an example of the hardware configuration of the information processing device according to each of the above-mentioned embodiments. Figure 6 As shown, it has a hardware structure as a general computer, which includes: a processor 101 such as a CPU (Central Processing Unit); a memory 102 such as a RAM (Random Access Memory) and a ROM (Read Only Memory); a storage device 103 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive); a display device 106 such as a liquid crystal panel, an input device 107 such as a keyboard and a pointing device, and a device I / F 104 for connecting the device; a communication I / F 105 for communicating with the outside of the device; and a bus 108 for connecting these parts.

[0095] In passing Figure 6 When the hardware configuration shown implements the information processing apparatus of each embodiment described above, for example, the processor 101 can use the memory 102 to read and execute the program stored in the storage device 103, etc., to implement the functions of the input receiving unit 1, the sequence prediction unit 2, the distance calculation unit 3, the merging unit 4, and the optimization unit 5, etc. In addition, the prediction model 10 described above can be stored in the storage device 103, etc., and appropriately read into the memory 102 such as RAM, etc., and used for processing by the sequence prediction unit 2, or updated by the optimization unit 5.

[0096] In addition, the functions of each part of the information processing device of each embodiment described above can also be implemented in part or in whole by dedicated hardware (not a general-purpose processor but a dedicated processor) such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array). In addition, it is also possible to use a plurality of processors to implement the functions of each part described above. In addition, the information processing device of each embodiment described above is not limited to the case where it is implemented by a single computer, and the functions can also be distributed to multiple computers for implementation.

[0097] The embodiments of the present invention are described above, but the embodiments are presented as examples and are not intended to limit the scope of the invention. The new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments and their variations are included in the scope or main purpose of the invention, and are included in the invention described in the claims and their equivalents.

Claims

1. An information processing device comprising: The sequence prediction unit uses a prediction model that outputs a sequence of L classification distributions, i.e., a classification distribution column, for one input data to obtain a first classification distribution column corresponding to the first input data and a second classification distribution column corresponding to the second input data, i.e., data that is a predetermined distance away from the first input data in the input data space, wherein: L is a natural number greater than 2; a distance calculation unit, for i=1 to L, respectively calculating the inter-distribution distance between the i-th category distribution in the first category distribution sequence and the i-th category distribution in the second category distribution sequence; A merging part, calculating the sum of the distances between the L distributions; as well as The optimization unit updates the parameters of the prediction model in such a way that the sum becomes smaller.

2. The information processing device according to claim 1, The prediction model is constructed using a recursive neural network that includes a recursive combination of adjacent category distributions in the output category distribution sequence that influence each other.

3. The information processing device according to claim 1 or 2, The optimization unit updates the parameters of the prediction model using a probability gradient method so that the sum becomes smaller.

4. The information processing device according to claim 1, The second input data is data that has the largest sum among data that is a predetermined distance away from the first input data in the input data space.

5. The information processing device according to claim 1 or 2, When the distance in the input data space between the learning data closest to the first input data t-th of the plurality of learning data used in learning the prediction model and the first input data is set to Dt, The second input data is data that is Dt away from the first input data in the input data space, wherein: t is a predetermined arbitrary natural number.

6. The information processing device according to claim 1 or 2, The distance calculation unit calculates the KL divergence between the class distributions or the square error between the class distributions as the inter-distribution distance.

7. An information processing method comprising the following steps: Using a prediction model that outputs a sequence of L categorical distributions, i.e., a categorical distribution column, for one input data, a first categorical distribution column corresponding to the first input data and a second categorical distribution column corresponding to the second input data that is a data at a predetermined distance from the first input data in the input data space are calculated, wherein: L is a natural number greater than 2; For i=1 to L, respectively, calculate the inter-distribution distance between the i-th classification distribution in the first classification distribution column and the i-th classification distribution in the second classification distribution column; Calculating the sum of the distances between L distributions; and The parameters of the prediction model are updated in such a way that the sum becomes smaller.

8. A storage medium storing a program for causing a computer to implement the following functions: Using a prediction model that outputs a sequence of L categorical distributions, i.e., a categorical distribution column, for one input data, a first categorical distribution column corresponding to the first input data and a second categorical distribution column corresponding to the second input data that is a data at a predetermined distance from the first input data in the input data space are calculated, wherein: L is a natural number greater than 2; For i=1 to L, respectively, calculate the inter-distribution distance between the i-th classification distribution in the first classification distribution column and the i-th classification distribution in the second classification distribution column; Calculating the sum of the distances between L distributions; and The parameters of the prediction model are updated in such a way that the sum becomes smaller.

Images