Information processing apparatus, information processing method, and recording medium

Abstract

An information processing apparatus of an embodiment includes a feature amount acquisition unit, a range estimation unit, a correspondence establishment unit, a merging unit, and an output unit. The feature amount acquisition unit acquires a feature amount extracted from data composed of a plurality of values. The range estimation unit estimates a range of data in which an element to which a predetermined label should be given is likely to exist, based on the acquired feature amount. The correspondence establishment unit establishes a correspondence between each label and at least one of a plurality of feature amounts. The merging unit performs merging processing in which one or more ranges of data estimated based on one or more feature amounts to which a label is corresponded are merged into one range of data. The output unit outputs a correspondence relationship between a label and a range of data after the merging processing.

Description

Technical Field

[0001] The embodiments of the present invention relate to information processing apparatus, information processing methods, and programs. Background Technology

[0002] In sequence label recognition, such as speech recognition or string recognition, it is known that combining deep neural networks (DNNs) with sequence label recognition techniques such as Connectionist Temporal Classification (CTC) improves recognition accuracy by recognizing labels without explicitly defining the boundaries between them. On the other hand, in sequence label recognition using techniques like CTC, the range of labels corresponding to the recognition results is unclear because the boundaries between labels in the resulting label column are not defined. In sequence label recognition using techniques like CTC, there is a need to know the range of labels corresponding to each recognition result. While methods for estimating the region of each character from a string image have existed previously, these methods are entirely different from sequence label recognition techniques like CTC and cannot match the high accuracy of CTC's recognition techniques.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 6-251195 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] The technical problem to be solved by the present invention is to provide an information processing device, information processing method and program that can determine the range of input data corresponding to each label in the label column of the recognition result in the recognition process of a sequence of labels with unclear boundaries between labels.

[0008] Means for solving technical problems

[0009] The information processing apparatus of this embodiment includes a feature acquisition unit, a range estimation unit, a correspondence establishment unit, a merging unit, and an output unit. The feature acquisition unit acquires feature quantities extracted from data consisting of multiple values. The range estimation unit estimates the range of data for which elements that should be assigned a predetermined label may exist based on the acquired feature quantities. The correspondence establishment unit establishes a correspondence between each label in a label column having multiple labels and at least one of the multiple feature quantities. The merging unit performs a merging process in which it merges the ranges of one or more data estimated based on one or more feature quantities corresponding to the labels into a single range of data. The output unit outputs the correspondence between the labels and the ranges of the data after the merging process. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating an example of the functional structure of the information processing system according to the first embodiment.

[0011] Figure 2 This is a block diagram illustrating the general functional structure of the information processing apparatus according to the first embodiment.

[0012] Figure 3 This diagram illustrates the processing when the labels of the first embodiment are associated with multiple feature quantities.

[0013] Figure 4 This is a weighted graph used to illustrate the case where the labels of the first embodiment are associated with multiple feature quantities.

[0014] Figure 5 This is a flowchart illustrating a series of operations of the information processing apparatus according to the first embodiment.

[0015] Figure 6 This is a block diagram illustrating the general functional structure of the information processing apparatus according to the second embodiment.

[0016] Figure 7 This is a flowchart illustrating a series of operations of the information processing apparatus according to the second embodiment.

[0017] Figure 8 This is a schematic diagram illustrating an example of the functional structure of the information processing system according to the third embodiment.

[0018] Figure 9 This is a block diagram illustrating the teaching system of the fourth embodiment.

[0019] Figure 10 This is a flowchart illustrating a series of operations of the information processing apparatus according to the fourth embodiment.

[0020] Figure 11This is a schematic diagram of the sequence recognition problem in existing technologies.

[0021] Figure 12 This is a schematic diagram of existing sequence tag recognition technology.

[0022] Figure 13 This diagram illustrates the judgment range of existing sequence tag recognition technologies.

[0023] Figure 14 This diagram illustrates the judgment range of existing sequence label recognition technologies that use attention methods. Detailed Implementation

[0024] Hereinafter, the information processing apparatus, information processing method and program of the embodiments will be described with reference to the accompanying drawings.

[0025] [Primary Technology]

[0026] Reference Figures 11 to 13 This paper describes the existing sequence tag recognition technology. Figure 11 This is a schematic diagram of a sequence recognition problem in the prior art. First, input data DI is fed into a neural network NN. In this example, an example where the input data DI is an image is illustrated. The neural network NN calculates the probability distribution of each feature sequence of the input data DI. The probability distributions of each feature sequence calculated by the neural network NN are combined using a sequence labeling technique 90. The sequence labeling technique 90 is, for example, a method such as CTC. The sequence labeling technique 90 combines the probability distributions of each sequence of features to calculate the probability distribution of a string. Based on the calculated probability distribution of the string, output data DO is output. In the example where the input data DI is an image, the output data DO is a string representing the result of character recognition of the input data DI.

[0027] Figure 12 This is a schematic diagram of existing sequence label recognition technology. The recognition model 85 calculates the probability distribution PC of each character while gradually shifting the judgment range based on the input data DI. The recognition model 85 is constructed, for example, using a neural network NN. In this example, an illustration is given for the case where the input data DI is an image of a string "1958" handwritten horizontally from left to right.

[0028] Figure 13This diagram illustrates the decision range of existing sequence tag recognition technology. In this example, the input data DI is a horizontally handwritten string image from left to right. Therefore, the recognition model 85 calculates the probability distribution PC for each character while moving the decision range 82 from left to right. That is, the probability distribution PC for each character is the probability distribution corresponding to the decision range 82.

[0029] The decision range 82 is a range with a width W along the string direction of the input data DI. The decision range 82 has a size that enables the recognition of each character of the string written in the input data DI. The recognition model 85 keeps the width W of the decision range 82 constant, and calculates the probability distribution PC of each character while moving the decision range 82 a predetermined distance from left to right each time.

[0030] First of all, Figure 13 In step (A), the recognition model 85 calculates the probability distribution PC within the range of the decision range 82. Next, the recognition model 85 takes the range 82 shifted to the right by a predetermined distance as the decision range 82 and calculates the probability distribution PC within the decision range 82. The above steps are repeated until the range 82 reaches the right end of the image. Figure 13 (B), (C), and (D) respectively represent the range of (A) after being offset by distances L1, L1+L2, and L1+L2+L3, which are respectively used as the range 82.

[0031] The distance by which the judgment range 82 is moved is preset by taking into account the width of the element that is to be identified (in this case, the width of the character), etc.

[0032] Return to Figure 12 The probability distribution PC calculated for each decision range 82 by the recognition model 85 is merged by CTC80 to calculate the probability distribution PS of the string. The probability distribution PS of the string is the probability distribution of the string recognized without dividing the boundaries between the labels of the input data DI. Here, since the string recognized by the probability distribution PS does not divide the boundaries between the labels, the range of labels corresponding to the recognition result is unknown. In this embodiment, the range of the input data corresponding to each label in the label column of the recognition result is determined.

[0033] [First Implementation Method]

[0034] The information processing apparatus of this embodiment is used to process the identification result of a sequence identification problem using sequence tag identification technology such as CTC, as described above. In the first embodiment, as an example of sequence tag identification technology, an example of the case using CTC will be described.

[0035] As a specific example of the sequence recognition problem applied to the information processing apparatus of this embodiment, recognition processing broadly includes string recognition, speech recognition, and dynamic image recognition. In this embodiment, string recognition is processed as an example of a label column recognition task. String recognition refers to the task of recognizing the string in an image by taking an image containing a string as input. In this embodiment, horizontally written strings read from left to right are described. However, this embodiment is not limited to horizontally written strings read from left to right; the same applies to vertically written strings. Furthermore, images containing strings broadly include images of handwritten characters, photographs of billboards, road signs, etc.

[0036] Figure 1 This is a schematic diagram illustrating an example of the functional structure of the information processing system 1 according to the first embodiment. Referring to this diagram, the functional structure of the information processing system 1 will be explained. First, input data DI is input into the neural network NN1. In this example, an example where the input data DI is an image of a string will be explained. Specifically, an example where the input data DI is an image of a string "011" handwritten horizontally from left to right will be explained.

[0037] The neural network NN1 computes a sequence of feature quantities F from the input string. When the input data DI is a horizontally handwritten string image from left to right, the neural network NN1 identifies the sequence of feature quantities F from left to right with a decision range width. In this example, the neural network NN1 computes feature quantities F1 to F6. Here, the neural network NN1 computes the number of feature quantities F corresponding to the length of the rows in the input data DI.

[0038] Neural network NN2 calculates the probability distribution P for each feature F calculated by neural network NN1. In this example, neural network NN1 calculates the features from feature F1 to feature F6, therefore neural network NN2 calculates the probability distribution P1 from feature F1 to feature F6.

[0039] The CTC80 combines the calculated probability distributions, calculates the probability distribution P of the string corresponding to the input data DI, and outputs the string identified based on the calculated probability distribution P as the output data DO.

[0040] The information processing device 10 obtains the feature quantity F calculated by the neural network NN1. Based on the obtained feature quantity F, the information processing device 10 uses the neural network NN3 to estimate the possible range of elements that should be assigned a specified label.

[0041] The information processing device 10 establishes a correspondence between each label of the output data DO identified by the CTC 80 and each feature quantity F. When one label in the label column of the output data DO corresponds to multiple feature quantities F, the information processing device 10 merges and outputs the ranges estimated based on the multiple feature quantities F corresponding to that label. The output result from the information processing device 10 is that the range of each label in the input data DI is determined. In the example shown in the figure, range A1 determines the range of "0", range A2 determines the range of "1", and range A3 determines the range of "1".

[0042] Figure 2 This is a block diagram illustrating the general functional structure of the information processing apparatus 10 according to the first embodiment. As shown, the information processing apparatus 10 includes a feature acquisition unit 21, a range estimation unit 23, a correspondence establishment unit 25, a merging unit 27, and an output unit 29. These functional units are implemented, for example, using electronic circuits. Furthermore, each functional unit may, as needed, incorporate a storage mechanism such as a semiconductor memory or a magnetic hard disk drive. Alternatively, each function can be implemented using a computer and software.

[0043] The feature acquisition unit 21 acquires the feature quantity F extracted from the input data DI. For example, the feature quantity F is extracted from the input data DI by a neural network. The feature quantity F acquired by the feature acquisition unit 21 is a feature quantity F corresponding to the dimension of the input data DI. That is, the feature acquisition unit 21 acquires the feature quantity F extracted from data consisting of multiple values, and it is a feature quantity F corresponding to the dimension of the input data DI.

[0044] The range estimation unit 23 estimates the range of possible input data DI for elements that should be assigned a specified label based on the acquired feature quantity F. The range estimation unit 23 is, for example, a neural network that learns by predicting the range of possible data for elements that should be assigned a label based on the acquired feature quantity F.

[0045] Specifically, the range estimation unit 23 predicts, for the (K-1) classes (excluding whitespace), i.e., characters, where each character would be located if it existed within the range of the corresponding input image for each feature quantity F. In this embodiment, a neural network is used that has been trained to predict the distance to the left end and the distance to the right end of each character based on the D-dimensional feature vector. The neural network transforms the D-dimensional vector into a (K-1)×2-dimensional vector, thus ultimately obtaining T×(K-1)×2 values ​​from the T feature vectors. The distances to the left end and the distances to the right end are used because, when estimating the position of characters from a string image, the range of axes other than the string direction (the height direction in this embodiment) is often unimportant. Although the computational cost increases, it can be configured to also predict the height direction.

[0046] Furthermore, in this embodiment, the quantity predicted by the neural network to estimate the range can also be the distance from the vector to the center of the range and the distance to the boundary closest to the width of the range or the vector to the center of the range. By using such an estimation quantity, the character range can be easily determined. In addition, in the case of speech recognition or dynamic image recognition, a network that has been learned in a way that predicts the start and end times of each label based on a D-dimensional vector can also be used.

[0047] The correspondence establishment unit 25 establishes a correspondence between each label in a label column with multiple labels and at least one of multiple feature quantities F. Specifically, the correspondence establishment unit 25 estimates the likelihood of a category based on the feature quantity F and establishes a correspondence between the labels in the label column and the feature quantity F whose likelihood of the category satisfies a predetermined condition.

[0048] More specifically, the correspondence establishment unit 25 estimates a K-class probability distribution based on the sequence of feature quantities F of length T extracted from the input data DI. This K-class probability distribution includes probability distributions of defined labels and blanks, and consists of K classes of probability distributions. Furthermore, the correspondence establishment unit 25 uses a label transformation unit B, which transforms a redundant label column of length T, including blanks, into a label column of length L, smaller than T, excluding blanks, to calculate the maximum likelihood label column. This maximum likelihood label column is the label column with the highest generation probability generated from the probability distribution sequence, i.e., the probability distribution sequence, among the redundant label columns of length T that could be transformed into label columns. Based on the maximum likelihood label column, the correspondence establishment unit 25 establishes a correspondence between each label in the label column and the feature quantity F.

[0049] More specifically, the correspondence establishment unit 25 establishes correspondences between each character of the maximum likelihood string and one or more feature vectors. In this embodiment, the correspondence is established using the label column with the highest probability among the label columns of length T that can be transformed into the maximum likelihood string by the label transformation unit B (hereinafter referred to as the maximum likelihood path). In this embodiment, the label transformation unit B uses a many-to-one transformation that transforms a label column of length T or less into a label column by deleting labels that are consecutive to blank labels from the label column including blank labels. Regarding the characteristics of transformation B, it is guaranteed that each character of the maximum likelihood string corresponds exclusively to a portion of the label columns of the maximum likelihood path while maintaining the same order. Exclusivity means that no character of the maximum likelihood string corresponds to the same label in the maximum likelihood path. For example, in the case where the maximum likelihood string is "011" and the maximum likelihood path is "0-11-1" (where - represents a blank label), the first character of the maximum likelihood string corresponds to the first character of the maximum likelihood path, the second character of the maximum likelihood string corresponds to the third and fourth characters of the maximum likelihood path, and the third character of the maximum likelihood string corresponds to the sixth character of the maximum likelihood path. The maximum likelihood path corresponds to T K-class probability distributions in a one-to-one manner, and therefore also corresponds to T feature vectors in a one-to-one manner.

[0050] The merging unit 27 merges the ranges of data estimated based on one or more feature values ​​F that have been associated with a label. The processing performed by the merging unit 27 is also described as merging processing. When a label is associated with multiple feature values ​​F, the merging unit 27 merges the ranges of multiple data estimated based on the multiple feature values ​​F, and uses this as the range of the merged data. Alternatively, when a feature value F is associated with a label, the merging unit 27 uses the range of that single associated data as the range of the merged data. For each label, the merging unit 27 merges the ranges estimated based on one or more associated feature values ​​F using a weighted average.

[0051] Alternatively, the weights of the weighted average can be calculated by assigning greater weights to regions of data that are closer to the boundaries of the range from which each feature is extracted.

[0052] The merging unit 27 merges the ranges predicted from the feature vectors corresponding to each character of the maximum likelihood string. When a feature vector is corresponding to a character, the merging unit 27 sets the range of the character's category from the range predicted by that feature vector for each category as the final range of the character.

[0053] When multiple feature vectors are mapped to a single character, the merging unit 27 merges the multiple ranges predicted for that character category and uses the merged range as the final range of the character category. In addition to averaging, median, and most frequent values, the merging unit 27 can also merge multiple ranges by considering methods such as a weighted average of the positions of the corresponding input image features F.

[0054] Here, refer to Figure 3 as well as Figure 4 An example will be given of the case where the merging unit 27 merges multiple feature vectors by weighted average when multiple feature vectors are associated with a single character. Figure 3 This diagram illustrates the processing in the case where the label L of the first embodiment is associated with multiple feature quantities F. As shown in the diagram, the label column LS has multiple labels L. Specifically, the label column LS has labels l1, l2, and l3. In this example, label l1 is associated with feature quantity F1, label l2 is associated with features F3 and F4, and label l3 is associated with feature quantity F6. That is, label l2 is associated with multiple feature quantities F.

[0055] In the input data DI, range A33 is defined by feature quantity F3. Range A34 is defined by feature quantity F4. In this case, refer to... Figure 4 An example of the merging unit 27 merging by taking into account the position of the input image will be explained. Figure 4 This is a weighted graph used to illustrate the case where the label L of the first embodiment is associated with multiple feature quantities.

[0056] Figure 4 (A) is a graph representing the weights of each feature quantity F. This graph shows the weights of each feature quantity F at the left, right, top, and bottom ends within each range. As shown in the graph, the weight of feature quantity F3 at the left end is "1.0", the weight at the right end is "0.0", the weight at the top end is "0.5", and the weight at the bottom end is "0.5". Similarly, the weight of feature quantity F4 at the left end is "0.0", the weight at the right end is "1.0", the weight at the top end is "0.5", and the weight at the bottom end is "0.5".

[0057] Figure 4 (B) is a diagram used to illustrate the edges in ranges A33 and A34. In the following description, the left end of range A33 is designated as edge A33L, the right end as edge A33R, the top end as edge A33T, and the bottom end as edge A33B. Similarly, the left end of range A34 is designated as edge A34L, the right end as edge A34R, the top end as edge A34T, and the bottom end as edge A34B.

[0058] Figure 4 (C) is a diagram representing the range A30 when the merging unit 27 performs merging by weighted average. The left end of range A30 is labeled as edge A30L, the right end as edge A30R, the upper end as edge A30T, and the lower end as edge A30B. In this example, regarding the weight of the left end, the weight of feature quantity F3 is "1.0" and the weight of feature quantity F4 is "0.0", so the merging unit 27 prioritizes the result of feature quantity F3 (i.e., the left end A33L of range A34) and sets it as edge A30L. Similarly, regarding the weight of the right end, the weight of feature quantity F3 is "0.0" and the weight of feature quantity F4 is "1.0", so the merging unit 27 prioritizes the result of feature quantity F4 (i.e., the right end A34R of range A34) and sets it as edge A30R. Furthermore, regarding the weights at the top, the weight of feature quantity F3 is "0.5", and the weight of feature quantity F4 is "0.5". Therefore, the merging unit 27 sets the simple average of feature quantity F3 and feature quantity F4 (i.e., the middle of the top end A33T of range A33 and the top end A34T of range A34) as edge A30T. Also, regarding the weights at the top, the weight of feature quantity F3 is "0.5", and the weight of feature quantity F4 is "0.5". Therefore, the merging unit 27 sets the simple average of feature quantity F3 and feature quantity F4 (i.e., the middle of the bottom end A33B of range A33 and the bottom end A34B of range A34) as edge A30B.

[0059] For reference Figure 3 and Figure 4 As explained, the merging unit 27 performs merging by weighted averaging, thereby enabling, for example, to more strongly reflect the feature quantity corresponding to the left side of the input image (closer to the left side) at the left end of the character. Figure 3 The range inferred from the eigenvectors of F1 is more strongly reflected on the right side based on the corresponding eigenvalues ​​(close to the eigenvalues ​​on the right). Figure 3 The range estimated by the characteristic quantity of F6.

[0060] Return to Figure 2 The output unit 29 outputs the correspondence between the labels of the input data DI and the range of those labels. The range of the labels is merged by the merging unit 27 as described above, thus establishing a one-to-one correspondence with the labels. The output unit 29 outputs, for example, the correspondence between the labels of the input data DI and the range of those labels to a storage unit (not shown), a display unit (not shown), or the like.

[0061] Figure 5 This is a flowchart illustrating the processing sequence of the information processing apparatus 10 in this embodiment. The sequence will be explained below according to this flowchart.

[0062] In step S11, the feature acquisition unit 21 acquires the feature quantity F extracted from the input data DI. In step S15, the range estimation unit 23 estimates the range for each category based on the feature quantity F. In step S16, the correspondence establishment unit 25 establishes a correspondence between the maximum likelihood string and the feature quantity F. In step S17, the merging unit 27 merges the ranges that have been established with each character. In step S18, the output unit 29 outputs the maximum likelihood string and the range of each character.

[0063] [Summary of the First Implementation]

[0064] According to the above-described embodiment, the information processing apparatus 10 includes a range estimation unit 23 and a correspondence establishment unit 25, thereby estimating the range of data that an element should be assigned a specified label based on the acquired feature quantity F, and establishing a correspondence between the label and at least one of a plurality of feature quantities F. Furthermore, the information processing apparatus 10 includes a merging unit 27 to merge multiple ranges that have been corresponded to a single label. That is, according to the above-described embodiment, the information processing apparatus 10 is able to determine the range of input data corresponding to each label in the label column of the recognition result.

[0065] Furthermore, according to the above embodiment, the information processing device 10 can determine the range of input data DI corresponding to each tag in the tag column of the recognition result, and therefore can prompt the user with the range of the tag column of the recognition result in the input data DI. For example, the information processing device 10 can interpret the recognition result by prompting the user with the output result via the GUI. In addition, according to the above embodiment, the recognition result can be interpreted, so the information processing device 10 can easily detect and correct misidentifications. That is, according to this embodiment, the information processing device 10 can improve the explanatory power of the recognition result. In addition, according to this embodiment, the explanatory power of the recognition result can be improved, so the information processing device 10 can easily correct misreadings using location information. In addition, this embodiment identifies the tag column using the CTC method, so it can perform tag column recognition with the same accuracy as the CTC method.

[0066] Furthermore, according to the above implementation method, when identifying the label column using the CTC method, a different neural network than CTC is used to determine the range. Therefore, according to the above implementation method, the range of input data corresponding to each label in the label column of the recognition result can be determined without affecting the character recognition accuracy of the CTC method.

[0067] Furthermore, as a variation of this embodiment, the recognition result can be corrected by automatically reducing false recognitions based on the range of each character obtained. For example, the range estimation unit 23 estimates the range of each character not only for the maximum likelihood string but also for the N-bit string, penalizing candidates with excessive overlap in the range of each character and candidates that are estimated to have no character at the position where a character should be present, and determining the final output string based on this, thereby achieving the desired result. Moreover, this embodiment can also be applied to beam search decoding, which is widely used in CTC decoding. In this case, when scoring beam candidates in each loop, penalties are applied to beams with large overlap between characters and beams that are estimated to have no character at the position where a character should be present. By automatically correcting the recognition result by considering range information as described above, a character recognition system with high recognition accuracy can be achieved.

[0068] [Second Implementation]

[0069] refer to Figure 6 and Figure 7 An example of the information processing apparatus 10A according to the second embodiment will be described. Figure 6 This is a block diagram illustrating the general functional structure of the information processing apparatus 10A according to the second embodiment. The information processing apparatus 10A differs from the information processing apparatus 10 in that it includes a feature acquisition unit 21A instead of a feature acquisition unit 21. Regarding... Figure 2 Structures described in the figure are sometimes omitted from the description by using the same reference numerals in the figures.

[0070] The feature acquisition unit 21A includes a feature extraction unit 211 and a recognition unit 212. In this embodiment, the feature acquisition unit 21A includes a feature extraction unit 211 and a recognition unit 212, thereby extracting feature values ​​from the input data DI and recognizing the label column.

[0071] The feature extraction unit 211 extracts feature quantities F from the input data DI. The input data DI is data consisting of more than one value. That is, the feature extraction unit 211 extracts multiple feature quantities F from the data consisting of more than one value.

[0072] When the input data DI is image data, the feature extraction unit 211 calculates a sequence of feature quantities F based on the string image. For example, an example will be described where the input image is transformed into T D-dimensional feature vectors v1, v2, ..., vT using a convolutional neural network (CNN). Alternatively, a recurrent neural network (RNN) can be inserted in the middle or at the end of the CNN.

[0073] Here, the T feature vectors are calculated by the CNN based on different parts of the input image, thus creating a spatial correspondence between the string image and the T feature vectors. This correspondence depends on the CNN's structure, for example, by sliding a rectangle with a height and width of 128px 4px from left to right. In this embodiment, the order from v1 to vT is assumed to correspond to the order from the left end to the right end of the image.

[0074] The recognition unit 212 identifies the label column from multiple feature quantities F. Based on each feature quantity, the recognition unit 212 calculates the probability distribution pn (n = 1, 2, ..., T) of K classes. By performing CTC decoding on these T K class probability distributions p1, p2, ..., pT, it calculates the maximum likelihood string with the highest probability. The K classes refer to the classes that are originally identified, after adding a blank class to represent spaces. For example, in a task of recognizing digit strings containing the numbers 0 to 9, a blank class is added to the 10 classes of 0 to 9, resulting in K = 11.

[0075] CTC decoding is a commonly used decoding method in label column recognition using CTC. In CTC decoding, a many-to-one transformation B is used, which transforms a label column of length T (including blank labels) into a label column of length T or less by removing labels consecutive to blank labels. The likelihood of the label column LS (excluding blank labels of length T or less) is calculated as the sum of the generation probabilities (p1(l1)*p2(l2)*…*pT(lT)) of the label column LS'(l1, l2, …, lT) (including blank labels) that could be transformed by B into L. The maximum likelihood label column with the highest probability is then calculated according to the above rules.

[0076] Typically, in string recognition and speech recognition, the scoring of the recognition results is corrected using pre-prepared grammar rules, language model information such as N-grams, and word lists. In this embodiment, such language information can also be considered to determine the maximum likelihood string.

[0077] Alternatively, the recognition unit 212 may perform decoding based on a high-speed approximation method, namely the swallowing method, instead of CTC decoding. In this case, the maximum likelihood string becomes a string obtained by transforming the tag column, which is formed by arranging the tags with the highest probabilities according to the T K-class probability distributions in the order of their feature vectors, using B.

[0078] Figure 7 This is a flowchart illustrating a series of operations of the information processing apparatus 10A according to the second embodiment. Hereinafter, the sequence of operations will be described according to this flowchart. Figure 7The flowchart shown differs from the one in that it includes steps S12, S13, and S14 instead of step S11. Figure 5 The flowcharts for the series of operations of the information processing device 10 described earlier differ from those in the previous section. For example, in... Figure 5 The content already explained is omitted.

[0079] In step S12, the feature extraction unit 211 calculates a sequence of feature quantities F based on the input data DI. In step S13, the recognition unit 212 calculates a probability distribution for each feature quantity. In step S14, the recognition unit 212 calculates a maximum likelihood string based on the calculated probability distribution.

[0080] [Summary of the Second Implementation]

[0081] According to the above embodiment, the information processing apparatus 10A includes a feature extraction unit 211, which calculates a sequence of feature quantities F based on the input data DI. That is, according to this embodiment, the feature quantity acquisition unit 21 can calculate the feature quantity F based on the input data DI instead of acquiring the feature quantity F calculated by the neural network NN1.

[0082] Furthermore, according to the above embodiment, the information processing apparatus 10A includes an identification unit 212, which identifies a tag column from a plurality of feature quantities F. That is, according to this embodiment, sequence tag identification based on sequence tag recognition technology such as neural network NN2 and CTC can be performed.

[0083] [Third Implementation Method]

[0084] In the above embodiments, an example of using the CTC method was described as an example of sequence label recognition technology. In this embodiment, an example of using the Attention method instead of the CTC method for sequence label recognition technology will be described. For example, it is known that the CTC method and the Attention method have different advantages and disadvantages in character recognition accuracy depending on the application. For example, the CTC method is advantageous when recognizing scanned handwritten strings, while the Attention method is advantageous when recognizing strings captured by a camera.

[0085] First, refer to Figure 14 Explain the existing technology. Figure 14 This diagram illustrates the decision range of a prior art sequence label recognition technique using the attention mechanism. As shown, firstly, features F are extracted from the input data DI. Features F are extracted, for example, through a neural network (not shown). In this example, an example is given where the input data DI is an image.

[0086] Let the process of estimating the nth character of the input data DI be process Pn (n is a natural number). In the case of estimating the EOS (End Of String) indicating the end of the string in step (j is a natural number), in the sequence label recognition technology using the attention method, the string corresponding to the input data DI is estimated by the processes from process P1 to process Pn.

[0087] In sequence label recognition technology using the Attention mechanism, the feature quantity F is copied in step P1. The copied feature quantity F is multiplied by the Attention mapping (weight mapping) M1, and a weighted sum is calculated. The calculated weighted sum is recorded as the feature vector V1. Here, in the Attention mapping M1, the horizontal axis represents the position of the input data DI, and the vertical axis represents the weight. The Attention mapping M1 corresponds to the range of characters on the left side of the input data DI, therefore the weights on the left side are larger than those on the right side.

[0088] The calculated feature vector V1 is input into the neural network NN2-2. The neural network NN2-2 predicts the label y0 of the first character. Specifically, the neural network NN2-2 infers which character is most likely to be the first character. Thus, by repeatedly performing steps P1 to Pj, the entire string of the input data DI is predicted using a sequence label recognition technique employing attention.

[0089] Alternatively, the attention map M can also be created using a neural network NN2-1 (not shown). In this case, the neural network NN2-1 creates the attention map M based on the character on the left (e.g., the character estimated in step P1 in step P2), the internal state of the neural network NN2-2 when predicting that character, and the feature quantity F.

[0090] Alternatively, the neural network NN2-2 can also predict characters based on the character on the left (e.g., the character estimated in step P1 in step P2), the internal state of the neural network NN2-2 when predicting that character, and the feature vector V. In this case, the neural network NN2-2 is a recurrent neural network. Thus, in sequence label recognition technology using the attention method, character prediction is repeatedly performed until the EOS (End of String) is predicted.

[0091] Figure 8This is a schematic diagram illustrating an example of the functional structure of the information processing system 1B according to the third embodiment. Referring to this diagram, the functional structure of the information processing system 1 will be explained. First, input data DI is input into the neural network NN1. In this example, an example where the input data DI is an image of a string will be explained. Specifically, an example where the input data DI is an image of a string "011" handwritten horizontally from left to right will be explained.

[0092] In this embodiment, the information processing device 10B establishes a correspondence between each label of the output data DO calculated using the attention method and the feature quantity F calculated by the neural network NN1. When each label of the output data DO calculated using the attention method is corresponded to multiple feature quantities F, the information processing device 10B merges and outputs the ranges estimated based on one or more feature quantities F that have been corresponded to the labels. The output result from the information processing device 10B is that the range of each label in the input data DI is determined. In one example shown in the figure, range A1 determines the range of "0", range A2 determines the range of "1", and range A3 determines the range of "1".

[0093] Specifically, in this embodiment, the feature extraction unit 211 calculates T feature vectors v1, v2, ..., vT in the same manner as in the embodiment described above. The recognition unit 212 first inputs the T feature vectors into the neural network 2-1, and weights the feature vectors that are useful for recognizing the first character. Here, the weights refer to T values ​​w1, w2, ..., wT that are greater than 0 and less than 1 and whose sum is 1. Using the weighted average of the T feature vectors with these weights (w1*v1 + ... + wT*vT), the first character is further recognized using the other neural network 2-2.

[0094] Next, the recognition result of the first character and the features obtained during the recognition process are added to the input of neural network 2-1, and the feature vectors useful for recognizing the second character are weighted. The second character is then recognized using a weighted average of the T feature vectors using these weights and neural network 2-2. Neural network 2-2 repeats this process until it outputs the EOS (Equation of Likelihood) indicating the end of the string. The maximum likelihood strings are arranged sequentially from the first character to the EOS.

[0095] Similar to the embodiment described above, the range estimation unit 23 predicts the range for each category for all feature vectors. The correspondence establishment unit 25 establishes a correspondence between the maximum likelihood string and T feature vectors. During correspondence establishment, the weights calculated for each character in the neural network 2-1 of the recognition unit 212 are used. The correspondence establishment unit 25 establishes a correspondence between each character and a feature quantity with a weight greater than or equal to a predetermined value (e.g., 0.01). When merging ranges, the merging unit 27 may also use a weighted sum that uses these weights.

[0096] [Summary of the Third Implementation]

[0097] Previously, in sequence label recognition technology based on the attention method, the range of input data corresponding to each label in the label column of the recognition result could only be determined by weight information. That is, according to the existing technology, only a vague range can be estimated. However, according to the above-described implementation method, the information processing device 10B can also determine the range of input data corresponding to each label in the label column of the recognition result in the attention method, instead of the CTC method.

[0098] In this embodiment, the ambiguous positions determined by the attention mechanism can be used to establish a correspondence between the range inferred by the neural network NN3 and each label of the output data DO, based on a predetermined threshold. When a label is corresponded to multiple feature quantities F, similarly to the first embodiment, the range of the label can be determined by merging the ranges.

[0099] [Fourth Implementation Method]

[0100] In this embodiment, a method for simplifying human-based range teaching is described. Range teaching is used, for example, to create datasets for machine learning. The processing differs from the implementation described above in that the string that should be presumed to be a range is not a recognition result when input from the user. (See also...) Figure 9 and Figure 10 The fourth embodiment will be described below.

[0101] Figure 9 This is a block diagram illustrating the teaching system 2 according to the fourth embodiment. In this embodiment, the teaching system 2 includes an information processing device 10C instead of the aforementioned information processing device 10. Furthermore, the teaching system 2 includes a display device 41, an input device 43, and a storage device 45.

[0102] The input device 43 acquires information input from the user U and outputs it to the information processing device 10C. The input device 43 may be, for example, a character input device such as a keyboard, or a pointing device such as a mouse.

[0103] Display device 41 displays information output from information processing device 10C. Display device 41 may be, for example, a liquid crystal display or other display device.

[0104] Storage device 45 is a storage device controlled by information processing device 10. Information processing device 10 can be located either inside or outside information processing device 10C.

[0105] Information processing apparatus 10C is a variation of information processing apparatus 10. Information processing apparatus 10C differs from information processing apparatus 10 in that it includes an operation receiving unit 31 and a storage control unit 33. For structures identical to those of information processing apparatus 10, the same reference numerals are used, and descriptions are sometimes omitted.

[0106] The operation receiving unit 31 receives operations from the user U via the input device 43. In this embodiment, the user U determines whether the result output by the information processing device 10C is correct. The user U inputs the determination result to the information processing device 10C via the input device 43. That is, the operation receiving unit 31 obtains the result obtained by the user U in determining whether the result output by the output unit 29 is correct. In addition, the information obtained by the operation receiving unit 31 may also include correction information indicating the correction of the range estimated by the range estimation unit 23.

[0107] The storage control unit 33 causes the storage device 45 to store the results obtained by the operation receiving unit 31. If the information obtained by the operation receiving unit 31 includes correction information, the storage control unit 33 causes the storage device 45 to store the results including the correction information obtained by the operation receiving unit 31.

[0108] Figure 10 This is a flowchart illustrating a series of operations of the information processing apparatus 10C according to the fourth embodiment. The series of operations of the information processing apparatus 10C according to the fourth embodiment will be explained with reference to this diagram.

[0109] In step S21, user U first inputs a string corresponding to the string image. That is, feature acquisition unit 21 obtains the string corresponding to the string image from user U via input device 43. Alternatively, feature acquisition unit 21 may be configured to pre-store pairs of strings corresponding to multiple string images in a storage device (not shown), and obtain the string corresponding to the string image from the storage device.

[0110] In step S22, the information processing device 10C determines the range of the input string. First, the information processing device 10C calculates a sequence of feature values ​​for the input string image, similar to the embodiment described above. Furthermore, similar to the embodiment described above, the information processing device 10C estimates the range of each category based on each feature value.

[0111] The correspondence establishment unit 25 establishes correspondences between each character of the input string and the feature quantity. For example, the correspondence establishment unit 25 calculates the probability distribution of K classes from each feature quantity, finds the label column with the highest probability among the label columns that can be transformed by B into the length T of the input string, and establishes the correspondence in the same way as in the embodiment described above. Alternatively, the correspondence establishment unit 25 may also calculate the weights of the feature quantities as described in the third embodiment, until the input string and the immediately following end string are output, and use the weights to establish the correspondence.

[0112] Furthermore, the correspondence establishment unit 25 can also roughly establish correspondences between each character of the input string and the image region based on the density and brightness of the input image, and establish correspondences between characters and feature quantities based on the positional correspondence between the input image and feature quantities. In this case, the approximate correspondence between the input image and characters can be established, for example, by estimating blank areas where no characters exist based on the brightness histogram of the string direction, and then dividing the region equally by the number of characters and corresponding it to the characters, excluding these blank areas. The correspondence establishment between image regions and feature quantities can also be configured such that the overlap rate between the target image region and the image region corresponding to the feature quantity is a certain or higher.

[0113] The merging unit 27 merges the ranges corresponding to each character in the same manner as in the embodiment described above. The output unit 29 causes the display device 41 to display the estimated ranges of each character. For example, the ranges of each character are drawn as rectangles in a manner that overlaps with the image, and the characters corresponding to each rectangle are displayed.

[0114] In step S23, user U confirms that the range of each character displayed on display device 41 is correct. If the range is correctly determined, user U inputs a confirmed meaning via input device 43. If the range is incorrect, user U corrects the range via input device 43 to make it correct. Range teaching provides an interface for correction. For example, this interface could be a method of dragging and dropping the vertices and edges of a rectangle using a mouse pointer. After correction, user U inputs a confirmed meaning and ends the operation.

[0115] In step S25, if the range of each character is correct, the information processing device 10C proceeds to step S27. If the range of each character is incorrect, the information processing device 10C proceeds to step S29.

[0116] In step S27, if the range of each character is correct, the storage control unit 33 causes the storage device 45 to store the range of each character. Specifically, the storage control unit 33 causes the storage device 45 to associate the determined character range with the string image and the string for storage.

[0117] In step S29, if the range of each character is incorrect, the storage control unit 33 causes the storage device 45 to store the corrected range of each character.

[0118] Furthermore, the input string does not necessarily have to be a correct string for the string image. An interface could also be added to verify the correctness of the string during range confirmation and correct any errors, even if the string contains erroneous automatic recognition results. In this case, the range is re-estimated based on the user's string corrections, thereby reducing the range teaching load on the user.

[0119] [Summary of the Fourth Implementation]

[0120] According to the above-described embodiment, the information processing device 10C includes an operation receiving unit 31 and a storage control unit 33, thereby obtaining information from the user U regarding whether the range of each character is correct and storing it in the storage device 45. By storing information regarding whether the range of each character is correct in the storage device 45, the information processing device 10C makes teaching the range of characters easier for the user.

[0121] As explained above, multiple variations are described in the embodiments. Here, multiple embodiments and variations can be combined for implementation as long as they can be combined.

[0122] Alternatively, the functions of the information processing device described in the above embodiments can also be implemented using a computer. In this case, it can be achieved by recording the program for implementing the function on a computer-readable recording medium, and then having the computer system read and execute the program recorded on the recording medium. Furthermore, the term "computer system" here includes hardware such as an operating system and peripheral devices. Additionally, "computer-readable recording medium" refers to removable media such as floppy disks, optical disks, ROMs, CD-ROMs, DVD-ROMs, and USB storage devices, as well as storage devices such as hard disks built into the computer system. Moreover, "computer-readable recording medium" can also include media that dynamically maintains a program for a short period of time, such as a communication line used when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or media that maintains a program for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the program described above can be a program used to implement the aforementioned functions, or it can be a program that can implement the aforementioned functions by combining with a program already recorded in the computer system.

[0123] According to at least one embodiment described above, by having a feature acquisition unit, a range estimation unit, a correspondence establishment unit, a merging unit, and an output unit, it is possible to determine the range of input data corresponding to each label in the label column of the recognition result in the recognition processing of sequence labels where the boundaries between labels are not clearly defined.

[0124] Several embodiments of the present invention have been described, but these embodiments are given by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope or spirit of the invention, as well as in the scope of the invention as described in the claims and its equivalents.

[0125] Explanation of reference numerals in the attached figures

[0126] 1…Information processing system, 2…Teaching system, 10…Information processing device, 21…Feature acquisition unit, 211…Feature extraction unit, 212…Identification unit, 23…Range estimation unit, 25…Correspondence establishment unit, 27…Merging unit, 29…Output unit, 31…Operation acceptance unit, 33…Storage control unit, 41…Display device, 43…Input device, 45…Storage device, 80…CTC, 81…All range, 82…Judgment range, 85…Identification model, 90…Sequence tag recognition technology, DI…Input data, DO…Output data, NN…Neural network, F…Feature quantity, P…Probability distribution, LS…Label column, L…Label.

Claims

1. An information processing device, comprising: The feature acquisition unit acquires feature quantities corresponding to the dimensions of data extracted from data consisting of multiple values, wherein the data is image or audio data; The range estimation unit estimates, based on the obtained feature quantities, the range of data that the elements that should be assigned a specified label may exist. The corresponding establishment unit establishes a correspondence between at least one of the labels in a label column having multiple labels and at least one of the multiple feature quantities; The merging section performs a merging process, in which the ranges of one or more data estimated based on one or more feature quantities corresponding to the label are merged into a single range of data. as well as The output unit outputs the correspondence between the labels and the range of the data after the merging process.

2. The information processing apparatus according to claim 1, wherein, The range estimation unit is a neural network that has been trained to predict the range of data that an element should be labeled based on the acquired feature quantities.

3. The information processing apparatus according to claim 1 or 2, wherein, The correspondence establishment unit estimates the likelihood of a category based on the feature quantity, and establishes a correspondence between the labels in the label column and the feature quantities whose likelihood of the category satisfies a predetermined condition.

4. The information processing apparatus according to claim 1 or 2, wherein, The correspondence establishment unit, based on the sequence of feature quantities of length T extracted from the data, estimates a K-class probability distribution, which includes the specified labels and blanks, and is a probability distribution of K categories. Using a label transformation unit that transforms the redundant label columns of length T, including blanks, into label columns of length L, which are shorter than length T and do not include blanks, it finds the maximum likelihood label column among the redundant label columns of length T that may be transformed into the label columns, which has the highest generation probability generated from the probability distribution sequence. Based on the maximum likelihood label column, it establishes a correspondence between each label in the label column and the feature quantity.

5. The information processing apparatus according to claim 1 or 2, wherein, The merging unit, for each of the labels, merges the data ranges estimated based on one or more corresponding feature quantities using a weighted average. The weights of the weighted average are calculated such that the closer the region of the data to which each of the features is extracted is to the partial boundary of the range of the data, the greater the weight.

6. The information processing apparatus according to claim 1 or 2, wherein, The feature acquisition unit also includes: The feature extraction unit extracts multiple feature quantities from the data consisting of one or more values; and The identification unit identifies the label column from multiple of the said feature quantities.

7. The information processing apparatus according to claim 1 or 2, wherein, It also has: The operation receiving unit obtains the result obtained by the user's judgment on whether the result output by the output unit is correct; and The storage control unit stores the results obtained by the operation receiving unit.

8. The information processing apparatus according to claim 7, wherein, The information obtained by the operation receiving unit includes correction information, which instructs on corrections to the range of the data estimated by the range estimation unit. The storage control unit stores the results obtained by the operation receiving unit, including the correction information.

9. A recording medium having a program recorded thereon for causing a computer to perform the following steps: The feature acquisition step involves obtaining feature quantities corresponding to the dimensions of data extracted from data consisting of multiple values, where the data is image or audio data. The range estimation step, based on the obtained feature quantities, estimates the range of data that the elements that should be assigned the specified labels may exist. In the corresponding establishment step, a correspondence is established between the multiple labels and at least one of the multiple feature quantities; The merging step involves performing a merging process, in which the ranges of one or more data estimated based on one or more feature quantities corresponding to the label are merged into a single range of data. as well as The output step outputs the correspondence between the label and the range of the data after the merging process.

10. An information processing method, comprising: The feature acquisition step involves obtaining feature quantities corresponding to the dimensions of data extracted from data consisting of multiple values, where the data is image or audio data. The range estimation step, based on the obtained feature quantities, estimates the range of data that the elements that should be assigned the specified labels may exist. In the corresponding establishment step, a correspondence is established between the multiple labels and at least one of the multiple feature quantities; The merging step involves performing a merging process, in which the ranges of one or more data estimated based on one or more feature quantities corresponding to the label are merged into a single range of data. as well as The output step outputs the correspondence between the label and the range of the data after the merging process.

Images