Information Processing Apparatus, Information Processing Method, and Information Processing Program

Abstract

In order to highly accurately determine a label to be given to an object even when only a single prediction model exists, this information processing device (1) comprises: an acquisition unit (11) which acquires a set of objects; an evaluation unit (12) which evaluates the similarity between the objects included in the set of objects, and identifies one or a plurality of similar objects that are similar to an object to be predicted; and a prediction unit (13) which determines a label to be given to the object to be predicted with reference to a similar label that is a label given to each of the one or the plurality of similar objects and predicted by a prediction model.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] When creating teacher data required for learning a machine learning model, an operation called annotation for manually assigning correct answers is necessary. Manual annotation is known to be costly. To reduce the cost of annotation, there is a technique for assisting annotation by predicting correct answers using a prediction model. In this technique, when predicting the correct answer to be assigned to an annotation target by a prediction model, the prediction result is corrected to supplement the accuracy of the prediction model. For example, Non-Patent Document 1 describes that predictions are made using a plurality of prediction models for the feature amounts of a prediction target, and the prediction result is corrected by a process using a statistical model.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Non-Patent Document 1 requires a plurality of prediction models. Therefore, when there is only a single prediction model, there is a problem that the prediction result cannot be appropriately corrected.

[0005] One aspect of the present invention has been made in view of the above problems, and an example of the object is to provide a technique capable of accurately determining a label to be assigned to an object even when there is only a single prediction model.

Means for Solving the Problems

[0006] An information processing apparatus according to one aspect of the present invention includes an acquisition unit that acquires a set of objects, an evaluation unit that evaluates the similarity between the objects included in the set of objects and identifies one or more similar objects similar to a prediction target object, and a prediction unit that determines a label to be assigned to the prediction target object by referring to a similar label that is a label assigned to each of the one or more similar objects and predicted by a prediction model.

[0007] An information processing method according to one aspect of the present invention includes acquiring a set of objects, evaluating the similarity between the objects included in the set of objects, identifying one or more similar objects similar to a prediction target object, and determining a label to be assigned to the prediction target object by referring to a similar label that is a label assigned to each of the one or more similar objects and predicted by a prediction model.

[0008] An information processing program according to one aspect of the present invention causes a computer to execute a process of acquiring a set of objects, a process of evaluating the similarity between the objects included in the set of objects, a process of identifying one or more similar objects similar to a prediction target object, and a process of determining a label to be assigned to the prediction target object by referring to a similar label that is a label assigned to each of the one or more similar objects and predicted by a prediction model.

Advantages of the Invention

[0009] According to one aspect of the present invention, even when there is only a single prediction model, the label assigned to an object can be determined with high accuracy.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Modes for Carrying Out the Invention

[0011] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0012] <Overview of Information Processing Apparatus> The information processing apparatus 1 according to this exemplary embodiment is a device that determines a label to be assigned to an object. Here, an object is a target to which a label is to be assigned, and as an example, it is data representing an image or text to be classified. Also, the object may be data representing a product that is a target of sales prediction. Further, the object may be an entity included in a sentence written in natural language, or may be data representing a pair of a sentence written in natural language and an entity included in that sentence. Here, an entity is a character string representing a specific concept or thing, and as an example, it is a proper noun or a common noun.

[0013] A label is a value or a set of values assigned to an object. As an example, a label has a data structure including numerical values such as a scalar, a vector, or a matrix. Also, a label may have a data structure including a character string. A plurality of labels may be assigned to an object. Also, a score representing the reliability of the label may be attached to the label. Assigning a plurality of labels to an object can also be expressed as the labels assigned to the object being a combination of the values of a plurality of labels. An object with a label is used, for example, as teaching data for training a machine learning model. Hereinafter, assigning a label to an object is also referred to as "annotation".

[0014] <Configuration of Information Processing Apparatus 1> The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing apparatus 1. As shown in FIG. 1, the information processing apparatus 1 includes an acquisition unit 11, an evaluation unit 12, and a prediction unit 13.

[0015] (Acquisition Unit 11) The acquisition unit 11 acquires a set of objects. The set of objects is, for example, a set of image data, a set of text data, or a set of data representing products. Also, the set of objects may be a set of data representing pairs of sentences and entities included in the sentences.

[0016] (Evaluation Unit 12) The evaluation unit 12 evaluates the similarity between the objects included in the set of objects and identifies one or a plurality of similar objects similar to the object to be predicted. Here, the object to be predicted is the object to which a label is to be assigned. Also, in this specification, "objects are similar" means that the objects have similar features to each other and that the objects have the same features. In other words, "objects are similar" in this specification includes "objects are identical".

[0017] The similarity between objects represents the degree of similarity of the objects. As an example, the evaluation unit 12 evaluates the similarity between objects based on the feature amounts of the objects. Here, the feature amount is a set of values representing the features of the object. As an example, the feature amount may have a data structure including numerical values such as a scalar, a vector, or a matrix, or may have a data structure including a character string. The feature amount includes, as an example, a set of pixel values of an image, a set of words included in text, or attribute values such as the price of a product.

[0018] The method by which the evaluation unit 12 identifies similar objects is not limited. For example, the methods include (i) a method in which the evaluation unit 12 outputs a set of similar objects similar to the object to be predicted, and (ii) a method in which the evaluation unit 12 outputs a graph or a hypergraph representing the similarity relationship between objects. However, the method by which the evaluation unit 12 identifies similar objects is not limited to these examples, and the evaluation unit 12 may identify similar objects by other methods.

[0019] When the evaluation unit 12 outputs (i) a set of similar objects, the evaluation unit 12, as an example, identifies an object whose similarity to the object to be predicted is equal to or greater than a predetermined threshold value. Further, the evaluation unit 12 may cluster the objects using a clustering method such as spectral clustering using the similarity between the objects, and identify the objects belonging to the same cluster as the object to be predicted as similar objects. Weight information corresponding to the similarity may be given as additional information to the similar objects.

[0020] Here, the similarity relationship in the set of similar objects does not need to be bidirectional. For example, when the object to be predicted is object OBJ_A, and the similar objects of object OBJ_A are objects OBJ_B, OBJ_C, OBJ_D, and OBJ_E, the set of similar objects similar to object OBJ_B may not include object OBJ_A. Also, the set of similar objects of object OBJ_A may include object OBJ_A.

[0021] When the evaluation unit 12 outputs (ii) a graph or hypergraph representing the similarity relationship between objects, as an example, the evaluation unit 12 outputs a graph or hypergraph having nodes as objects and edges or hyperedges connecting the nodes whose similarity has been evaluated. These edges or hyperedges may be given weight information according to the similarity between the corresponding nodes.

[0022] (Prediction unit 13) The prediction unit 13 determines the label to be assigned to the object to be predicted by referring to the similar labels, which are the labels assigned to each of the one or more similar objects identified by the evaluation unit 12 and predicted by the prediction model. The similar labels may be pre-assigned to the similar objects, or the prediction unit 13 may predict the similar labels using the prediction model. Also, multiple similar labels may be assigned to one similar object.

[0023] A prediction model is a model that predicts the label of an object or the value included in the label. The prediction model may be a machine learning model generated by machine learning, or may be a rule-based system or a system that refers to an external database. The input of the prediction model is, for example, the feature amount of the object. The output of the prediction model is, for example, the label for the input feature amount or the value included in the label. Here, the value included in the label is a value that constitutes all or part of the label, and for example, each element of the vector when the label is a vector. Further, the output of the prediction model may include a pair of values of a plurality of labels and scores such as the reliability of each label. The prediction model may be stored in the memory of the information processing device 1, or may be stored in another device communicable with the information processing device 1.

[0024] The label predicted by the prediction model may be directly assigned as a similar label to the similar object, or a part of the label predicted by the prediction model may be assigned as a similar label to the similar object. As an example, the top K (K is a natural number of 1 or more) likely labels among a plurality of labels may be assigned as similar labels to the similar object.

[0025] The method by which the prediction unit 13 determines the label to be assigned to the object to be predicted with reference to the similar label is not limited. As an example, the prediction unit 13 may first predict the label of the object to be predicted using the prediction model, and then replace it with the label obtained by referring to the similar label of the similar object. Further, the prediction unit 13 may omit the prediction of the label of the object to be predicted by the prediction model and assign the label obtained by referring to the similar label of the similar object to the object. Further, the label determined by the prediction unit 13 may be attached with additional information such as a score.

[0026] More specifically, as an example, the prediction unit 13 may sum the scores assigned to the similar labels for each similar label, and determine the similar label with the largest total score value as the label to be assigned to the object to be predicted.

[0027] Also, as an example, the prediction unit 13 predicts the label of the object to be predicted by the prediction model. When the label predicted by the prediction model is included in the set of similar labels, the predicted label may be directly determined as the label to be assigned to the object to be predicted. On the other hand, when the label of the object to be predicted predicted by the prediction model is not included in the set of similar labels, the prediction unit 13 may determine the label to be assigned to the object to be predicted from the set of similar labels. In this case, as an example, the prediction unit 13 may determine the most likely similar label among the set of similar labels as the label to be assigned. Here, the prediction unit 13 may evaluate the likelihood of the similar label by referring to additional information such as the score assigned to the similar label, or may also evaluate it by referring to the frequency (number) of the similar label in the set of similar labels.

[0028] Note that the method for the prediction unit 13 to determine the label to be assigned to the object to be predicted is not limited to the above-described examples. The prediction unit 13 may determine the label to be assigned to the object to be predicted by other methods.

[0029] As described above, in the information processing apparatus 1 according to this exemplary embodiment, a set of objects is acquired, the similarity between the objects included in the set of objects is evaluated, one or more similar objects similar to the object to be predicted are specified, and the label to be assigned to the object to be predicted is determined by referring to the similar labels, which are the labels assigned to each of the one or more similar objects and predicted by the prediction model. Therefore, according to the information processing apparatus 1 according to this exemplary embodiment, even when there is only a single prediction model, the effect of being able to accurately determine the label to be assigned to the object can be obtained.

[0030] <Flow of the information processing method> The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the information processing method S1.

[0031] In step S11, the acquisition unit 11 acquires a set of objects. In step S12, the evaluation unit 12 evaluates the similarity between the objects included in the set of objects, and identifies one or more similar objects similar to the object to be predicted. In step S13, the prediction unit 13 determines the label to be assigned to the object to be predicted by referring to the similar labels that are the labels assigned to each of the one or more similar objects and are predicted by the prediction model.

[0032] As described above, in the information processing method S1 according to this exemplary embodiment, a set of objects is acquired, the similarity between the objects included in the set of objects is evaluated, one or more similar objects similar to the object to be predicted are identified, and the label to be assigned to the object to be predicted is determined by referring to the similar labels that are the labels assigned to each of the one or more similar objects and are predicted by the prediction model. Therefore, according to the information processing method S1 according to this exemplary embodiment, even when there is only a single prediction model, the effect that the label to be assigned to the object can be determined with high accuracy is obtained.

[0033] 〔Exemplary Embodiment 2〕 The second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in Exemplary Embodiment 1 are denoted by the same reference numerals, and the description thereof will not be repeated.

[0034] <Configuration of the information processing apparatus 1A> FIG. 3 is a block diagram showing the configuration of the information processing apparatus 1A according to this exemplary embodiment. The information processing apparatus 1A includes a control unit 10A, a storage unit 20A, an input / output unit 30A, and a communication unit 40A.

[0035] (Communication unit 40A) The communication unit 40A communicates with a device external to the information processing apparatus 1A via a communication line. The specific configuration of the communication line does not limit this exemplary embodiment, but as an example, the communication line is a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public switched telephone network, a mobile data communication network, or a combination thereof. The communication unit 40A transmits the data supplied from the control unit 10A to another device, or supplies the data received from another device to the control unit 10A.

[0036] (Input / output unit 30A) Input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel are connected to the input / output unit 30A. The input / output unit 30A receives input of various types of information to the information processing apparatus 1A from the connected input devices. Further, the input / output unit 30A outputs various types of information to the connected output devices under the control of the control unit 10A. Examples of the input / output unit 30A include an interface such as USB (Universal Serial Bus).

[0037] (Control unit 10A) As shown in FIG. 3, the control unit 10A includes an acquisition unit 11, an evaluation unit 12, and a prediction unit 13.

[0038] (Acquisition unit 11) The acquisition unit 11 acquires a set of objects. As an example, the acquisition unit 11 acquires a set of objects from another device via the communication unit 40A. Further, as an example, the acquisition unit 11 may acquire a set of objects input via the input / output unit 30A. Further, the acquisition unit 11 may acquire a set of objects by reading a set of objects from the storage unit 20A or an externally connected storage device.

[0039] (Evaluation unit 12) The evaluation unit 12 evaluates the similarity between objects included in the set of objects, and identifies one or more similar objects similar to the object to be predicted. Details of the process in which the evaluation unit 12 identifies similar objects will be described later.

[0040] (Prediction unit 13) The prediction unit 13 determines the label to be assigned to the object to be predicted by referring to the similar labels, which are the labels assigned to each of the one or more similar objects and predicted by the prediction model M1. Details of the process in which the prediction unit 13 determines the label will be described later.

[0041] (Storage unit 20A) The storage unit 20A stores the object set OC, which is the set of objects acquired by the acquisition unit 11. The storage unit 20A also stores a prediction model M1 for predicting the label of an object and an evaluation model M2 for evaluating the similarity between objects. Here, storing the prediction model M1 in the storage unit 20A means that the parameters defining the prediction model M1 are stored in the storage unit 20A. Also, storing the evaluation model M2 in the storage unit 20A means that the parameters defining the evaluation model M2 are stored in the storage unit 20A.

[0042] (Prediction model M1) The prediction model M1 is a model that predicts the label of an object or the value included in the label. As an example, the prediction model M1 is a prediction model constructed by machine learning that takes the feature amount of an object as input and outputs a label. The learning of the prediction model M1 may be performed by the control unit 10A of the information processing apparatus 1A, or may be performed by another apparatus. The machine learning method of the prediction model M1 is not limited. As an example, a decision tree-based, linear regression, or neural network method may be used, or two or more of these methods may be used. Examples of decision tree-based methods include LightGBM (Light Gradient Boosting Machine), random forest, and XGBoost. Examples of linear regression include Bayesian regression, support vector regression, Ridge regression, Lasso regression, and ElasticNet. An example of a neural network is deep learning.

[0043] The output of the prediction model M1 may include a plurality of labels, and may also include a score representing the reliability of each label.

[0044] As an example, the prediction model M1 is constructed by machine learning using training data including a pair of the feature amount of an object and a label.

[0045] (Evaluation model M2) The evaluation model M2 is a model for evaluating the similarity between objects. As an example, the evaluation model M2 is a model for clustering objects. For object clustering, for example, techniques such as the k-means method or spectral clustering can be applied, but it is not limited to this. In this case, the similarity can be calculated using the clustering result by the evaluation model M2. That is, based on the clustering result of the evaluation model M2, it is possible to calculate whether they are in the same cluster as the similarity.

[0046] The learning of the evaluation model M2 may be performed by the control unit 10A of the information processing apparatus 1A, or may be performed by another apparatus. The similarity between objects is, for example, the distance between objects in the feature space in which the objects are embedded, or a value calculated based on the distance. Further, when an object includes a character string, whether the character strings match, or an index (Hamming distance, edit distance, etc.) regarding the degree of similarity defined between the character strings may be used as the similarity between the objects. Further, the similarity between objects may be represented by an edge between nodes representing the objects, for example. As an example, the presence of an edge between nodes indicates that the objects are similar, and the absence of an edge between nodes indicates that the objects are not similar. Such a graph structure may be given from outside the evaluation model M2, or may be stored in advance in the storage unit 20A or the like as a parameter of the evaluation model M2. However, the similarity between objects is not limited to the examples described above.

[0047] <Flow of the information processing method by the information processing apparatus 1A> The flow of the information processing method executed by the information processing apparatus 1A configured as described above will be described with reference to the drawings. Here, as the information processing method executed by the information processing apparatus 1A, the following information processing methods S100 to S400 will be described. (i) Information processing method S100: The prediction unit 13 determines the label to be assigned to the object to be predicted with reference to the similar label. At this time, the prediction unit 13 does not predict the label of the object to be predicted using the prediction model M1. (ii) Information processing method S200: The prediction unit 13 predicts the pre-modification label of the object to be predicted using the prediction model M1, and corrects the pre-modification label with reference to the similar label. (iii) Information processing method S300: The prediction unit 13 determines a plurality of labels as the labels to be assigned to the object to be predicted. (iv) Information Processing Method S400: The evaluation unit 12 outputs a graph or hypergraph representing the similarity relationship between objects, and the prediction unit 13 determines a label to be assigned to the object to be predicted using the graph or hypergraph.

[0048] (Flow of Information Processing Method S100) FIG. 4 is a flowchart showing the flow of an information processing method S100 which is an example of the information processing method executed by the information processing apparatus 1A. Note that descriptions of already explained content will not be repeated.

[0049] (Step S111) In step S111, the acquisition unit 11 acquires the object set OC. As an example, the acquisition unit 11 may receive the object set OC from another device via the communication unit 40A, or may acquire the object set OC input via the input / output unit 30A. Further, the acquisition unit 11 may acquire the object set OC by reading the object set OC from the storage unit 20A or an external storage device.

[0050] (Step S112) In step S112, the evaluation unit 12 evaluates the similarity between the objects included in the object set OC and identifies one or more similar objects similar to the object to be predicted. The object to be predicted may be specified by a user operation, or the evaluation unit 12 may select the object to be predicted from the object set OC based on a predetermined selection condition.

[0051] In this example, the evaluation unit 12 evaluates the similarity between objects using the evaluation model M2 and outputs a set of similar objects similar to the object to be predicted. The similarity evaluated by the evaluation unit 12 may be the similarity between two objects or the similarity between three or more objects. As an example, the evaluation unit 12 identifies an object whose similarity to the object to be predicted is equal to or greater than a predetermined threshold as a similar object. Also, as an example, the evaluation unit 12 may identify an object belonging to the same cluster as the object to be predicted as a similar object. Further, the evaluation unit 12 may assign weight information corresponding to the similarity as additional information to the similar object.

[0052] (Step S113) In step S113, the prediction unit 13 predicts the similar label using the prediction model M1. As an example, the prediction unit 13 uses, as the similar label, the label obtained by inputting the feature amounts of the similar objects to the prediction model M1, that is, the label to be assigned to the similar objects. A score representing the reliability of the label may be added as additional information to the similar label assigned to the similar objects. As an example, the score assigned to the similar label is the score output from the prediction model M1.

[0053] Also, a single label value may be assigned to one similar object, or a plurality of label values may be assigned to one similar object. As an example, the prediction unit 13 may determine, as the similar label, one or a plurality of labels whose reliability satisfies a predetermined condition among the plurality of labels predicted by the prediction model M1. As an example, the predetermined condition is a condition such that the reliability ranks among the top K (K is a natural number of 1 or more).

[0054] (Step S114) In step S114, the prediction unit 13 determines the label to be assigned to the object to be predicted by referring to the similar labels. As an example, the prediction unit 13 performs statistical processing (such as majority vote, average, etc.) on a plurality of similar labels, and determines, as the label to be assigned to the object to be predicted, the one among the plurality of similar labels whose frequency satisfies a predetermined condition. Here, the predetermined condition is, for example, the one with the highest frequency, the frequency being equal to or higher than a threshold value, the frequency being included in the top K (K is a natural number of 0 or more), and the like.

[0055] Also, in step S114, the evaluation unit 12 may calculate a score for each of the similar labels, and the prediction unit 13 may further refer to the scores calculated by the evaluation unit 12 to determine the label to be assigned to the object to be predicted. In this case, in step S114, the similar labels and scores predicted by the prediction unit 13 in step S113 are passed to the evaluation unit 12, the evaluation unit 12 calculates the scores for each similar label in consideration of the similarity degree, the scores calculated by the evaluation unit 12 are passed to the prediction unit 13, and the prediction unit 13 determines the label by referring to these scores. As an example, the evaluation unit 12 may use the scores for each label output by the prediction model M1 as the scores for the similar labels as they are, or may calculate the scores for the similar labels by referring to the scores for each label output by the prediction model M1. Also, as an example, the evaluation unit 12 may calculate the scores by referring to the additional information assigned to the similar objects. The scores calculated by the evaluation unit 12 may be, as an example, values representing the ranks of the similarity degree or reliability. Also, the evaluation unit 12 may calculate the scores for the similar labels by referring to both the scores output by the prediction model M1 and the additional information assigned to the similar objects. In other words, the scores calculated by the evaluation unit 12 for each of the similar labels may be values corresponding to the reliability regarding the labels, or may be values corresponding to the similarity degree between the object to be predicted and the similar objects.

[0056] In this case, as an example, the prediction unit 13 may determine, as the label to be assigned to the object to be predicted, a similar label whose total score satisfies a predetermined condition. Here, the predetermined condition is, for example, that the total score is the highest, the total score is equal to or greater than a threshold value, or the total score is included in the top K (K is a natural number of 0 or more).

[0057] Also, when information indicating the rank is given as additional information, the prediction unit 13 may, as an example, obtain the MRR (Mean Reciprocal Rank) for each label from the ranks for each similar object, and determine, as the label to be assigned to the object to be predicted, the label with the largest MRR.

[0058] By executing the information processing method S100 by the information processing apparatus 1A, the label to be assigned to one object to be predicted is determined. The information processing apparatus 1A may not only determine the label for one object to be predicted, but also repeatedly execute the information processing method S100 to determine the labels for a plurality of objects. For example, after determining the label by executing the information processing method S100 with the object OBJ_A as the object to be predicted, the information processing method S100 may be executed with another object OBJ_B as the object to be predicted using the determined label of the object OBJ_A to determine the label of the object OBJ_B.

[0059] (Specific example of information processing method S100) FIG. 5 is a diagram for explaining a specific example of the processing executed by the evaluation unit 12 and the prediction unit 13 in the information processing method S100. Note that the arrows in the figure simply indicate the direction of the flow of certain data and do not exclude bidirectionality. In the example of FIG. 5, the evaluation unit 12 identifies the set G11 of similar objects of the object OBJ_A (step S112). The set G11 includes similar objects OBJ_B to OBJ_E. Similar labels LBL_1, LBL_2, LBL_1, and LBL_1 are respectively assigned to the similar objects OBJ_B, OBJ_C, OBJ_D, and OBJ_E (step S113).

[0060] The prediction unit 13 determines the label to be assigned to the object OBJ_A by referring to the similar labels LBL_1 and LBL_2 assigned to the similar objects OBJ_B to OBJ_E (step S114). As an example, the prediction unit 13 determines the similar label LBL_1 with the highest frequency among the similar labels included in the similar label group G21, which is a set of similar labels, as the label to be assigned to the object OBJ_A.

[0061] (Flow of the information processing method S200) FIG. 6 is a flowchart showing the flow of an information processing method S200, which is an example of an information processing method executed by the information processing apparatus 1A. In the information processing method S200, the prediction unit 13 corrects the pre-correction label of the object to be predicted by referring to the similar labels. The information processing method S200 includes steps S211 to S213 in addition to steps S111 to S113. Note that the description of the content already explained will not be repeated. Also, the steps included in the information processing method S200 may be executed in parallel or in a different order. For example, the process of step S211 may be executed before the process of step S112.

[0062] (Step S211) In step S211, the prediction unit 13 predicts the pre-correction label of the object to be predicted by the prediction model M1. More specifically, the prediction unit 13 predicts the pre-correction label by inputting the feature amount of the object to be predicted into the prediction model M1. A score representing the confidence level of the label may be added as additional information to the pre-correction label.

[0063] (Step S212) In step S212, the evaluation unit 12 extracts one or more similar labels from the plurality of similar labels assigned to the plurality of similar objects. As an example, the evaluation unit 12 extracts the similar labels that appear a predetermined number of times or more from among the plurality of similar labels. At this time, the evaluation unit 12 may extract the similar labels with reference to the additional information of the similar labels. As an example, the evaluation unit 12 may extract the similar labels whose MRR is equal to or greater than the threshold value. However, the method of extracting the similar labels is not limited to the examples described above. The evaluation unit 12 may extract the similar labels by other methods. Hereinafter, the similar labels extracted by the evaluation unit 12 are also referred to as "correction candidate labels". Also, a set of one or more correction candidate labels is also referred to as a "correction candidate label set". The number of correction labels included in the correction candidate label set may be one or more than one.

[0064] (Step S213) In step S213, the prediction unit 13 determines, as the label to be assigned to the object to be predicted, the corrected label obtained by correcting the pre-correction label with reference to the similar labels. As an example, when the pre-correction label is included in the set of similar labels, the prediction unit 13 determines the pre-correction label as the corrected label as it is. On the other hand, when the pre-correction label is not included in the set of similar labels, the prediction unit 13 determines, as the corrected label, the similar label that satisfies a predetermined condition from among the similar labels. The predetermined condition is, for example, the condition that the frequency is the highest, the frequency is equal to or greater than the threshold value, or the frequency rank is included in the top K.

[0065] At this time, in step S213, the prediction unit 13 may determine the corrected label by comparing the one or more similar labels extracted in step S212 with the label before correction. As an example, when the label before correction is included in the set of candidate labels for correction, the prediction unit 13 sets the label before correction as the label after correction. On the other hand, when the label before correction is not included in the set of candidate labels for correction, the prediction unit 13 determines, as the label after correction, a candidate label for correction that satisfies a predetermined condition from among the candidate labels for correction. The predetermined condition is, for example, the condition that the frequency is the highest, the frequency is equal to or greater than a threshold value, or the frequency rank is included in the top K.

[0066] In addition, the prediction unit 13 may determine the label to be assigned to the object to be predicted by referring to the additional information of the candidate label for correction. As an example, when the label before correction is not included in the set of candidate labels for correction, the prediction unit 13 calculates the MRR of the candidate labels for correction and may determine, as the label to be assigned to the object to be predicted, the candidate label for correction with the largest MRR.

[0067] By executing the information processing method S200, the information processing apparatus 1A determines the label to be assigned to one object to be predicted. The information processing apparatus 1A may repeatedly execute the information processing method S200 to determine labels for a plurality of objects.

[0068] FIG. 7 is a diagram schematically showing the processes executed by the evaluation unit 12 and the prediction unit 13 in the information processing method S200. Note that the arrows in the figure simply indicate the direction of the flow of certain data and do not exclude bidirectionality. In the example of FIG. 7, the prediction unit 13 predicts the labels of the objects OBJ_1, OBJ_2, and OBJ_3 included in the object set OC by inputting the feature amounts x1, x2, and x3 of the objects OBJ_1, OBJ_2, and OBJ_3 into the prediction model M1. By the prediction model M1, the labels of the objects OBJ_1, OBJ_2, and OBJ_3 are predicted to be LBL_A, LBL_B, and LBL_A, respectively.

[0069] Further, the evaluation unit 12 evaluates the similarity relationship between the objects included in the object set OC and identifies the similar objects of the object to be predicted. The prediction unit 13 refers to the similarity labels assigned to the similar objects identified by the evaluation unit 12 and determines the label to be assigned to the object to be predicted. In the example of FIG. 7, the labels of the objects OBJ_1 and OBJ_3 remain unchanged as "LBL_A", but the label of the object OBJ_2 is changed from "LBL_B" to "LBL_A".

[0070] (Specific example of the information processing method S200) FIG. 8 is a diagram for explaining a specific example of the processing executed by the evaluation unit 12 and the prediction unit 13 in the information processing method S200. Note that the arrows in the figure simply indicate the direction of the flow of certain data and do not exclude bidirectionality. In the example of FIG. 8, the evaluation unit 12 identifies a set G11 of similar objects of the object OBJ_A (S112). The set G11 includes similar objects OBJ_B to OBJ_E. Similar labels LBL_1, LBL_2, LBL_1, and LBL_1 are assigned to the similar objects OBJ_B, OBJ_C, OBJ_D, and OBJ_E, respectively (step S113).

[0071] Also, in the example of FIG. 8, the prediction model M1 predicts the label LBL_3 as the label of the object OBJ_A (step S211). The prediction unit 13 extracts a group G31 of similar labels including the similar labels LBL_1 and LBL_2 from the group G21 of similar labels (step S212) and determines the label to be assigned to the object OBJ_A from the extracted group G31 of similar labels (step S213). In the example of FIG. 8, the prediction unit 13 determines the similar label LBL_1 included in the group of similar labels as the label to be assigned to the object OBJ_A to be predicted.

[0072] (Flow of the information processing method S300) FIG. 9 is a flowchart showing the flow of an information processing method S300 which is an example of the information processing method executed by the information processing apparatus 1A. The information processing method S300 includes steps S311 to S313 in addition to steps S111 to S113 and step S211. Note that the description of the content already described will not be repeated. Also, the steps included in the information processing method S300 may be executed in parallel or in a different order. For example, the process of step S211 may be executed before step S112.

[0073] (Step S311) In step S311, the prediction unit 13 refers to a plurality of similarity labels assigned to each of the plurality of similar objects and the scores of the respective similarity labels, and rearranges the plurality of similarity labels assigned to the plurality of similar objects. As an example, the prediction unit 13 calculates the value of MRR from the rank (rank indicated by additional information) for each similar object, and rearranges the similarity labels using the calculated value. Note that the value used by the prediction unit 13 for rearranging the similarity labels is not limited to MRR, and other values may be used. For example, the prediction unit 13 may calculate the average value of the scores of the similarity labels (or similar objects) and perform the rearrangement using the calculated average value. Also, for example, the prediction unit 13 may extract the top K similarity labels for each similar object as a set of top labels, and rearrange the similarity labels according to the frequencies included in these sets of top labels.

[0074] In step S311, the prediction unit 13 may further rearrange a plurality of similar labels assigned to a plurality of similar objects with reference to the hierarchical relationship between the plurality of similar labels. The hierarchical relationship may be provided by a predetermined database, or the prediction unit 13 may generate the hierarchical relationship between similar labels from data such as objects. As an example of a method for generating a hierarchical relationship, the method described in the literature of "Wu, Wentao, et al. 'Probase: A probabilistic taxonomy for text understanding.' Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 2012" may be used. In this case, the prediction unit 13 may rearrange the similar labels so that those in the higher hierarchy have higher ranks.

[0075] Specifically, for example, when the plurality of similar labels predicted by the prediction unit 13 include similar labels of "person", "president", "politician", and "businessman", and it is assumed that the hierarchical relationship is such that the similar label of "person" is at a higher hierarchy than "president", "politician", and "businessman". In this case, as an example, the prediction unit 13 rearranges the similar labels so that the rank of "person" is higher than the ranks of "president", "politician", and "businessman".

[0076] (Step S312) In step S312, the prediction unit 13 rearranges a plurality of pre-modification labels related to the target object with reference to the plurality of pre-modification labels related to the target object and the scores of each pre-modification label. As an example, the prediction unit 13 rearranges the pre-modification labels in descending order of the added scores. Note that the rearrangement method by the prediction unit 13 is not limited to the above-described example, and other methods may be used. Also, in step S312, the prediction unit 13 may perform rearrangement with reference to the hierarchical relationship between the plurality of labels, similarly to step S311.

[0077] (Step S313) In step S313, the prediction unit 13 determines, as the post-correction label, the pre-correction label included in the top M-th (M is a natural number) similar labels assigned to a plurality of similar objects among the top N-th (N is a natural number) pre-correction labels regarding the target object.

[0078] By the information processing method S300, the information processing apparatus 1A can determine more accurately a plurality of labels to be assigned to one prediction target object.

[0079] However, in step S313, the prediction unit 13 may determine the post-correction label by referring to the similar labels without referring to the pre-correction label. In this case, as an example, the prediction unit 13 may determine, as the post-correction label, the set of the top M-th similar labels sorted in step S311.

[0080] (Flow of information processing method S400) FIG. 10 is a flowchart showing the flow of an information processing method S400 which is an example of the information processing method executed by the information processing apparatus 1A. In the information processing method S400, the evaluation unit 12 outputs a graph or a hypergraph representing the similarity relationship between objects, and the prediction unit 13 determines the label to be assigned to the prediction target object using the graph or the hypergraph. The information processing method S400 includes steps S401 to S403 in addition to steps S111 and S113. Note that the description of the content already explained will not be repeated.

[0081] (Step S401) In step S401, the evaluation unit 12 identifies one or more similar objects by outputting a graph representing the similarity relationship between objects. The graph output by the evaluation unit 12 is, for example, a graph or hypergraph having objects as nodes and edges or hyperedges connecting the nodes whose similarity has been evaluated. More specifically, the graph / hypergraph is, for example, a graph having edges / hyperedges between objects having a similarity relationship and no edges / hyperedges between objects having no similarity relationship. Further, weight information corresponding to the similarity between the corresponding nodes may be assigned to these edges or hyperedges.

[0082] (Step S402) In step S402, the prediction unit 13 extracts one or more similar objects from the object set OC with reference to the graph output by the evaluation unit 12. The prediction unit 13, for example, extracts one or more similar objects existing within a predetermined number of hops from the object to be predicted with reference to the graph output by the evaluation unit 12. The evaluation unit 12, for example, extracts similar objects connected to the object to be predicted via up to k edges / hyperedges.

[0083] (Step S403) In step S403, the prediction unit 13 determines the label to be assigned to the object to be predicted with reference to the similarity labels assigned to each of the one or more extracted similar objects.

[0084] However, the method by which the prediction unit 13 determines the label is not limited to the above-described example. As an example, the prediction unit 13 may determine the label using a neural network (graph neural network) capable of performing calculations considering the structure of a graph or a hypergraph. As the graph neural network, for example, the neural network described in the literature of "Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." European Semantic Web Conference. Springer, Cham, 2018" or "Feng, Yifan, et al. "Hypergraph neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019" may be used.

[0085] In the learning of the graph neural network, while giving the feature amount of the object corresponding to the node corresponding to each object, and giving, as feature amounts, the degree of similarity between the objects corresponding to each edge / hyperedge, etc., the model such as the graph neural network is trained to predict the label for each node. In this learning, a part or all of the pre-modification labels may be used as training data for the labels to be predicted by the graph neural network. In this case, the prediction unit 13 determines the label of the prediction target object corresponding to each node based on the label predicted by the above model.

[0086] (Effect of the information processing apparatus 1A) As described above, in the information processing apparatus 1A according to the present exemplary embodiment, a configuration is adopted in which scores are calculated for each of the similar labels, and the labels to be assigned to the objects to be predicted are determined by further referring to the calculated scores. By calculating scores according to the reliability of similar labels or the similarity of similar objects or the like in the information processing apparatus 1A according to the present exemplary embodiment, according to the information processing apparatus 1A according to the present exemplary embodiment, in addition to the effects achieved by the information processing apparatus 1 according to the first exemplary embodiment, an effect that labels can be determined taking into account the reliability of similar labels or the like can be obtained.

[0087] Further, in the information processing apparatus 1A according to the present exemplary embodiment, a configuration is adopted in which the label before correction of the object to be predicted is predicted by a prediction model and corrected by referring to similar labels. Therefore, according to the information processing apparatus 1A according to the present exemplary embodiment, in addition to the effects achieved by the information processing apparatus 1 according to the first exemplary embodiment, an effect that the label before correction can be corrected with high accuracy can be obtained.

[0088] 〔Exemplary Embodiment 3〕 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first or second exemplary embodiment are denoted by the same reference numerals, and the description thereof will not be repeated.

[0089] The information processing apparatus 1B according to the present exemplary embodiment classifies entities in a sentence written in a natural language. An entity is a character string representing a specific concept or thing, and as an example, it is a proper noun or a common noun. As an example, the entities in the sentence "Mr. Joe Taylor is the president of the United States of America." are "Mr. Joe Taylor", "the United States of America", and "president". Entities are classified into classes. A class indicates the classification result of an entity, and for example, it is "person", "company", and "country". One or more classes are annotated for one entity. In other words, a plurality of classes may be annotated for one entity.

[0090] <Configuration of Information Processing Apparatus 1B> FIG. 11 is a block diagram showing the configuration of information processing apparatus 1B. Information processing apparatus 1B includes a control unit 10A, a storage unit 20A, an input / output unit 30A, and a communication unit 40A. The acquisition unit 11 of the control unit 10A includes a reception unit 111B and an entity extraction unit 112B. Further, the evaluation unit 12 includes an identity evaluation unit 121B. Also, the storage unit 20A stores a prediction model M1B in addition to the object set OC and the evaluation model M2.

[0091] (Reception Unit 111B) The reception unit 111B receives a set of sentences. The set of sentences includes one or more sentences. As an example, the reception unit 111B may receive a set of sentences from another device via the communication unit 40A. Also, as an example, the reception unit 111B may acquire a set of sentences input via the input / output unit 30A. Further, the reception unit 111B may acquire a set of sentences by reading the set of sentences from the storage unit 20A or an externally connected storage device.

[0092] (Entity Extraction Unit 112B) The entity extraction unit 112B extracts a plurality of entities from the set of sentences. As an example, the entity extraction unit 112B performs natural language processing (such as morphological analysis, N-gram analysis, etc.) on the sentences included in the set of sentences, and extracts the entities included in each sentence. More specifically, as an example, the entity extraction unit 112B performs syntactic analysis on the sentence, and extracts a character string that matches a predetermined grammar pattern, such as a noun phrase or an adjective phrase, as an entity. Alternatively, the entity extraction unit 112B may extract a character string obtained by collating with a character string in a predetermined dictionary as an entity. Alternatively, the entity extraction unit 112B may extract an entity using the technique described in the document "Shang, Jingbo, et al. 'Automated phrase mining from massive text corpora,' IEEE Transactions on Knowledge and Data Engineering 30.10 (2018): 1825-1837".

[0093] (Acquisition unit 11) The acquisition unit 11 acquires, as a set of objects, a set of pairs of a plurality of entities extracted by the entity extraction unit 112B and the sentences from which the plurality of entities are extracted. An object is represented, as an example, by feature amounts such as a character string representing a sentence and the position of an entity in the character string. For example, when the character string of sentence d is "Mr. Joe Taylor is the president of the United States of America." and the entity is "Mr. Joe Taylor", the feature amount is represented, as an example, by (sentence d, <from the 1st character to the 8th character>).

[0094] In this exemplary embodiment, entities that appear in different sentences are treated as different objects even if they are character strings representing the same entity. For example, if there are two sentences "Mr. Joe Taylor was elected." and "Mr. Joe Taylor is the president.", objects are created for these two character strings "Mr. Joe Taylor" respectively.

[0095] Note that the set of objects acquired by the acquisition unit 11 is not limited to the above-described example. As an example, the set of objects may be a plurality of entities extracted by the entity extraction unit 112B.

[0096] (Identity evaluation unit 121B) The identity evaluation unit 121B evaluates the identity between the objects included in the set of objects, and identifies an object identical to the object to be predicted as a similar object. As an example, the identity evaluation unit 121B refers to the feature amounts of a plurality of objects and evaluates whether the entities corresponding to the objects refer to the same entity (thing or event).

[0097] Specifically, the identity evaluation unit 121B evaluates the identity between the objects by the following method. First, the identity evaluation unit 121B refers to the feature amounts of the objects and acquires the character strings of the entities corresponding to the objects. The character strings of the entities can be acquired from the sentences and string positions that are the feature amounts. The identity evaluation unit 121B sets the similarity between the objects with the same acquired character strings to "1", and sets the similarity between the objects with different character strings to "0".

[0098] However, the method by which the identity evaluation unit 121B evaluates the identity between the objects is not limited to the above-described example, and other methods may be used. As an example, the identity evaluation unit 121B may specify the instances in the knowledge base corresponding to the entities corresponding to each object by using the method described in the literature of "Wu, Ledell, et al., "Scalable zero-shot entity linking with dense entity retrieval," arXiv preprint arXiv:1911.03814 (2019)". In this case, as an example, the identity evaluation unit 121B sets the similarity between the objects for which the same instance is specified to "1", and sets the similarity between the objects for which the same instance is not specified to "0".

[0099] (Prediction Model M1B) The prediction model M1B is a model for predicting the label of an object. In this exemplary embodiment, the label assigned to an object is a set of classes into which the object is classified. The class can be represented, for example, by a character string or an integer ID. For example, when the feature amount of an object is the above-mentioned (sentence d, <the 1st character to the 8th character>), the label is, as an example, a set of classes such as {person, president, politician, male, father, American}.

[0100] The prediction model M1B is, as an example, a language model constructed by unsupervised learning. In this case, the language model is, as an example, a model that outputs the accuracy as a natural language sentence for the input word sequence. Also, by using such a language model, it is possible to predict a word that complements the input word sequence. A word that complements a word sequence is a word that, when the word is complemented to the word sequence, the resulting word sequence can become a natural language sentence. As the prediction model M1B, for example, the model described in the literature of "Devlin, Jacob, et al., "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805 (2018)" can be used.

[0101] <Flow of the information processing method by the information processing apparatus 1B> The flow of the information processing method S500, which is an example of the information processing method executed by the information processing apparatus 1B configured as described above, will be described with reference to FIG. 12. FIG. 12 is a flowchart showing the flow of the information processing method S500. Note that the description of the content already described will not be repeated.

[0102] (Steps S501·S502) In step S501, the reception unit 111B receives a set of sentences. In step S502, the entity extraction unit 112B extracts a plurality of entities from the set of sentences. Specifically, for example, the entity extraction unit 112B extracts "Mr. Joe Taylor", "the United States of America", and "president" as entities from the sentence "Mr. Joe Taylor is the president of the United States of America."

[0103] (Step S503) In step S503, the acquisition unit 11 acquires, as a set of objects, pairs of the plurality of entities extracted by the entity extraction unit 112B and the sentences from which the plurality of entities were extracted.

[0104] (Step S504) In step S504, the identity evaluation unit 121B evaluates the identity between the objects included in the set of objects, and identifies an object identical to the object to be predicted as a similar object.

[0105] (Step S505) In step S505, the prediction unit 13 determines the label to be assigned to the object to be predicted by referring to the similar labels assigned to the similar objects. An example of the method for determining the label of the prediction unit 13 according to this exemplary embodiment will be described below.

[0106] First, the prediction unit 13 predicts the pre-modification label of the object to be predicted using the prediction model M1B. First, the prediction unit 13 creates a string with a missing part of the character string that is an entity from the feature amount of the object. As an example, the prediction unit 13 creates from the sentence d "Mr. Joe Taylor is the president of the United States of America." a string such as " <mask>creates a string that says, "is the President of the United States of America." In this string, " <mask>" indicates omission. Next, the prediction unit 13 treats the class name as a word and calculates, using the prediction model M1B, the confidence levels that the respective class names enter the omitted part as scores. As an example, as the confidence levels for the respective classes of "person" and "country", the prediction unit 13 calculates the score that "person" enters the omitted part as "0.9", and also calculates the score that "country" enters the omitted part as "0.1".

[0107] Note that when creating a string with a part of the entity string omitted, the prediction unit 13 may not only simply omit the entity, but also modify a part of the sentence so that the class name is more likely to enter the omitted part. For example, from sentence d, the prediction unit 13 may change " <mask>It may generate a string such as "is the president of the United States of America." As long as the text is modified so that the class name is easier to enter due to the missing part, the prediction unit 13 does not necessarily have to leave out the part of the string that is an entity. For example, from text d, the prediction unit 13 can extract "such as Mr. Joe Taylor" <mask>It may generate a string that says, "is the President of the United States of America."

[0108] As described above, the prediction unit 13 uses the prediction model M1B to predict a set of pairs of class names and scores as the pre-modification labels. The similar labels attached to similar objects are also predicted using the prediction model M1B in the same way as the pre-modification labels. That is, in this exemplary embodiment, the similar label is a set of pairs of class names and scores. The similar label may be predicted by the prediction unit 13, or may be predicted by another device other than the information processing apparatus 1B.

[0109] The prediction unit 13 determines the pre-modification labels with reference to the similar labels. The method for determining the pre-modification labels using the similar labels is the same as the method described in the above exemplary embodiment 2.

[0110] As an example, the prediction unit 13 determines the pre-modification labels by the above-described information processing method S300. In this case, more specifically, the prediction unit 13 calculates the ranks of the classes ("country", "person", etc.) included in the similar labels for each of the similar objects OBJ_B, OBJ_C, OBJ_D, OBJ_E of the object OBJ_A to be modified, calculates the MRR of each class from the obtained ranks, and rearranges the similar labels in the order of the MRR values (step S311). Note that the value used by the prediction unit 13 for rearranging the classes is not limited to the MRR, and other values may be used. For example, the prediction unit 13 may rearrange the classes using the average value of the scores corresponding to each class.

[0111] Next, the prediction unit 13 performs a process of sorting a plurality of classes included in the pre-modification label based on the scores (step S312). Further, the prediction unit 13 determines, as the post-modification label, a set of the top M classes sorted in step S311 among the top N classes sorted in step S312 (step S313). However, the method for determining the post-modification label is not limited to this, and other methods may be used. As an example, the prediction unit 13 may determine, as the post-modification label, a set of the top M classes sorted in step S311 without referring to the pre-modification label.

[0112] Also, in steps S311 and S312, as described above, the prediction unit 13 may sort a plurality of classes with reference to the hierarchical relationship between the classes. For example, as sub-classes of the class "person", "president", "politician", "businessman", etc. can be cited. In this case, as an example, the prediction unit 13 replaces the score of each class with the maximum value among the scores of the sub-classes of that class and then sorts them. Alternatively, as an example, the prediction unit 13 may perform sorting so that the upper class comes before the lower class.

[0113] FIG. 13 is a diagram showing a specific example of the hierarchical relationship of classes and the sorting by the prediction unit 13. In FIG. 13, the hierarchical relationship TC1 shows that there are classes "president" and "businessman" below the class "person". Also, in FIG. 13, the pre-modification label includes classes "president", "city", "businessman", "person", and the scores of each class are calculated as "0.9", "0.7", "0.5", "0.3", respectively.

[0114] In the class hierarchical relationship TC1, since the class "person" is higher than the class "president", the prediction unit 13 changes the score of the class "person" from "0.3" to "0.9" and performs sorting so that the rank of the class "person" is higher than the rank of the class "president".

[0115] (Effect of the information processing apparatus) As described above, in the information processing apparatus 1B according to the present exemplary embodiment, a plurality of entities are extracted from a text set, and a label to be assigned to an object including the extracted entities is determined. Thus, according to the information processing apparatus 1B according to the present exemplary embodiment, for the entities extracted from the text, the label to be assigned to the object including the entity can be determined with high accuracy.

[0116] Further, in the information processing apparatus 1B according to the present exemplary embodiment, a configuration is adopted in which the identity between objects is evaluated, and an object identical to the object to be predicted is specified as a similar object. Therefore, according to the information processing apparatus 1B according to the present exemplary embodiment, by evaluating the identity between objects, the label to be assigned to the object to be predicted can be determined with higher accuracy.

[0117] 〔Exemplary Embodiment 4〕 A fourth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to third exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0118] FIG. 14 is a block diagram showing the configuration of an information processing apparatus 1C according to the present exemplary embodiment. The information processing apparatus 1C includes a control unit 10A, a storage unit 20A, an input / output unit 30A, and a communication unit 40A. The control unit 10A includes a display unit 15C in addition to an acquisition unit 11, an evaluation unit 12, and a prediction unit 13.

[0119] The display unit 15C displays various screens by outputting data representing a display screen to a display device connected to the input / output unit 30A. As an example, the display device includes a liquid crystal display or a projector. The display unit 15C, as an example, displays an object to be predicted. The display unit 15C also displays at least one of the similar objects or at least one of the similar labels.

[0120] FIG. 15 is a diagram showing a screen SC11 which is an example of a screen displayed by the display unit 15C. The screen SC11 includes a first region a111 and a second region a112. In the first region a111, sentences and entities included in the object to be predicted are displayed. Also, in the first region a111, classes included in the label determined by the prediction unit 13 and corresponding to each entity are displayed. Specifically, “person” and “president” are displayed as the label LBL11 of the entity E11 named “Mr. Joe Taylor”, and “country” and “organization” are displayed as the label LBL12 of the entity E12 named “United States of America”. Also, “job title” is displayed as the label LBL13 of the entity E13 named “president”.

[0121] In the second region a112, sentences and entities included in the similar object are displayed. Also, in the second region a112, classes included in the similar label of the similar object and corresponding to the entities included in the similar object are displayed. For example, “person” and “politician” are displayed as the classes of the entity named “Mr. Joe Taylor”.

[0122] Also, the screen SC11 includes a text box TB11 and a button B11. The text box TB11 is a text box for a user of the information processing apparatus 1C to input a class name using an input device (mouse, keyboard, etc.) connected to the input / output unit 30A. When the user selects an entity with the pointer P11, inputs a character string into the text box TB11, and performs an operation of selecting the button B11, the information processing apparatus 1C adds the class name of the input character string to the label assigned to the object to be predicted. In other words, the user can change the label assigned by the prediction unit 13 to the object to be predicted using the input device connected to the input / output unit 30A.

[0123] Also, the display unit 15C may display the label before correction of the object to be predicted or the label assigned to the target object. Also, at this time, the display unit 15C may display the similar label.

[0124] FIG. 16 is a diagram showing a screen SC21 which is an example of a screen displayed by the display unit 15C. The screen SC21 includes a first region a111, a fourth region a212, and a fifth region a213. In the first region a111, similar to the screen S11, the text and entities included in the object to be predicted are displayed. Also, in the first region a111, the classes included in the label assigned to the object to be predicted (i.e., the label determined by the prediction unit 13), which are the classes corresponding to the respective entities, are displayed.

[0125] In the fourth region a212, the label before correction of the object to be predicted is displayed. In the fifth region a213, similar labels are displayed. Also, on the screen SC21, similar to the screen SC11, a text box TB11 and a button B11 are displayed.

[0126] According to this exemplary embodiment, the user of the information processing apparatus 1C can grasp the object to be predicted and the similar object or similar label by checking the screen illustrated in FIG. 15. Also, the label of the object to be predicted can be changed using the input device.

[0127] Also, according to this exemplary embodiment, by checking the screen illustrated in FIG. 16, the label before correction of the object to be predicted or the label assigned to the object to be predicted and the similar labels can be grasped.

[0128] 〔Example of Realization by Software〕 Some or all of the functions of the information processing apparatuses 1, 1A, 1B, 1C may be realized by hardware such as an integrated circuit (IC chip) or may be realized by software.

[0129] In the latter case, the information processing apparatuses 1, 1A, 1B, and 1C are realized by, for example, a computer that executes instructions of a program which is software for realizing each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 17. Computer C includes at least one processor C1 and at least one memory C2. A program P for operating computer C as information processing apparatuses 1, 1A, 1B, and 1C is recorded in memory C2. In computer C, each function of information processing apparatuses 1, 1A, 1B, and 1C is realized by processor C1 reading and executing program P from memory C2.

[0130] As the processor C1, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination thereof can be used. As the memory C2, for example, a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a combination thereof can be used.

[0131] Note that computer C may further include a RAM (Random Access Memory) for expanding program P at the time of execution or temporarily storing various data. Also, computer C may further include a communication interface for transmitting and receiving data to and from other devices. Also, computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0132] Also, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. As such a recording medium M, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit can be used. The computer C can obtain the program P via such a recording medium M. Also, the program P can be transmitted via a transmission medium. As such a transmission medium, for example, a communication network or a broadcast wave can be used. The computer C can also obtain the program P via such a transmission medium.

[0133] [Supplementary Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope indicated in the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0134] [Supplementary Note 2] Some or all of the above-described embodiments may also be described as follows. However, the present invention is not limited to the aspects described below.

[0135] (Supplementary Note 1) An acquisition means for acquiring a set of objects; An evaluation means for evaluating the similarity between the objects included in the set of objects and specifying one or more similar objects similar to the object to be predicted; A prediction means for determining the label to be assigned to the object to be predicted by referring to a similar label that is a label assigned to each of the one or more similar objects and predicted by a prediction model; An information processing apparatus comprising the above.

[0136] According to the above configuration, even when there is only a single prediction model, the label to be assigned to the object to be predicted can be determined with high accuracy.

[0137] (Appendix 2) The evaluation means calculates a score for each of the similar labels, and the prediction means further refers to the score to determine a label to be assigned to the object to be predicted. The information processing apparatus according to Appendix 1.

[0138] According to the above configuration, it is possible to determine a label to be assigned to the object to be predicted in consideration of the scores of the similar labels.

[0139] (Appendix 3) The prediction means, predicts the label before correction of the object to be predicted by the prediction model, predicts the similar label by the prediction model, and determines, as the label to be assigned to the object to be predicted, a label after correction obtained by correcting the label before correction with reference to the similar label. The information processing apparatus according to Appendix 1 or 2.

[0140] According to the above configuration, it is possible to accurately correct the label before correction of the object to be predicted predicted by the prediction model.

[0141] (Appendix 4) The evaluation means identifies a plurality of similar objects, and the prediction means, extracts one or more similar labels from the plurality of similar labels assigned to the plurality of similar objects, and determines the label after correction by comparing the one or more extracted similar labels with the label before correction. The information processing apparatus according to Appendix 3.

[0142] According to the above configuration, by using the extracted similar labels, it is possible to more accurately correct the label before correction of the object to be predicted predicted by the prediction model.

[0143] (Appendix 5) The prediction means refers to a plurality of similarity labels assigned to each of the plurality of similar objects and scores of each similarity label, rearranges the plurality of similarity labels assigned to the plurality of similar objects, refers to a plurality of pre-modification labels related to the target object and scores of each pre-modification label, rearranges the plurality of pre-modification labels related to the target object, determines, as the post-modification label, a pre-modification label included in the top M-th (M is a natural number) similarity labels assigned to the plurality of similar objects among the top N-th (N is a natural number) pre-modification labels related to the target object; The information processing apparatus according to Supplementary Note 3 or 4.

[0144] According to the above configuration, a plurality of labels to be assigned to a prediction target object can be determined with high accuracy.

[0145] (Supplementary Note 6) The prediction means further rearranges the plurality of similarity labels assigned to the plurality of similar objects with reference to the hierarchical relationship among the plurality of similarity labels, determines, as the post-modification label, a pre-modification label included in the top M-th (M is a natural number) similarity labels assigned to the plurality of similar objects among the top N-th (N is a natural number) pre-modification labels related to the target object; The information processing apparatus according to Supplementary Note 5.

[0146] According to the above configuration, a plurality of labels to be assigned to a prediction target object can be determined with higher accuracy in consideration of the hierarchical relationship.

[0147] (Supplementary Note 7) The evaluation means identifies the one or more similar objects by outputting a graph representing the similarity relationship between objects, The prediction means With reference to the graph, extract one or more similar objects existing within a predetermined number of hops from the object to be predicted, Determine the label to be assigned to the object to be predicted with reference to the similarity labels assigned to each of the one or more extracted similar objects. The information processing apparatus according to any one of Appendices 1 to 6.

[0148] According to the above configuration, the label to be assigned to the object to be predicted can be determined with higher accuracy using a graph representing the similarity relationship between objects.

[0149] (Appendix 8) The acquisition means A reception means for receiving a set of texts, An entity extraction means for extracting a plurality of entities from the set of texts, and The plurality of entities extracted by the entity extraction means, or A set of the plurality of entities extracted by the entity extraction means and the texts from which the plurality of entities are extracted, is acquired as the set of objects. The information processing apparatus according to any one of Appendices 1 to 7.

[0150] According to the above configuration, even when there is only a single prediction model, for entities extracted from texts, the label to be assigned to the object including the entity can be determined with high accuracy.

[0151] (Appendix 9) The evaluation means Evaluates the identity between objects included in the set of objects, and includes an identity evaluation means for identifying an object identical to the object to be predicted as the similar object. The information processing apparatus according to any one of Appendices 1 to 8.

[0152] According to the above configuration, by evaluating the identity between objects, the label to be assigned to the object to be predicted can be determined with higher accuracy.

[0153] (Appendix 10) The object to be predicted, and display means for displaying at least any one of the similar objects or at least any one of the similar labels; The information processing apparatus according to any one of Appendices 1 to 9, comprising the above.

[0154] According to the above configuration, the user of the information processing apparatus can grasp the object to be predicted and the similar object or similar label.

[0155] (Appendix 11) The label before correction of the object to be predicted, or the label assigned to the object, and the similar label, display means for displaying the above; The information processing apparatus according to any one of Items 1 to 9 of the appendix, comprising the above.

[0156] According to the above configuration, the user of the information processing apparatus can grasp the label before correction of the object to be predicted or the label assigned to the object and the similar label.

[0157] (Appendix 12) Obtaining a set of objects, evaluating the similarity between the objects included in the set of objects, and identifying one or more similar objects similar to the object to be predicted, and determining the label to be assigned to the object to be predicted with reference to the similar labels which are the labels assigned to each of the one or more similar objects and predicted by the prediction model. An information processing method including this.

[0158] According to the above information processing method, the same effects as those of the above-described information processing apparatus are achieved.

[0159] (Appendix 13) Cause a computer to perform a process of obtaining a set of objects, evaluate the similarity between the objects included in the set of objects, and identify one or more similar objects similar to the object to be predicted, and determine the label to be assigned to the object to be predicted by referring to the similar labels that are the labels assigned to each of the one or more similar objects and predicted by a prediction model. An information processing program for causing the above to be executed.

[0160] According to the above configuration, the same effects as those of the above-described information processing apparatus are achieved.

[0161] 〔Appendix Item 3〕 Some or all of the above-described embodiments can be further expressed as follows.

[0162] An information processing apparatus including at least one processor, the processor performing an acquisition process of acquiring a set of objects, an evaluation process of evaluating the similarity between the objects included in the set of objects and identifying one or more similar objects similar to the object to be predicted, and a prediction process of determining the label to be assigned to the object to be predicted by referring to the similar labels that are the labels assigned to each of the one or more similar objects and predicted by a prediction model. Note that this information processing apparatus may further include a memory, and a program for causing the processor to perform the acquisition process, the evaluation process, and the prediction process may be stored in this memory. Further, this program may be recorded on a non-transitory tangible computer-readable recording medium.

Explanation of Reference Numerals

[0163] 1, 1A, 1B, 1C Information processing apparatus 10A Control Unit 11 Acquisition Unit 12 Evaluation Unit 13 Prediction Unit 15C Display Unit 20A Memory Unit 30A Input / Output Unit 40A Communication Unit 111B Reception Unit 112B Entity Extraction Unit 121B Identity Evaluation Unit< / mask> < / mask> < / mask> < / mask>

Claims

1. An acquisition means for acquiring a set of objects; An evaluation means for evaluating the similarity between objects included in the set of objects and specifying one or more similar objects similar to the object to be predicted; A prediction means for determining the label to be assigned to the object to be predicted by referring to the similar labels, which are the labels assigned to each of the one or more similar objects and predicted by a prediction model Comprising: The prediction means: Predicts the label before correction of the object to be predicted by the prediction model; Predicts the similar label by the prediction model; Determines, as the label to be assigned to the object to be predicted, the corrected label obtained by correcting the label before correction with reference to the similar label; Performs sorting of the plurality of similar labels assigned to the plurality of similar objects with reference to the plurality of similar labels assigned to each of the plurality of similar objects and the score of each similar label; Performs sorting of the plurality of labels before correction related to the object to be predicted with reference to the plurality of labels before correction related to the object to be predicted and the score of each label before correction; Determines the corrected label by comparing the sorted plurality of similar labels with the sorted plurality of labels before correction An information processing apparatus.

2. The evaluation means calculates a score for each of the similar labels; The prediction means further determines the label to be assigned to the object to be predicted with reference to the score. The information processing apparatus according to claim 1.

3. The prediction means: Among the labels before correction up to the Nth (N is a natural number) for the object to be predicted, determines, as the corrected label, the label before correction included in the similar labels up to the Mth (M is a natural number) for the plurality of similar objects. The information processing apparatus according to claim 1 or 2.

4. The prediction means: Further sorts the plurality of similar labels assigned to the plurality of similar objects with reference to the hierarchical relationship between the plurality of similar labels; Among the labels before correction up to the Nth (N is a natural number) for the object to be predicted, determines, as the corrected label, the label before correction included in the similar labels up to the Mth (M is a natural number) for the plurality of similar objects. The information processing apparatus according to claim 3.

5. The evaluation means identifies the one or more similar objects by outputting a graph representing the similarity relationship between objects, The prediction means refers to the graph, extracts one or more similar objects existing within a predetermined number of hops from the object to be predicted, refers to the similarity labels assigned to each of the one or more extracted similar objects, and determines the label to be assigned to the object to be predicted The information processing apparatus according to any one of claims 1 to 4.

6. The acquisition means includes a reception means for receiving a set of sentences, and an entity extraction means for extracting a plurality of entities from the set of sentences, and the plurality of entities extracted by the entity extraction means, or the plurality of entities extracted by the entity extraction means and the set of the sentences from which the plurality of entities are extracted are acquired as the set of objects The information processing apparatus according to any one of claims 1 to 5.

7. The object to be predicted, and display means for displaying at least any one of the similar objects or at least any one of the similarity labels The information processing apparatus according to any one of claims 1 to 6, comprising.

8. the label before correction of the object to be predicted, or the label assigned to the object to be predicted, and the similarity label display means for displaying The information processing apparatus according to any one of claims 1 to 7, comprising.

9. One or more processors acquire a set of objects, the one or more processors evaluate the similarity between the objects included in the set of objects, and identify one or more similar objects similar to the object to be predicted, the one or more processors determine the label to be assigned to the object to be predicted by referring to the similarity label which is the label assigned to each of the one or more similar objects and predicted by a prediction model, and in the step of determining the label to be assigned to the object to be predicted, the one or more processors predict the label before correction of the object to be predicted by the prediction model, predict the similarity label by the prediction model, Determine the corrected label obtained by correcting the pre-correction label with reference to the similar label as the label to be assigned to the object to be predicted, Refer to the plurality of similar labels assigned to each of the plurality of similar objects and the scores of each similar label, and rearrange the plurality of similar labels assigned to the plurality of similar objects, Refer to the plurality of pre-correction labels related to the object to be predicted and the scores of each pre-correction label, and rearrange the plurality of pre-correction labels related to the object to be predicted, Determine the corrected label by comparing the rearranged plurality of similar labels with the rearranged plurality of pre-correction labels Information processing method.

10. On a computer, A process of acquiring a set of objects, A process of evaluating the similarity between objects included in the set of objects and identifying one or more similar objects similar to the object to be predicted, A process of determining the label to be assigned to the object to be predicted with reference to the similar labels that are the labels assigned to each of the one or more similar objects and are predicted by a prediction model An information processing program for causing the computer to execute, The process of determining is Predict the pre-correction label of the object to be predicted by the prediction model, Predict the similar label by the prediction model, Determine the corrected label obtained by correcting the pre-correction label with reference to the similar label as the label to be assigned to the object to be predicted, Refer to the plurality of similar labels assigned to each of the plurality of similar objects and the scores of each similar label, and rearrange the plurality of similar labels assigned to the plurality of similar objects, Refer to the plurality of pre-correction labels related to the object to be predicted and the scores of each pre-correction label, and rearrange the plurality of pre-correction labels related to the object to be predicted, Determine the corrected label by comparing the rearranged plurality of similar labels with the rearranged plurality of pre-correction labels Information processing program.

Images