Information processing device and information processing method for machine learning

Abstract

To provide an information processing device and information processing method for machine learning, which allow for accurately attaching multiple labels to data.SOLUTION: An information processing device 10 disclosed herein comprises a touch point generation unit and a learning unit. The touch point generation unit generates a touch point consisting of multiple question sets. Each of the multiple question sets consists of a question and multiple labels indicative of selectable answers to the question. The learning unit trains multiple learning models using the multiple question sets.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method for machine learning.

Background Art

[0002] In recent years, machine learning techniques have been used to solve various problems. Machine learning is performed by using a learned learning model (machine learning model) that can solve general or specific tasks. The learning model is learned using a large amount of teacher data, where the teacher data is composed of data and labels for the data. The operation of attaching labels to data is called annotation, and conventionally, the annotation has been performed manually by humans. However, for preparing a large amount of teacher data, such manual work is inefficient and there is a possibility that the quality cannot be guaranteed.

[0003] In response to such problems, free or paid annotation tools configured to perform automatic labeling have become widespread in order to perform annotation efficiently. Also, as a technique for guaranteeing the quality of the generated teacher data (raising the accuracy of annotation), Patent Document 1 discloses an information processing apparatus for evaluating the accuracy of the generated teacher data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the conventional annotation technique, it is expected to generate labelings for data with high accuracy and efficiency, but it has not been configured to attach a plurality of types of labels to data with high accuracy.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide a mechanism for accurately attaching a plurality of labels to data.

Means for Solving the Problems

[0007] In order to solve the above problems, one aspect of an information processing apparatus according to the present invention is a creation unit that creates a plurality of question sets, wherein each of the plurality of question sets is composed of a question and a plurality of labels indicating answers selectable for the question, and a learning unit that learns a plurality of learning models using the plurality of question sets.

[0008] In order to solve the above problems, one aspect of an information processing method according to the present invention includes creating a plurality of question sets, wherein each of the plurality of question sets is composed of a question and a plurality of labels indicating answers selectable for the question, and learning a plurality of learning models using the plurality of question sets.

Effects of the Invention

[0009] According to the present invention, a mechanism for accurately attaching a plurality of labels to data is provided. Those skilled in the art will be able to understand the above-described objects, aspects, and effects of the present invention, as well as the objects, aspects, and effects of the present invention not described above, from the embodiments for carrying out the following invention with reference to the descriptions in the accompanying drawings and the claims.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Among the components disclosed below, those having the same function are denoted by the same reference numerals, and the description thereof will be omitted. Note that the embodiments disclosed below are examples of means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. Also, not all combinations of the features described in this embodiment are essential for the solution means of the present invention.

[0012] The information processing apparatus according to the present embodiment creates a plurality of question sets based on a user's input operation. Each question set is composed of a question and a plurality of labels indicating selectable answers to the question. Then, the information processing apparatus uses the plurality of question sets to train a plurality of learning models. Further, the information processing apparatus acquires a data set including a plurality of items to be labeled (annotated), and applies the plurality of items to the plurality of learning models to predict and output labels for the question set.

[0013] [Functional Configuration of Information Processing Apparatus] FIG. 1 shows an example of the functional configuration of an information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 shown in FIG. 1 includes a job management unit 101, a touch point creation unit 102, a data acquisition unit 103, a learning model configuration unit 104, a learning unit 105, a prediction unit 106, a data management unit 107, and a data storage unit 110. The data storage unit 110 is configured to store first learning models 111-1 to nth learning models 111-n (n is a natural number of 2 or more), a labeling rule 112, and annotated data 113.

[0014] The job management unit 101 manages a job including a series of processes when performing annotation on one data set. The touch point creation unit 102 creates a plurality of touch points. In the present embodiment, each touch point is a question set including a question and a plurality of labels (also referred to as values) indicating selectable answers to the question. The data acquisition unit 103 acquires a data set including a plurality of items to be annotated. Specifically, the data acquisition unit 103 acquires a data set composed of a list of items to be annotated. For example, the data acquisition unit 103 acquires the data set based on an input operation by a user or according to a predetermined program. The learning model configuration unit 104 configures a plurality of learning models to be used for labeling items to be annotated.

[0015] The learning unit 105 learns a learning model. For example, the learning unit 105 learns the first learning models 111-1 to nth learning models 111-n. The first learning models 111-1 to nth learning models 111-n are each a learning model learned from past labeled data and correspond to a learning model with weak teachers. The first learning models 111-1 to nth learning models 111-n may be learning models configured to identify attributes of specific items, or may be learning models configured to generally identify attributes of items.

[0016] The prediction unit 106 predicts the label to be attached to the item by applying a plurality of items to be annotated to any one of the learned first learning models 111-1 to nth learning models 111-n. In the present embodiment, the prediction unit 106 predicts the label to be attached to the item by applying the plurality of items to the plurality of learning models configured by the learning model configuration unit 104.

[0017] The data management unit 107 stores the first learning model 111-1 to the nth learning model 111-n learned by the learning unit 105 and the data labeled by the prediction unit 106, that is, the annotated data 113, in the data storage unit 110. Further, the data management unit 107 may store the labeling rule 112 in the data storage unit 110. The labeling rule 112 will be described later.

[0018] [Hardware Configuration of Information Processing Apparatus] Next, an example of the hardware configuration of the information processing apparatus 10 will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 according to the present embodiment can be implemented on a single or multiple, any computer, mobile device, or any other processing platform. Referring to FIG. 2, an example in which the information processing apparatus 10 is implemented on a single computer is shown, but the information processing apparatus 10 according to the present embodiment may be implemented in a computer system including a plurality of computers. The plurality of computers may be connected to be communicable with each other by a wired or wireless network.

[0019] As shown in FIG. 2, the information processing apparatus 10 may include a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an HDD (Hard Disk Drive) 204, an input unit 205, a display unit 206, a communication I / F (interface) 207, a GPU (Graphics Processing Unit) 208, and a system bus 209. The information processing apparatus 10 may also include an external memory. The CPU 201 comprehensively controls the operations in the information processing apparatus 10, and controls each component (202 to 208) via the system bus 209 which is a data transmission path.

[0020] The ROM 202 is a non-volatile memory that stores control programs and the like necessary for the CPU 201 to execute processing. Note that the program may be stored in a non-volatile memory such as the HDD 204 or an SSD (Solid State Drive), or an external memory such as a removable storage medium (not shown). The RAM 203 is a volatile memory and functions as the main memory, work area, etc. of the CPU 201. That is, when executing processing, the CPU 201 loads necessary programs and the like from the ROM 202 into the RAM 203, and realizes various functional operations by executing the programs and the like. The RAM 203 may include the data storage unit 110 shown in FIG. 1.

[0021] The HDD 204 stores various data and various information necessary, for example, when the CPU 201 performs processing using a program. Also, the HDD 204 stores various data and various information obtained, for example, when the CPU 201 performs processing using a program or the like. The input unit 205 is composed of a pointing device such as a keyboard or a mouse. The display unit 206 is composed of a monitor such as a liquid crystal display (LCD). The display unit 206 may function as a GUI (Graphical User Interface) by being configured in combination with the input unit 205. The user input operations (including selection operations, etc.) described below can be performed via the input unit 205 or the GUI.

[0022] The communication I / F 207 is an interface that controls the communication between the information processing apparatus 10 and an external device. The communication I / F 207 provides an interface with a network and executes communication with an external device via the network. Various data, various parameters, etc. are transmitted and received between the external device and the communication I / F 207. In the present embodiment, the communication I / F 207 may execute communication via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network that can be used in the present embodiment is not limited to this, and it may be configured by a wireless network. This wireless network includes wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). In addition, it includes wireless LAN (Local Area Network) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). Furthermore, it includes wireless WAN (Wide Area Network) such as 4G and 5G. Note that the network may connect each device so that they can communicate with each other, and as long as communication is possible, the communication standard, scale, and configuration are not limited to the above. The GPU 208 is a processor specialized for image processing. The GPU 208 can perform predetermined processing in cooperation with the CPU 201.

[0023] At least some of the functions of each element of the information processing apparatus 10 shown in FIG. 1 can be realized by the CPU 201 executing a program. However, at least some of the functions of each element of the information processing apparatus 10 shown in FIG. 1 may operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 201.

[0024] [Annotation Processing] Subsequently, the procedure of the annotation processing according to the present embodiment will be specifically described. FIG. 3 shows a flowchart of the annotation processing executed by the information processing apparatus 10. The information processing apparatus 10 processes the labeling process for one data set as one annotation job (hereinafter referred to as a job).

[0025] First, the job management unit 101 sets the basic information of the job (job name, administrator, etc.) based on the user's input operation (S31). The basic information is set so as not to overlap with other jobs. After setting the basic information of the job, the touch point creation unit 102 creates a plurality of touch points based on the user's input operation (S32). As described above, each touch point (each question set) of the plurality of touch points is composed of a question and a plurality of labels (also referred to as values) indicating selectable answers to the question.

[0026] When the user inputs a touch point, the user can select any one of a plurality of types with different question types. In the present embodiment, the types are three types: "Backfill", "Binomial", and "Identify". "Backfill" is a touch point composed of a question and three or more dictionary values or custom values as answers to the question. The dictionary value can be an attribute of a predetermined item, such as the color of an item to be annotated (e.g., a product) or the type of the item. The custom value can be a value customized by the user (e.g., an attribute of an item). "Binomial" is a touch point composed of a question and two options or custom values as answers to the question. "Identify" is a touch point applied to a use case where different dictionary values are set for each item, and is a touch point composed of a question and a plurality of values as answers to the question. That is, it can be said that "Backfill" takes three or more values (labels), "Binomial" takes two values (labels), and "Identify" is a question type that takes at least two values (labels).

[0027] FIG. 5A shows an example of a screen displayed on the display unit 206 of the information processing apparatus 10 in S32 of FIG. 3. In FIG. 5A, a screen 50 for creating the first touch point (Touchpoint 1) among a plurality of touch points is shown. As described above, the question type (QUESTION TYPE) 501 can be selected from three types: "Backfill", "Binomial", and "Identify". The user can input the content of the question in the input space 503 for the question label (QUESTION LABEL) 502.

[0028] Figure 5B shows an example of screen 51 that is displayed when the user selects "Backfill" on screen 50 shown in Figure 5A. On screen 51, the user can input the content of the question into the input space 503 for the question label 502, and can select a dictionary value 512 or a custom value 513 as the answer options (ANSWER OPTIONS) 511 for the question. Further, on a screen (not shown) that is displayed according to the selection, the user can input the data of the dictionary value or the custom value by manual input, upload, or the like.

[0029] Figure 5C shows an example of screen 52 that is displayed when the user selects "Binomial" on screen 50 shown in Figure 5A. On screen 52, the user can input the content of the question into the input space 503 for the question label 502, and can select any one of YES / NO 521, True / False 522, Agree / Disagree 523, and a custom value 524 as the answer options (ANSWER OPTIONS) 511 for the question.

[0030] Figure 5D shows an example of screen 53 that is displayed when the user selects "Identify" on screen 50 shown in Figure 5A. On screen 53, the user can input the question into the input space 503 for the question label 502 and can set the answer that can be selected for the question.

[0031] As described with reference to Figures 5B to 5D, the user selects at least one of the question types of "Backfill", "Binomial", and "Identify" for each touch point and inputs a plurality of touch points. Thereby, the touch point creation unit 102 can create a plurality of touch points. The plurality of touch points may be touch points of the same question type (for example, "Backfill") or touch points of different question types (for example, "Backfill" and "Binomial").

[0032] Next, the job management unit 101 configures a workflow for annotation (labeling) (S33). In the present embodiment, the job management unit 101 sets a flow regarding human verification (check) after the labeling in S36 based on an input operation by the user. For example, the job management unit 101 sets to perform the verification in one stage (curating) or two stages (curating and review). Curating corresponds to a direct verification operation for the prediction result, and review corresponds to the verification of the curation result. Further, the job management unit 101 may set to perform the verification by one user or a plurality of users. The result of the verification corresponds to the verification result received in S37.

[0033] Next, the data acquisition unit 103 acquires a data set composed of a list of items to be annotated (S34). For example, the data acquisition unit 103 acquires the data set based on an input operation by the user (for example, an instruction for uploading or downloading from the cloud) or according to a predetermined program. The data set includes at least information on a plurality of items to be annotated, and may further include text information such as identification information, description, and brand name of each item (or associated with the item). The information of the item can be an image, a video, or the like. The image or video may be data or information indicating the location of the image or video, such as a URL (Uniform Resource Locator). The format of the data set can be any of data technical languages such as CSV, JSON, and ZIP format.

[0034] The data acquisition unit 103 may assign priorities at multiple levels according to the influence of the data set on the learning model in accordance with the input operation by the user. For example, the user sets "high priority", "medium priority", and "low priority" for the data set. Accordingly, in S36, when the prediction unit 106 performs prediction processing on a data set with a higher priority, the learning unit 105 may perform learning processing with a higher weighting on the verification result of the prediction. Thereby, the learning effect can be improved.

[0035] After acquiring the data set, the learning model configuration unit 104 configures a learning model for labeling in S36 for each of the plurality of touch points created in S32 (S35). The learning model is configured to automatically attach a label to the input data and is also referred to as an auto-labeler. In this sense, the process of S36 corresponds to configuring an auto-labeler. Specifically, in S35, the learning model configuration unit 104 applies (assigns) a learning model to each of the plurality of touch points created in S32 based on the input operation by the user. The learning model may be any one of the learned first learning models 111-1 to nth learning models 111-n stored in the data storage unit 110.

[0036] FIG. 5E shows an example of a screen 54 displayed on the display unit 206 during the process of S35. On the screen 54, learning models (auto labelers) applicable to all touch points of "backfill", "binomial", and "identification" are shown. The learning models 541 to 544 are learned learning models and can be any of the first learning model 111-1 to the nth learning model 111-n. Also, to assist the user in making a selection, below each of the learning models 541 to 544, item categories and item attributes that can be labeled (identified) by the learning model are shown. For example, the learning model 541 is a general-purpose learning model that can identify all item categories and item attributes. The learning model 542 is a learning model that can identify the item category: TV and the item attribute: display type. The learning model 543 is a learning model that can identify the item category: men's clothing and the item attribute; pattern type. The learning model 544 is a learning model that can identify the item category: men's clothing and the item attribute: sleeve type.

[0037] In the example of FIG. 5E, it shows that the user has selected the learning model 543 based on the characteristics of a certain touch point among the multiple touch points created in S32. In response to this, the learning model configuration unit 104 applies the learning model 543 to the touch point. In this way, based on the input operation by the user, the learning model configuration unit 104 applies a learning model to each of the multiple touch points created in S32.

[0038] When the user determines that there is no appropriate learning model for at least any of the plurality of touch points created in S32 among the first learning model 111-1 to the nth learning model 111-n, the user can be instructed to input a labeling rule (labeling function). For example, when it is desired to identify an attribute in a domain different from the attributes of the items that the first learning model 111-1 to the nth learning model 111-n can identify, it can be determined that there is no appropriate learning model. In response to the instruction, the learning model configuration unit 104 displays an input screen for the labeling rule on the display unit 206. Then, the learning model configuration unit 104 configures a labeling rule according to the input operation by the user and applies the labeling rule as the learning model for the touch point.

[0039] Figure 5F shows an example of the input screen 55 for the labeling rule. The screen 55 is a screen for setting a labeling rule for identifying either of the two-valued labels for the touch point having the question type of "binomial" among the plurality of touch points created in S32. The labeling rule can be configured based on the data set acquired in S34 by combining one rule or a plurality of rules. The learning model configuration unit 104 analyzes the data set acquired in S34, creates an input screen for the labeling rule from the information included in the data set, and displays it on the display unit 206. In this example, it is assumed that the data set acquired in S34 includes text information about the item, and the text information includes at least a "Name" part and a "Description" part. The learning model configuration unit 104 generates the screen 55 based on the text information and displays it on the display unit 206.

[0040] The user can set the conditions for the "Name" part for Rule #1 and the conditions for the "Description" part for Rule #2 by performing an input operation on Screen 55, and can also set to attach a predetermined label to the items that meet the conditions. Accordingly, the learning model configuration unit 104 configures a learning model that attaches a predetermined label to the items that meet Rule #1 and Rule #2. The learning model configuration unit 104 can save the labeling rule created in this way as the labeling rule 112 in the data storage unit 110.

[0041] The learning model configured by S35 is either one of the learned first learning models 111-1 to nth learning models 111-n, or a labeling rule set by the user, and is not a learning model pre-learned using all the items of the dataset acquired in S34. That is, since the learning model configured by S35 is not configured to be able to perform complete labeling on all the items of the dataset acquired in S34, it can be called a weakly-supervised model (weakly-supervised algorithm).

[0042] In the processing up to this point, the preparation for labeling is completed. Subsequently, the prediction unit 106 predicts a label for each item included in the dataset acquired in S34 by using the learning models (i.e., multiple learning models) applied to each of the plurality of touch points created in S32 for each of the plurality of touch points (S36). That is, the prediction unit 106 labels each item. The process of S36 is an automatic labeling by the learning model (auto-labeler) configured in S35 and can be referred to as auto-labeling.

[0043] After the labeling of the dataset is completed, the job management unit 101 displays the prediction result (labeling result) on the display unit 206 and presents it to the user (S37). FIG. 5G shows an example of the screen 56 of the prediction result. On the screen 56, the information of the items included in the dataset acquired in S34 is displayed. For example, it is the image of the item 561 and the text information 562 regarding the item. "GTIN" indicates the product identification code of the item 561. "NAME" indicates the name of the item 561. "DESCRIPTION" indicates the description of the item 561. Also, as a result predicted by the learning model configured in S35 for any one of the plurality of touch points created in S32, the screen 56 displays the result of labeling "LED" as the predicted label for the item 561. On the screen 56, in order to indicate that "LED" is the prediction result, "LED" is displayed in a manner different from other labels, and an indicator that it is the prediction result of "PREDICTED" is displayed near "LED". In this way, the job management unit 101 displays the prediction result so that the user can recognize the predicted label among the plurality of labels (values).

[0044] The user can verify the prediction result shown on the screen 56 displayed on the display unit 206 and provide feedback on the verification result. The flow regarding the verification is the flow set in S33. Here, it is assumed that one user verifies it in one step (curating). Curating corresponds to a direct verification operation for the prediction result, and the user provides feedback on the verification result.

[0045] For example, if the user determines that the prediction result shown on screen 56 is correct, the user may not provide feedback on the verification result. If the user determines that the result is incorrect, the user may provide feedback on the verification result indicating the correct (modified) label (in the example of FIG. 5G, any one of "4K", "3D", "CRT", "DLP", "HDR", or a custom value). The verification result to be fed back is not limited to this, and the user may also provide feedback on the verification result indicating that the prediction result is correct when the user determines that the prediction result is correct. The learning unit 105 receives the verification result (S37). The job management unit 101 sequentially causes the display unit 206 to display the prediction results for all items included in the data set acquired in S34, and the learning unit 105 receives the verification results by the user for the prediction results.

[0046] In S33, if the verification flow is set to two stages (curating and reviewing) by one user, one user directly verifies the prediction result during curation, and then reviews the verification result during curation during review. Also, if the verification flow is set to two stages (curating and reviewing) by multiple users, one user directly verifies the prediction result during curation, and then another user reviews the verification result during curation during review. In two-stage verification, double-checking is performed, so it becomes easier to find labeling errors, and feedback on highly accurate verification results is realized.

[0047] When the learning unit 105 receives the verification result for one dataset, it causes the learning model to be learned (i.e., relearned) (S38). Here, the learning model to be learned is the learning model used in the auto-labeling in S36. After learning, the prediction unit 106 predicts the label for each item included in the dataset acquired in S34 again (S36). Subsequently, the presentation of the prediction result and the reception of the verification result (S37) and the learning of the learning model (S38) are repeated. When the user who verifies the prediction result determines, for example, that there is no error in the prediction result and there is no need for learning, the user instructs to save the data. In response to this, the data management unit 107 saves the first learning model 111-1 to the nth learning model 111-n learned by the learning unit 105 and the annotated data 113 which is the data labeled by the prediction unit 106 in the data storage unit 110 (S39). Also, when creating a labeling rule, the data management unit 107 saves the labeling rule in the data storage unit 110 as the labeling rule 112.

[0048] Note that the learning unit 105 may learn the first learning model 111-1 to the nth learning model 111-n using the plurality of touch points created in S32, without the verification result obtained in S37, for the dataset acquired in S34. Then, after that, the learning unit 105 may learn the first learning model 111-1 to the nth learning model 111-n using the dataset acquired in S34 and the verification result obtained in S37. The learning unit 105 can learn the learning model according to the structure and characteristics that the plurality of touch points themselves have.

[0049] Thereafter, the job management unit 101 ends the job created in S31. Here, the job management unit 101 may save the information related to the job in the data storage unit 110. The information related to the job includes, for example, the information set in S31 and the data created or acquired from S32 to S35.

[0050] Referring to FIG. 4, the features of the annotation according to this embodiment will be described. FIG. 4 is a diagram showing the features of the annotation. Specifically, it shows the flow of annotation by the machine and annotation by humans employed in this embodiment. Here, it is assumed that the label is verified in two steps as annotation by humans. The prediction unit 106 predicts (labels) the label for each item included in the dataset using the learned learning model (S41, S36 in FIG. 3). The labeling in S41 corresponds to auto-labeling by the machine. Subsequently, the user who visually confirms the prediction result verifies the item and the accuracy of the label for the item in the first-stage curation, and corrects the label to the correct one if it is incorrect (S42). Subsequently, in the second-stage review, the verification result in the curation is reviewed (S43). In this way, by combining annotation by the machine and annotation by humans, more accurate labeling is realized.

[0051] Also, in this embodiment, after receiving the verification result in S37 in FIG. 3, the learning model is immediately learned (S38). Thereby, it becomes possible to strengthen the learning model in real time (improve the annotation accuracy). Also, in this embodiment, by storing the learning model that has been learned for another use in the past in the data storage unit 110, the number of applicable learning models increases in S35 in FIG. 3 for the learning model configuration unit 104, and the load on the user to create a labeling rule can be reduced.

[0052] Note that although a specific embodiment has been described above, the embodiment is merely an example and is not intended to limit the scope of the present invention. The apparatus and method described in this specification can be embodied in forms other than those described above. Also, without departing from the scope of the present invention, omissions, substitutions, and changes can be appropriately made to the above-described embodiment. The forms with such omissions, substitutions, and changes are included in the scope of what is described in the claims and their equivalents, and belong to the technical scope of the present invention.

[0053] The disclosure of this embodiment includes the following configurations. [1] A creation unit that creates a plurality of question sets, where each question set is composed of a question and a plurality of labels indicating selectable answers for the question, a creation unit, and a learning unit that uses the plurality of question sets to train a plurality of learning models. An information processing apparatus.

[0054] [2] An acquisition unit that acquires a data set including a plurality of items to be labeled, and a prediction unit that predicts labels for the plurality of question sets by applying the plurality of learning models to the plurality of items. The information processing apparatus according to [1].

[0055] [3] Further includes a presentation unit that presents the labels predicted by the prediction unit to the user, and the learning unit trains the plurality of learning models based on the verification results by the user for the presented labels. The information processing apparatus according to [2].

[0056] [4] Further includes an assignment unit that assigns any one of the plurality of learning models to each of the plurality of question sets, and the prediction unit predicts labels for the plurality of question sets using the assigned learning model. The information processing apparatus according to [2] or [3].

[0057] [5] The plurality of learning models include learning models that have been pre-trained using a data set different from the data set. The information processing apparatus according to any one of [2] to [4].

[0058] [6] The plurality of learning models include one or more labeling rules set based on the data set. The information processing apparatus according to [5].

[0059] [7] The creation unit creates the plurality of question sets with different question types, and the question types include a first type that takes three or more values and a second type that takes two values. The information processing apparatus according to any one of [1] to [6].

[0060] [8] Creating a plurality of question sets, each question set being composed of a question and a plurality of labels indicating selectable answers for the question, and training a plurality of learning models using the plurality of question sets. An information processing method.

Explanation of Signs

[0061] 10: Information processing apparatus, 101: Job management unit, 102: Touch point creation unit, 103: Data acquisition unit, 104: Learning model configuration unit, 105: Learning unit, 106: Prediction unit, 107: Data management unit, 110: Data storage unit, 111-1: First learning model, 111-n: nth learning model, 112: Labeling rule, 113: Annotated data

Claims

1. a creation unit that creates a plurality of question sets, each of the plurality of question sets being composed of a question and a plurality of labels indicating selectable answers to the question; a learning unit that learns a plurality of learning models using the plurality of question sets; An information processing device having the above configuration.

2. an acquisition unit for acquiring a dataset including a plurality of items to be labeled; a prediction unit that predicts labels for the plurality of question sets by applying the plurality of learning models to the plurality of items; The information processing device according to claim 1 , further comprising:

3. A presentation unit that presents the label predicted by the prediction unit to a user, The learning unit trains the plurality of learning models based on a verification result by the user for the presented label. The information processing device according to claim 2 .

4. An assignment unit that assigns one of the plurality of learning models to each of the plurality of question sets, The prediction unit predicts labels for the plurality of question sets using the assigned learning model. The information processing device according to claim 2 .

5. The information processing device according to claim 2 , wherein the plurality of learning models includes a learning model that has been trained in advance using a data set different from the data set.

6. The information processing device according to claim 5 , wherein the plurality of learning models include one or more labeling rules set based on the dataset.

7. the creation unit creates the plurality of question sets having different question types, the question types including a first type taking three or more values ​​and a second type taking two values; The information processing device according to claim 1 .

8. creating a plurality of question sets, each of the plurality of question sets comprising a question and a plurality of labels indicating possible answers to the question; training a plurality of learning models using the plurality of question sets; An information processing method comprising:

Images