Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To solve the problem that the efficiency of label design is low since much labor is required for trying various labeling patterns.SOLUTION: In the information processing apparatus 1 including the label management unit 102 that manages label items to be assigned to data, the label assignment unit 103 that assigns the label items managed by the label management unit 102 to the data, the data management unit 101 that manages the label items and the data to which the labels are assigned, and the label-related information storage unit 121 that stores information on the labels assigned to the data, the label management unit 102 has a function of applying a relational attribute being a relation between a plurality of label items to the label items to manage the relational attribute. The label assignment unit 103 also assigns, to the data assigned with the label item, a label item having a predetermined relationship with the label item.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device that labels data, a control method for the information processing device, and a program. [Background technology]

[0002] The technology of labeling data is known as annotation. Annotation is primarily used to create training data for machine learning, and by training the labeled training data, a learning model that performs the desired processing is obtained. However, setting appropriate label items for performing desired processing and determining labeling rules often require trial and error. Deciding on these label items and labeling rules is called label design.

[0003] Patent Document 1 discloses a technique for determining whether or not correction of label design is necessary based on the results of clustering word vectors in order to assign appropriate labels to words. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-170255 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, it was difficult to try out various labeling patterns and carry out a trial-and-error process for label design, such as when redoing a label design. As such, with conventional technology, trying out various labeling patterns required a great deal of effort, resulting in poor efficiency in label design.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to provide a mechanism that allows various labeling patterns to be easily tried out and improves the efficiency of label design. [Means for solving the problem]

[0007] The present invention comprises label management means for managing label items to be assigned to data, assignment means for assigning the label items managed by the label management means to the data, data management means for managing the label items and the data to which the labels have been assigned, and storage means for holding information regarding the labels assigned to the data, wherein the label management means has a function of managing relational attributes, which are relationships between a plurality of label items, by applying the relational attributes between the label items, and the assignment means controls so that, to data to which the label item has been assigned, label items that have a predetermined relationship with the label item are also assigned. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the efficiency of label design when assigning labels to data. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an example of the functional configuration of the information processing device according to the first embodiment. [Figure 2] 10 is a flowchart showing an example of a processing procedure up to assigning a label to data. [Figure 3] FIG. 10 is a diagram illustrating a UI for a user to assign a label. [Figure 4] FIG. 10 is a diagram illustrating an example of a UI for a user to select a target label group. [Figure 5] 10 is a flowchart showing an example of a processing procedure when a change occurs in the label design. [Figure 6] FIG. 10 is a diagram illustrating an example of a UI when a user checks a list of labels. [Figure 7]10 is a flowchart showing an example of a processing procedure for assigning labels of a learning label set. [Figure 8] FIG. 10 is a diagram illustrating a UI for allowing a user to set a relation attribute between labels. [Figure 9] FIG. 10 is a diagram illustrating a UI for allowing a user to label a disassembly state. [Figure 10] FIG. 10 is a diagram showing an example of the functional configuration of an information processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of a processing procedure for using an evaluation result for an output data set. [Figure 12] FIG. 10 is a diagram illustrating an example of an extraction result of unique information. [Figure 13] FIG. 10 is a diagram illustrating an example of a UI in which analysis results are superimposed on relational attributes between labels. [Figure 14] FIG. 10 is a diagram showing an example of a table for managing confirmation flags. [Figure 15] FIG. 10 is a diagram showing an example of a training set. [Figure 16] 10 is a flowchart showing an example of a processing procedure for selecting an optimum label design under certain conditions. [Figure 17] 1 is a diagram showing an example of a hardware configuration of an information processing apparatus according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0010] [First embodiment] First, before describing the details of each embodiment of the present invention, the hardware configuration of an information processing device of each embodiment will be described with reference to FIG. FIG. 17 is a diagram illustrating an example of a hardware configuration of an information processing apparatus according to an embodiment of the present invention. 17, a CPU 1810 comprehensively controls each device connected via a bus 1800. The CPU 1810 reads and executes processing steps and programs stored in a read-only memory (ROM) 1820 or a solid-state drive (SSD) 1860. An operating system (OS), as well as each processing program, device driver, and the like according to this embodiment are stored in the ROM 1820 or the SSD 1860, temporarily stored in a random access memory (RAM) 1830, and executed by the CPU 1810 as appropriate.

[0011] The input I / F 1840 inputs input information from external devices (input devices not shown, such as a keyboard, pointing device (e.g., a mouse or touch panel), scanner, imaging device, etc.) as an input signal in a format that can be processed by this information processing device. The output I / F 1850 outputs the processing result of this information processing device as an output signal to an external device such as a display device (not shown) in a format that can be processed by the external device. The SSD 1860 is a non-volatile storage device that stores programs and various data. Note that the configuration may include other storage devices such as an HDD (Hard Disk Drive) or eMMC (embedded MultiMediaCard) instead of or in addition to the SSD.

[0012] The information processing apparatus of this embodiment will be described in detail below. In the first embodiment, an example will be described in which the information processing device 1 defines and manages the relationships between labels assigned to data used for learning and evaluation, so that even when label items are added and label relationships are changed from the initially assigned labeling, the user can efficiently replace the labels with new labels (label design) with less effort.

[0013] Fig. 1 is a diagram showing an example of the functional configuration of an information processing device according to the first embodiment. Each function shown in Fig. 1 is realized by a CPU 1810 reading and executing a program stored in a ROM 1820 or an SSD 1860.

[0014] 1, information processing device 1 is an annotation terminal that assigns and manages labels to data. Information processing device 1 includes a request receiving unit 100, a data management unit 101, a label management unit 102, a label assignment unit 103, a list extraction unit 104, a display control unit 105, a labeled data extraction unit 106, an output unit 107, a data storage unit 120, and a label-related information storage unit 121. However, the functional configuration shown in FIG. 1 is merely an example and does not limit the scope of application of the present invention.

[0015] The request receiving unit 100 receives requests from a user regarding label assignment and management, and outputs commands based on the requests to a label management unit 102, a label assignment unit 103, a list extraction unit 104, and a labeled data extraction unit 106, which will be described later. The requests are input via an input I / F 1840 when a user issues an instruction from an input device (not shown).

[0016] The data management unit 101 manages information in a data storage unit 120 and a label-related information storage unit 121, which will be described later, in association with each other. In addition, the data management unit 101 receives requests from the label assignment unit 103, list extraction unit 104, labeled data extraction unit 106, etc., which will be described later, and returns appropriate information.

[0017] The label management unit 102 manages label items to be assigned in annotations. The label management unit 102 also manages relationship attributes such as parent-child relationships, sibling relationships, and join-and-break relationships as relationships between labels. Details of the relationship attributes will be described later. The label management unit 102 also defines a set consisting of a portion of the set of labels it manages as a target label group, and manages the target label group. Multiple target label groups can be registered. The label management unit 102 also defines a portion of the set of target labels as a training set, and manages the training set. This information managed by the label management unit 102 is linked to data via the data management unit 101 and stored in the label-related information storage unit 121.

[0018] The label assignment unit 103 assigns a label to data managed by the data management unit 101, using label items managed by the label management unit 102, based on a request received by the request reception unit 100. Information about the assigned label is held in the label-related information holding unit 121, and is managed in association with the data in the data holding unit 120 via the data management unit 101.

[0019] The list extraction unit 104 extracts a part of the data managed by the data management unit 101 as a list based on the request received by the request reception unit 100 , and outputs the list to the display control unit 105 . The display control unit 105 converts the list of data extracted by the list extraction unit 104 into a format that can be viewed by the user, and outputs the converted list to a display device (not shown).

[0020] The labeled data extraction unit 106 extracts data labeled based on the learning set managed by the label management unit 102 from the list of data extracted by the list extraction unit 104 based on the request received by the request receiving unit 100. The labeled data extraction unit 106 outputs the extracted labeled data to the output unit 107. The output unit 107 converts the labeled data extracted by the labeled data extraction unit 106 into an outputtable format and outputs the converted data. The output labeled data is used for learning and evaluation.

[0021] The data storage unit 120 stores data used for learning and evaluation. The data stored in the data storage unit 120 includes, for example, images, videos, text information, vector data, etc. The data is divided into appropriate units such as files, such as one image, one video, one sentence or document, and each data is assigned an ID so that it can be identified, and is managed by the data management unit 101.

[0022] The label-related information storage unit 121 stores information related to labels assigned to data in the form of a relational database, etc. The data stored in the label-related information storage unit 121 includes, for example, label items managed by the label management unit 102, relational attributes between labels, target label groups, learning sets, labels of data assigned by the label assignment unit 103, and the like. The actual data stored in the data storage unit 120 and the label-related information storage unit 121 is stored in the SSD 1860, for example.

[0023] Next, we will explain the details of relational attributes. Here, we will explain three relational attributes between labels: parent-child relationship, sibling relationship, and join-decomposition relationship. However, relational attributes are not limited to these three.

[0024] First, the parent-child relationship will be described. A parent-child relationship refers to the relationship between a label item and a label item that further classifies that label item. For example, in the relationship between the label item "human" and two label items, "male" and "female," which are further classifications of the label item "human," the "human" label is defined as the parent, and the "male" and "female" labels are defined as children. In this embodiment, such a parent-child relationship is expressed as parent:human, child:male, and child:female. Furthermore, for parent:human, other parent-child relationships such as child:adult and child:minor may be simultaneously defined. Furthermore, to further classify child:male, different relationship attributes can be defined between different labels, such as parent:male, child:lean, child:normal, and child:obese. In other words, the "male" label is a child in relation to the "human" label, but a parent in relation to the "obese" label.

[0025] Next, the sibling relationship will be described. A sibling relationship is a relationship that represents a set of children of the same parent. For example, a sibling relationship is between parent: human and child: male, and child: female. In this embodiment, a sibling relationship is expressed as (siblings: male, female). A parent-child relationship is indicated by expressing parent: human, (siblings: male, female) for the parent. The same label "banana" may be included in different sibling sets for different parents, such as parent: fruit and (siblings: oranges, apples, bananas) or parent: yellow and (siblings: bananas, chicks, sunflowers).

[0026] Next, the bond decomposition relationship will be described. A join / decomposition relationship is a relationship in which a combined label can be decomposed into multiple decomposed labels. For example, a combined label called a "date" has three decomposed labels: a "year," a "month," and a "day." In this embodiment, such a join / decomposition relationship is expressed as "combined state: date, (decomposed state: year, month, day)." For example, it can be expressed as "combined state: human, (decomposed state: head, torso, hands, feet)." A join / decomposition relationship does not always have to have the same decomposition state; it can also have a case such as "combined state: date, (decomposed state: month, day)." Furthermore, if a decomposition state of a join / decomposition relationship can be defined for some of the label items in a parent-child or sibling relationship, such as parent: date, (sibling: deadline, creation date, period) and combined state: date, (decomposed state: year, month, day), the decomposition state can also be defined for the remaining parent-child or sibling relationships.

[0027] Next, the processing procedure in this embodiment will be described. Here, as an example, we will explain how to extract named entities such as company names, personal names, and total amounts from document data of an invoice, by assigning labels corresponding to named entity items to the document data, and then modifying the label design. This will be explained in detail below.

[0028] First, let us assume that the following label items are defined as named entities to be extracted. For example, let us assume that four label items are defined: "Destination company name," "Issuing company name," "Buyer company name," and "Billing company name." Then, these four label items are used to initially label the data. We will then explain what would happen if the label design were changed so that the "company name" label is first classified into two categories, "destination company name" and "issuing company name," rather than into the four categories mentioned above, and the "destination company name" is then classified into three categories, "buyer company name," "billing company name," and "shipping company name."

[0029] In the following explanation, each process (step) will be denoted with an S at the beginning. 2 is a flowchart showing the processing steps up to when the information processing device 1 assigns the first label to data. The processing of this flowchart is realized by the functions shown in FIG. 1, that is, by the CPU 1810 reading and executing a program stored in the ROM 1820 or the SSD 1860. Here, a specific example will be described in which labels such as "destination company name," "issuing company name," "buyer company name," and "billing company name" are assigned to document data of an invoice.

[0030] In S1000, the request receiving unit 100 receives from the user an instruction to define the label items to be labeled, and sends it to the label management unit 102. Upon receiving this instruction, the label management unit 102 defines the label items. In the above example, the label management unit 102 receives an instruction to define four labels, "destination company name," "issuing company name," "buyer company name," and "billing company name," as well as a label called "company name" that is their parent label, and defines these. The label items defined by the label management unit 102 are stored in the label-related information storage unit 121. Then, processing proceeds to S1001.

[0031] In S1001, the request receiving unit 100 receives an instruction from the user regarding the setting of relationship attributes between labels and sends it to the label management unit 102. Upon receiving this instruction, the label management unit 102 sets relationship attributes between defined label items based on the instruction from the user. That is, it sets relationship attributes that are parent: company name, child: destination company name, child: issuer company name, child: buyer company name, child: billing company name, and also (siblings: destination company name, issuer company name, buyer company name, billing company name). The relationship attributes between labels set by the label management unit 102 are held in the label-related information holding unit 121. Then, the process proceeds to S1002.

[0032] In S1002, the request receiving unit 100 receives information about the target labels to be labeled from the user and sends it to the label management unit 102. The label management unit 102, which has received this information, defines a target label group to be labeled. For example, it sets four label items, "destination company name," "issuing company name," "buyer company name," and "billing company name," as "target label group ID1." The target label group set by the label management unit 102 is stored in the label-related information storage unit 121. Then, the process proceeds to S1003.

[0033] In S1003, the request receiving unit 100 receives an instruction from the user regarding the data to be annotated and sends it to the list extraction unit 104. Upon receiving this, the list extraction unit 104 extracts a list of data to be annotated via the data management unit 101. For example, all file paths to invoice document data stored in the data storage unit 120 are extracted as a list. Then, the process proceeds to S1004. Note that, although an example in which everything is extracted as a list has been described here, this is not limiting. The list may be extracted so as to limit the data to be labeled by extracting a portion of the data to be labeled.

[0034] In S1004, the list extraction unit 104 acquires one unprocessed data item from the list extracted in S1003 and sends it to the display control unit 105. The display control unit 105 converts the data received from the list extraction unit 104 into a format that is easy for the user to label. That is, the data is displayed as a document image of invoice data. Furthermore, the label management unit 102 sends the label items of the "target label group ID1" to the display control unit 105 so that the label items can be presented to the user. The display control unit 105 receives the information of the "target label group ID1" and generates an image (list of label items) to display based on the information. Furthermore, the display control unit 105 outputs the information to a display device (not shown) to present the invoice data and information related to labeling (list of label items) to the user and enable the user to request labeling. For example, the display is presented to the user as shown in FIG. 3. The display device is, for example, a display or a tablet terminal. Here, the UI output by the display control unit 105 in S1004 will be described with reference to FIG. 3.

[0035] FIG. 3 is a diagram showing an example of a UI that allows a user to assign a label. In the UI of Fig. 3, the user selects a character string 11 to which a label is to be assigned from the invoice image data 10 while referring to the invoice image data 10 and a list of label items to be assigned 12. Then, the user selects and confirms a label that they deem appropriate from the list of label items to be assigned 12 using the mouse pointer 13 or the like. In the example of Fig. 3, this allows the user to request that the "Destination Company Name" label be assigned to "ABC Co., Ltd."

[0036] Note that although the example described here is one target label group, this is not the only option. Multiple target label groups can be defined, and when assigning labels, there are two ways to do it: first, determine the target label group to assign labels to and then process the data list in order, or, second, check the data list in order, select a target label group, and assign labels.

[0037] First, in the case of processing the data list in order after determining a target label group as the label to be assigned, it is sufficient to specify one target label group and then process as described in S1004. The case of selecting a target label group while checking the data list in order and assigning labels will be explained using the UI example in Fig. 4.

[0038] FIG. 4 is a diagram illustrating an example of a UI for the user to select a target label group. The user references the image data 10 to find data to which labels should be assigned and selects the character string 11 to which the label should be assigned. Then, the user uses the mouse pointer 13 to select a target label group ID containing the label item to be assigned from the target label group list 14. As a result, a list of label items 12 contained in the selected target label group ID is displayed, as shown in FIG. 3, and the user can further select a label item. By displaying only the target label group list in this way, even if there are a large number of label items, it is possible to quickly find the label to be assigned. Note that the target label group list 14 in FIG. 4 displays the ID names (target label group ID1, ..., target label group ID4) as the target label group list. However, the display name may be set to a name that succinctly represents the target label group, such as "company name" or "person's name." Furthermore, if there are a large number of target label groups, a portion of the target label groups may be defined as a learning set that the user intends to learn, and only the target label groups included in a certain learning set may be displayed as a list. Furthermore, the target label groups may be sorted in descending order of likelihood as the items the user intends to label.

[0039] As described above, when a label assignment request is received from the user, the process proceeds to S1005. In S1005, the request receiving unit 100 receives the request from the user and sends information regarding label assignment to the label assignment unit 103. Upon receiving this, the label assignment unit 103 receives, as information regarding label assignment, the ID of the current invoice document data as the ID of the target data, bounding box information indicating the position of the character string in the invoice document data as information identifying the label assignment target, and a specific label ID managed by the label management unit 102 as label information to be assigned. Based on this, the label assignment unit 103 then registers "ABC Co., Ltd." on this invoice as a "destination company name" label in the label-related information holding unit 121 via the data management unit 101. Furthermore, the label assignment unit 103 confirms via the label management unit 102 that a "company name" label exists as a parent label of the "destination company name" label, and registers "ABC Co., Ltd." as also a "company name" label in the label-related information holding unit 121. In this way, when a child label is assigned, the corresponding parent label is also automatically assigned, thereby enabling efficient label assignment. Then, the process proceeds to S1006.

[0040] In S1006, the list extraction unit 104 determines whether or not unprocessed data (data that has not been presented to the user via the display control unit 105) remains in the list extracted in S1003 above. The list extraction unit 104 presents data to the user one by one via the display control unit 105, and sets a confirmed flag for each target label group ID for data that has been confirmed. The confirmed flags for each target label group ID managed by the list extraction unit 104 are held in the data holding unit 120 via the data management unit 101 as a flag for each target label group ID for each data, as shown in FIG. 14, for example. FIG. 14 is a diagram illustrating an example of a table for managing the confirmation flag. Each row has a name indicating the data (Invoice0001, Invoice0002, Estimate0001, ...) and indicates which target label group ID has been confirmed. In the example in Figure 14, confirmed items are indicated by a circle.

[0041] Here, if there is unpresented data (list remaining) for which the confirmation flag of the target label group ID of interest is not yet set (Yes in S1006), the list extraction unit 104 returns the process to S1004.

[0042] On the other hand, if the confirmation flag for the target label group ID currently being annotated has been set for all data and there is no unpresented data (remaining in the list) (No in S1006), the list extraction unit 104 ends the processing of this flowchart. However, the method of ending is not limited to this. For example, even if there is unconfirmed data remaining in the extracted list, the processing may be ended if the user sends a request to end label assignment at any time and the request receiving unit 100 receives the request.

[0043] By defining the relationship attributes between labels in advance through the above process, parent labels can be automatically assigned simply by assigning child labels, allowing for efficient annotation. Also, by defining label items in advance in units of target label groups or learning sets (described later), the number of label items displayed to the user can be limited, making it easier for the user to select labels.

[0044] Next, the process of changing the label design will be described. Fig. 5 is a flowchart showing an example of the processing procedure when a change occurs in the label design. The processing of this flowchart is realized by the functions shown in Fig. 1, that is, by the CPU 1810 reading and executing a program stored in the ROM 1820 or SSD 1860. Here, we will explain an example of changing the label design from one that classifies the "company name" label into the four categories mentioned above to one that first classifies the "company name" label into two categories, "recipient company name" and "issuing company name," and then classifies the "recipient company name" into three categories, "buyer company name," "billing company name," and "shipping company name."

[0045] In S1100, the request receiving unit 100 receives an instruction from the user to change a label item and sends it to the label management unit 102. Upon receiving this instruction, the label management unit 102 adds, deletes, or changes the label item in accordance with the instruction. In the above example, a label called "Name of destination company" is defined as a new label item. The label management unit 102 registers this content in the label-related information holding unit 121. Then, the process proceeds to S1101.

[0046] In S1101, the request receiving unit 100 receives an instruction from the user to change the relationship attributes between labels and sends it to the label management unit 102. Upon receiving this, the label management unit 102 changes the relationship attributes based on the change instruction from the user. That is, it changes the relationship attributes between labels to parent: company name, (sibling: issuer company name, destination company name), parent: destination company name, (sibling: buyer company name, billing company name, shipping company name). The relationship attributes between labels changed by the label management unit 102 are held in the label-related information holding unit 121. Then, the process proceeds to S1102.

[0047] In S1102, the request receiving unit 100 receives information about the target label group from the user and sends it to the label management unit 102. The label management unit 102, which has received this, defines the target label group to be labeled. Here, three label items, "buyer company name," "billing company name," and "shipping company name," are set as "target label group ID2." At this time, "destination company name," which is the parent label of "buyer company name," "billing company name," and "shipping company name," is also set as "target label group ID2." The target label group set by the label management unit 102 is held in the label-related information holding unit 121. Then, the process proceeds to S1103.

[0048] In S1103, the request receiving unit 100 receives a command from the user regarding information on label items to be confirmed and sends it to the label management unit 102 and the list extraction unit 104. Upon receiving this command, the label management unit 102 compares the label items of the current target label group, "target label group ID2," with the target label groups that have been labeled so far. If identical label items exist, and if their relational attributes have changed, they are listed as label items to be confirmed. At the same time, new label items to be labeled are identified and listed. The list extraction unit 104 then sends this information to the list extraction unit 104. Using the information obtained from the label management unit 102, the list extraction unit 104 extracts a list of data for which annotation confirmation is to be performed via the data management unit 101. The data to be confirmed are the label items "destination company name," "buyer company name," and "billing company name" that are included in both "target label group ID2" and "target label group ID1," as well as data that has already been labeled. The list extraction unit 104 extracts those file paths as a list from the data storage unit 120 via the data management unit 101. Furthermore, it also extracts the labeling results for those files from the label-associated information storage unit 121. The extracted list is output to the display control unit 105. Then, the process proceeds to S1104.

[0049] In S1104, the display control unit 105 displays the labeled data in a list based on the data and its labels received from the list extraction unit 104. This will be described using the diagram of FIG. 6 showing an example of a UI when the user checks the labels in a list. FIG. 6 is a diagram illustrating an example of a UI when the user checks a list of labels. In FIG. 6, 15a to 15e represent extracted document data. These are arranged in a grid on a display device such as a monitor. 16a and 16b represent bounding boxes indicating the positions of label items included in the "target label group ID2" currently being checked that have already been labeled. 17a to 17e represent information detailing the assigned labels, here displaying the named entity strings of the assigned label items, prompting the user to check the data. By displaying the data in a list, the user can quickly check whether the labels have been assigned correctly by comparing them with other examples. Furthermore, if a case requiring correction is found in the list, the user can select the data of that case to transition to the UI for labeling individual data, as shown in FIG. 3, and correct the label. When the request receiving unit 100 receives from the user a notification that some or all of the confirmation process has been completed, it sets a confirmation flag for the "target label group ID" of the confirmed data. Once all confirmation processes are completed, the process proceeds to S1105.

[0050] In S1105, the list extraction unit 104 extracts a list of data to be annotated via the data management unit 101 based on the new label items to be labeled received from the label management unit 102 in S1103. That is, a list of file paths to the document data of all invoices to be labeled with "recipient company name" is extracted. Then, the process proceeds to S1106.

[0051] In S1106, the list extraction unit 104 retrieves one unprocessed data item from the list extracted in S1105 and sends it to the display control unit 105. The display control unit 105 converts the data received from the list extraction unit 104 into a format that is easy for the user to label and displays it for labeling via a display device (not shown) similar to S1004 in FIG. 2. However, since only the "recipient company name" is to be labeled in this case, only the "recipient company name" is displayed in the list of target labels as shown in 12 in FIG. 3. Alternatively, label items other than the "recipient company name" may be displayed grayed out to make the "recipient company name" easier to select. Alternatively, the "recipient company name" may always be selected by default. Furthermore, if there is only one label item displayed in the target label list 12, the operation of selecting an appropriate label with the mouse pointer 13 can be omitted. In other words, the user can simply select the character string to be labeled from the invoice image data 10 and request that the character string be labeled as the recipient company name. Then, the process proceeds to .

[0052] The processing in S1107 and S1108 is similar to that in S1005 and S1006 in FIG. 2, and therefore a description thereof will be omitted. With the above process, users can efficiently check annotations for existing label items by viewing a list. Furthermore, for new label items, by limiting the display to a small number of label items, users can concentrate on finding and labeling the parts that correspond to the target label.

[0053] Next, we will explain the process flow for preparing a dataset by defining a learning set to perform learning, determining the label items to be used for learning from among multiple label items assigned to the dataset. 7 is a flowchart showing an example of a processing procedure for assigning labels from a training label set. The processing of this flowchart is realized by the functions shown in FIG. 1, that is, by the CPU 1810 reading and executing a program stored in the ROM 1820 or the SSD 1860.

[0054] In S1200, the request receiving unit 100 acquires information from the user regarding the data to be used for learning and the definition of the learning set, and sends this information to the label management unit 102, the list extraction unit 104, and the labeled data extraction unit 106. This information from the user is expressed as part of a configuration file for learning. By recording information about the data and information about the labels to be assigned as a learning set in the configuration file, there is an advantage in that it is easy to carry out trial and error learning, including label design. The learning set is expressed as a collection of target label groups. For example, as shown in "Learning Set 01" in Figure 15, four target label groups, ID1, ID2, ID3, and ID5, are used as the learning set. FIG. 15 is a diagram illustrating an example of a learning set. The label management unit 102 receives information about the learning set and sends information about the target label group set in the learning set to the list extraction unit 104. Furthermore, the label management unit 102 breaks down the target label group set in the learning set into label item units, acquires information about the label items used for learning, and sends it to the label-associated information holding unit 121 and the labeled data extraction unit 106. Then, the process proceeds to S1201.

[0055] In S1201, the list extraction unit 104 acquires a data list from the data storage unit 120 via the data management unit 101 based on the received information about the learning data. Then, the process proceeds to S1202.

[0056] In S1202, the list extraction unit 104 checks whether or not the target label group IDs in the list acquired in S1201 have been confirmed, based on the information on the target label group set received from the label management unit 102. If data that has not been confirmed is found here (No in S1202), the list extraction unit 104 lists them and proceeds to S1203. On the other hand, if no unconfirmed data is found (No in S1202), the process proceeds to S1204.

[0057] In S1203, the list extraction unit 104 sequentially presents unconfirmed data to the user and performs label assignment processing. Specifically, by repeatedly performing the processing in S1004, S1005, and S1006 in FIG. 2, all data is flagged as confirmed for all target label groups corresponding to the learning set. When sequentially displaying unconfirmed data, the list extraction unit 104 generates a list so that data with the same set of target label groups without a confirmed flag are consecutive. For example, first, a list of data with a confirmed flag only for "target label group ID 1" is displayed. Once all of the data have been confirmed, a list of data with a confirmed flag only for "target label group ID 3" is displayed. After processing data with a confirmed flag only for a single target label group ID, data with a confirmed flag for multiple target label group IDs is displayed. For example, a list of data with a confirmed flag only for target label group ID 1 and ID 3 is displayed. By displaying the list in this order, the user can continuously assign labels to the same target label group, making it easier for the user to concentrate on the task. In this way, once all data has been confirmed for all target label groups corresponding to the learning set, the process proceeds to S1204.

[0058] In S1204, the labeled data extraction unit 106 acquires labeled data via the data management unit 101 based on the list of data acquired from the list extraction unit 104 and the label items acquired from the label management unit 102. The acquired data is all the data in the list, and only the label items corresponding to the learning set are assigned to the data. The labeled dataset is sent to the output unit 107. Then, the process proceeds to S1205.

[0059] In S1205, the output unit 107 converts the labeled dataset received from the labeled data extraction unit 106 into an outputtable format and outputs it. The output format is a format that is easy for a learning model to read, such as CSV format or TFRecord format. This completes the processing of this flowchart.

[0060] By defining the training set in this way, users can easily generate a labeled dataset for training, even if they have assigned multiple different label items to a single piece of data, making it easy to change the label design and conduct trial and error.In addition, by checking the label assignment for each target label group, it is possible to reduce omissions in label assignment.

[0061] With the above configuration, labeling can be performed efficiently, and even when the label design is changed, processes such as checking already-applied labels and applying additional labels can be performed efficiently and without omissions.

[0062] (Modification of the first embodiment) In Figure 5, the process flow when a change occurs in the label design is explained, in which the label confirmation list is extracted and confirmed, but confirmation is not necessarily required. For example, if the user determines that confirmation is not required for the "Buyer Company Name" or "Billing Company Name," the processes of S1103 and S1104 may be skipped. Whether confirmation is not required can also be set when defining the target label group in S1102 during label design by presenting the user with an option to use labels with a specific target label group ID as is without confirmation.

[0063] In the example of Figure 5 illustrating the processing procedure for changing a label design, a new label item, "Destination Company Name," is defined, and in S1105, all invoice data is extracted and labeled. However, the extracted list does not necessarily include all data. For example, consider a case in which new label items are separated into "Destination Company Name with Title" and "Destination Company Name without Title" based on whether or not they include honorifics such as "Mr." or "Dear." The labels are then added to the "Destination Company Name with Title" and "Destination Company Name without Title" patterns. In such a case, rather than simply processing the relationship attribute as "Parent: Destination Company Name" (Sibling: Destination Company Name with Title, Destination Company Name without Title), an additional relationship attribute can be added indicating that these new labels are contained within the existing "Destination Company Name." When handling contained label items, the data to be extracted by the list extraction unit 104 in S1103 can be limited to data labeled with the "Destination Company Name" label for data with a confirmed flag set for target label group ID1. However, this does not apply to data that does not have a confirmed flag for target label group ID1, in other words, data that has not yet been assigned the "destination company name" label. Also, for data that has been assigned the "destination company name" label, labeling can be performed simply by sorting the existing "destination company name" label into "with title" or "without title," which makes it possible to complete the labeling work very efficiently compared to extracting and labeling all data.

[0064] Furthermore, attributes such as "with title" and "without title" mentioned above are not only attributes that appear in "destination company name," but can also be classified as child labels of "destination company name," such as "buyer company name," "billing company name," and "shipping company name." In such cases, "with title" and "without title" labels can be assigned as attribute labels to "destination company name." When attribute labels are assigned, they can be considered to have been assigned not only to the assigned label but also to its child labels. By registering these as attribute labels, if you want to change the label design and learn using "buyer company name with title" and "buyer company name without title," you can immediately extract labeled data without having to assign or confirm new labels, because the attribute labels "with title" and "without title" have already been assigned to the destination company name, which is the parent label of "buyer company name."

[0065] Furthermore, the concept of "being included" described above may change depending on how the labeling rules are handled. For example, in the labeling process in the flowcharts of FIGS. 2 and 5 in the first embodiment, the description was given on the implicit assumption that "recipient company name" is not included in "destination company name." However, if "recipient company name" is labeled to be included in "destination company name," i.e., if labeling of "destination company name" is performed using a rule that classifies company names as belonging to either the destination or the issuer, then "recipient company name" can be said to be included in "destination company name." In this case, when newly labeling "recipient company name," the list extraction unit 104 can limit the data to data to which the "destination company name" label is attached. Furthermore, since "Buyer Company Name" and "Billing Company Name" are siblings of "Shipping Destination Company Name," and sibling labels are never assigned simultaneously, the data to be extracted by the list extraction unit 104 can be limited to data that has the "Destination Company Name" label assigned but does not have the "Buyer Company Name" or "Billing Destination Company Name" assigned. This reduces the amount of data to be extracted, and enables new labels to be assigned in a short time.

[0066] In the first embodiment, the label change was described as adding a new child label, "Destination Company Name," to "Destination Company Name." However, if a new parent label is added due to a label design change after the child labels have been assigned, the parent label assignment can be performed more easily. For example, if the "Buyer Company Name," "Billing Company Name," and "Destination Company Name" labels are assigned during the initial label assignment, and then a label design change results in a new parent label, "Destination Company Name," the destination company name label can be automatically assigned to data to which these three child labels are assigned. If the definition of "Destination Company Name" is such that it is included in these three child labels, the assignment of the "Destination Company Name" label is complete. In this way, by using the concept of parent-child relationship attributes, label assignment can sometimes be completed almost automatically.

[0067] In the previous example, we defined the "Destination Company Name" label as the parent label of the "Buyer Company Name," "Billing Company Name," and "Shipping Company Name." Here, we explain how to define "Destination Company Name" as a conditional label. For example, let's say that a single data item can contain only one "Destination Company Name"—either "Buyer Company Name," "Billing Company Name," or "Shipping Company Name"—with the order of priority being "Buyer Company Name" > "Billing Company Name" > "Shipping Company Name." In other words, if a data item contains only a "Shipping Company Name," that "Shipping Company Name" is the "Destination Company Name." Also, if a data item contains both a "Buyer Company Name" and a "Shipping Company Name," the "Buyer Company Name," which has the highest priority, is the "Shipping Company Name." Even when assigning conditional labels like this, you can automatically assign destination company name labels by simply applying the defined label item condition processing to each item. The ease of designing such conditional labels makes it easy to test whether the conditions should be implemented in post-processing or whether they should be incorporated into the learning model and trained.

[0068] In addition, in S1001 of FIG. 2, an example has been described in which the label management unit 102 sets the relational attributes between labels, but the relational attributes may also be set graphically using a UI. FIG. 8 is a diagram showing an example of a UI that allows a user to set a relation attribute between labels. FIG. 8(a) shows the relationship attributes between labels and the target label group ID1 when the label design is first performed in the first embodiment. Furthermore, FIG. 8(b) shows the relationship attributes and the target label group ID2 after the label design has been changed and in a modified example in the first embodiment. The user can freely arrange label items and define and check the relationship attributes on the UI displayed as in FIG. 8(a) or 8(b). In this way, the display control unit 105 obtains information from the label management unit 102 and graphically displays the relationship attributes between labels, allowing the user to design labels with intuitive operations.

[0069] Furthermore, in the labeling in the first embodiment, an example of managing label items and their related attributes has been described, but the items to be managed are not limited to these. For example, the ID of the person who assigned the label may also be managed. By managing the person who assigned the label, when a labeling error is discovered, it is likely that the same person will assign the same erroneous label, making it possible to efficiently check for errors.

[0070] Furthermore, the confidence level at the time of labeling may be managed. For example, when labeling, the user may be asked to input one of several levels (for example, five levels) of confidence level, and the confidence level of the labeling may be managed in accordance with this. For items that the user is not confident in labeling (items below a predetermined level), the user can have someone else label them. Also, if the definition of a label item is vague, this may lead to activities to review the labeling definition itself.

[0071] Furthermore, the date and time of labeling may also be managed. By recording the date and time of labeling, desired data can be identified more efficiently by searching or filtering by date and time, even if the labeling rule for the same label item is changed.

[0072] In addition, the person (user) who assigned the label may also be managed. By recording the person who assigned the label, for example, since the same person is likely to make the same labeling error, it becomes easier to find the target mislabeled data when correcting the labeling error.

[0073] (Example of bond decomposition relationship) In the first embodiment, a case where a parent-child relationship and a sibling relationship are used as relationship attributes is described. Here, a case where a join-break relationship is used is described. For example, the initial label design, as in the first embodiment, assigns labels of "destination company name," "issuing company name," "buyer company name," and "billing company name" to invoice document data. Subsequent label design changes decompose the "destination company name" into "corporate name" and "legal entity." In other words, the combined state is expressed as "destination company name," (decomposition state: destination corporate name, destination corporate entity). Specifically, this example decomposes the destination company name "AIU LLC" into the destination corporate name "AIU" and the destination corporate entity "Limited Company." Furthermore, "corporate name" and "legal entity" can be defined as attribute labels for the parent label "company name." Therefore, this example describes an example in which "corporate name" and "legal entity" are defined as attribute labels, and data labeled with "destination corporate name" and "destination corporate entity" is extracted as a learning set.

[0074] Here, the process is the same as that shown in the flowchart of FIG. 5 described above in the modified example of the first embodiment, so only the differences between the steps will be explained. In S1100, the label management unit 102 defines attribute labels "corporate name" and "legal entity" of the "company name" label as new label items.

[0075] In S1101, the label management unit 102 defines the combined state: company name (decomposed state: corporate name, legal entity). When setting on the UI, it can be set as shown in Figure 8(b). Furthermore, in the decomposed state, the "corporate name" is an item that must be present, while the "corporate entity" may or may not be present, and may be placed either before or after the "corporate name". Furthermore, the "corporate entity" has the characteristic of being composed only of specific keywords such as "joint stock company" or "corporate association". In this way, by managing detailed label settings as well, it is possible to realize automatic label assignment and validation functions, as will be shown later.

[0076] In S1102, the label management unit 102 sets "company name corporate name" and "company name legal entity" as a new target label group ID 7. In this example, since there is no list to be checked, S1103 and S1104 are skipped. In S1105, the list extraction unit 104 extracts the data to which the "company name" label has been added as a list.

[0077] In S1106, the display control unit 105 displays, via the display device, the diagram showing an example of a UI in which the user assigns labels to the disassembled states, as shown in FIG. FIG. 9 is a diagram showing an example of a UI that allows the user to label the disassembly state. When dealing with only decomposed label items, such as target label group ID7, if the corresponding named entity in the combined state has already been registered, it is displayed in the combined state display field 18. If the combined state has not been registered, the label item is labeled and displayed in the combined state display field 18. The user can then select a portion of the string within the combined state display field 18, as indicated by the dotted bounding box 19, and send a label assignment request by selecting that label item with the mouse pointer 13. Reference numeral 20 denotes a legal entity list. This list is displayed when a legal entity label is set to be characterized by being composed only of specific keywords. Character strings that the user has previously assigned as legal entity labels are stored in the legal entity list 20 without duplication. If a character string stored in the legal entity list 20 exists in the combined state display field 18, a request to automatically assign a legal entity label is sent. As shown in 19, a character string may be selected, and then a legal entity label may be selected from the target label group list 12 to send a label assignment request. Furthermore, if the corporate name is defined as the part of the company name excluding the legal entity, the corporate name can be determined automatically once the company name and legal entity are determined. By using the legal entity list 20, it is also possible to send a request to automatically label both the legal entity and the corporate name. Since S1107 and S1108 are the same as the processes in FIG. 5, the description thereof will be omitted.

[0078] Finally, when preparing a dataset for training, you can define a new target label group ID8 consisting of "Destination Corporate Name" and "Destination Corporate Status" by applying the attribute labels "Corporate Name" and "Corporate Status" to "Destination Company Name" in advance, define a training set that includes this, and output two labeled sets using the processing flow shown in Figure 7.

[0079] In this modified example, the list extraction unit 104 assigns labels to each piece of data in the list extracted by the list extraction unit 104, but this is not limiting. For example, a list display such as that shown in FIG. 6 may be used to check whether the "corporate entity" and "corporate name" have been correctly and automatically assigned. Automatic assignment is not possible in the case of data with a company name that has a corporate entity and is not registered in the corporate entity list 20. In such cases, labels are assigned to the "corporate entity" and "corporate name" as shown in FIG. 9, and the corporate entity is registered. Once a new corporate entity is registered in the corporate entity list 20, the additional data to which the "corporate entity" and "corporate name" have been automatically assigned can be checked in a list. In this way, by defining the characteristics of the label items related to the join / disjoint relationship in detail, the efficiency of labeling large amounts of data can be significantly improved.

[0080] Also, an example of validation that uses the characteristic that "company name" always contains "corporate name" is explained. For example, let's consider the case where "Public Interest Incorporated Foundation" is registered with the "Company Name" label. If other data indicates that "Public Interest Incorporated Foundation" is registered with the "Corporate Status" label, then the company name "Public Interest Incorporated Foundation" will consist only of "Corporate Status" and will not include the "Corporate Name." In other words, this contradicts the first characteristic that "Company Name" always includes the "Corporate Name." Therefore, it can be determined that the company name "Public Interest Incorporated Foundation" is not a correct "Company Name." In this way, validation can be performed using the characteristics of the label, and labeling errors can be automatically detected.

[0081] In the example of a join decomposition relationship, join state: company name, (decomposition state: legal entity name, legal entity), but in a join decomposition relationship, combining decomposition states does not necessarily result in a join state. For example, in the example of join state: two-dimensional coordinate, (decomposition state: X coordinate, Y coordinate), for a named entity with a two-dimensional coordinate label of "(3,8)", the value of the X coordinate label is "3" and the value of the Y coordinate label is "8". In this way, there are cases where simply listing the values ​​of the decomposition state labels together does not result in the value of the join state. In such cases, it is possible to automatically assign or validate labels by defining a conversion method, such as expressing the two-dimensional coordinate label value in the format "(X coordinate label value, Y coordinate label value)".

[0082] Second Embodiment The information processing device 2 of the second embodiment defines a training set, generates training data and evaluation data, performs model training and evaluation, and feeds back the evaluation results for use in label design. By feeding back the evaluation results for the training set and mapping them during label design, label design can be performed more intuitively.

[0083] Fig. 10 is a diagram showing an example of the functional configuration of the information processing device 2 in the second embodiment. The configuration of the information processing device 2 in the second embodiment has some parts in common with the example configuration in the first embodiment shown in Fig. 1. In Fig. 10, parts in common with Fig. 1 are given the same reference numerals. Here, only the evaluation result acquisition unit 208, analysis unit 209, and evaluation result storage unit 222, which are different, will be described.

[0084] The evaluation result acquisition unit 208 acquires the results of the learning model evaluating the evaluation data as feedback, and outputs the acquired evaluation results to the analysis unit 209 and the evaluation result storage unit 222. The analysis unit 209 analyzes the evaluation results received from the evaluation result acquisition unit 208, identifies label items that are not going well in learning or evaluation, and target label group IDs, outputs suggestions for improving them (improvement suggestions), and presents them to the user via the display control unit 105. The evaluation result holding unit 222 receives the evaluation results from the evaluation result acquisition unit 208, associates them with the data in the data holding unit 120 via the data management unit 101, and holds them as evaluation results for each data.

[0085] Next, we will explain the process of utilizing the results of learning and evaluation on a dataset for which labels have been designed and assigned. 11 is a flowchart showing an example of a processing procedure for using the evaluation results for the output data set. The processing of this flowchart is realized by the functions shown in FIG. 1, that is, by the CPU 1810 reading and executing a program stored in the ROM 1820 or the SSD 1860.

[0086] In S2000, the output unit 107 outputs a dataset for the learning set defined by the process shown in FIG. 7. In the subsequent process, two datasets are output for learning and evaluation: a learning dataset and an evaluation dataset for the evaluation. Like the learning dataset, the evaluation dataset can also be output by the process shown in FIG. 7. These datasets are linked to the ID of the learning set. Then, the process proceeds to S2001.

[0087] In S2001, a learning evaluation unit (not shown) performs learning using the learning dataset output from the output unit 107. The learning evaluation unit may be included in an information processing device different from the information processing device 2. For example, learning may be performed as a multi-class classification problem using a BERT (Bidirectional Encoder Representations from Transformers) model. However, the model is not limited to BERT. After learning is completed, the learning evaluation unit evaluates the trained learner using the evaluation dataset. For example, in the case of the named entity extraction task shown in the example of the first embodiment, evaluation is performed assuming that the answer is correct if the position of labeled character strings and label items can be correctly inferred from document data. The results of evaluation performed on the entire evaluation dataset are output to the evaluation result acquisition unit 208. Then, processing proceeds to S2002.

[0088] In S2002, the evaluation result acquisition unit 208 acquires the evaluation results of the evaluation dataset. Then, the evaluation result acquisition unit 208 outputs the acquired evaluation results to the analysis unit 209 and the evaluation result storage unit 222. Then, the process proceeds to S2003.

[0089] In S2003, the analysis unit 209 acquires the evaluation results from the evaluation result acquisition unit 208 and analyzes the evaluation results to obtain an evaluation value. For example, the accuracy for each label item may be calculated as the recall rate, precision rate, F1 score, or support number in the evaluation data. The support number in the evaluation data refers to the total number of named entities that appear in all documents in the evaluation data for each label item. The analysis unit 209 may also acquire information about the training set used to train the trained model used for the evaluation from the label management unit 102 to obtain the support number in the training data. Based on these evaluation values, the analysis unit 209 proposes measures that the user can take to improve label items with low accuracy. The proposed measures include at least one of improvement proposals for label design based on the values ​​and attributes of each label item and data-based countermeasures.

[0090] An example of the proposal will be described below with reference to FIG. Fig. 12 is a diagram showing an example of the unique information extraction results. The example in Fig. 12 is a table summarizing the F1 value, the number of supports in the evaluation data, and the number of supports in the training data for each of three label items: "Buyer company name," "Billing company name," and "Shipping company name."

[0091] First, the "buyer company name" has a high F1 value, and the number of supporting evaluation data and training data appears to be sufficient. Next, "billing company name" has a low F1 value and fewer training data supports than the others. Based on this, the analysis unit 209 generates a suggestion statement saying, "The F1 value may be low because the number of training data supports for the 'billing company name' label is small. To improve the F1 value, please add data labeled 'billing company name' to the training data."

[0092] Furthermore, although the F1 value for "recipient company name" is high, the number of evaluation data supports is small, and based on this, the analysis unit 209 generates a suggestion statement saying, "The number of evaluation data supports for the 'recipient company name' label is small, so the F1 value may not be reliable. To calculate the F1 value more accurately, please add data labeled with the 'recipient company name' to the evaluation data."

[0093] Furthermore, based on the fact that the "destination company name" label is defined as the parent label of these three, the analysis unit 209 generates a suggestion statement saying, "The 'billing company name' and 'shipping company name' labels may have low F1 values ​​or may be unreliable due to a small number of supports. Please consider using the 'destination company name' label, which is a parent label that combines the 'buyer company name', 'billing company name' and 'shipping company name' labels, for learning and evaluation."

[0094] The analysis unit 209 generates a suggested sentence based on the evaluation value, such as the F1 score, the evaluation data support count, the training data support count, and the relationship attributes between the labels of the label items. The generation method may be determined in advance by a rule, or the analysis unit 209 may learn to generate suggested sentences using machine learning or other methods based on various examples. The analysis unit 209 may also change the information to be displayed or generate suggested sentences based on the evaluation results and information associated therewith accumulated by the evaluation result storage unit 222 in S2005 (described later). For example, the analysis unit 209 may generate suggested sentences by searching for similarities with past examples with low evaluation values ​​or examples that have been improved from those results. The analysis unit 209 outputs the generated evaluation values ​​and suggested sentences to the display control unit 105. Then, the process proceeds to S2004.

[0095] In S2004, the display control unit 105 superimposes and displays the analysis results on the relational attributes of the labels based on the evaluation values ​​and suggested sentences acquired from the analysis unit 209 and the information on the relational attributes of the labels acquired from the label management unit 102. For example, the display is as shown in FIG. Fig. 13 shows an example of a UI that displays analysis results superimposed on the relationship attributes between labels. This figure is based on the UI shown in Fig. 8(b) that allows the user to set the relationship attributes between labels.

[0096] 13, by superimposing evaluation values ​​and suggestions based on the analysis results on a UI for label design, the user can consider label designs on a screen that consolidates all the information necessary for label design. Furthermore, in S2005 (described later), the analysis unit 209 extracts past evaluation results and candidate label design information that can serve as references for label design from the evaluation results accumulated in the evaluation result storage unit 222, and the display control unit 105 presents these to the user as a list or thumbnails, displaying the candidate selected by the user. This allows the user to consider label designs while checking past label designs and their evaluation results. Then, the process proceeds to S2005.

[0097] In S2005, the evaluation result holding unit 222 holds and accumulates the evaluation results received from the evaluation result acquisition unit 208. When storing the results, not only the evaluation results but also the learning data and evaluation data corresponding to the evaluation results, as well as the labels and label design information assigned to these data, are stored in association with each other. The accumulated data can be acquired when requested by the user or the system in the information processing device 2. This completes the processing of this flowchart.

[0098] By learning and evaluating based on the labels assigned in this way, analyzing the results, and displaying the information necessary for label design on a consolidated screen, users can efficiently consider label design, such as whether label items are necessary and the relationship attributes between labels, as well as adding or deleting data.

[0099] Next, we will explain the process flow for selecting the optimal label design for learning and evaluation under the conditions specified by the user in order to perform label design and obtain the desired evaluation results. The conditions are, for example, "we want to classify a certain label A and a certain label B," "we can use child labels and attribute labels of the label to be inferred," "the minimum support number for each label during learning is N," and "we want the F1 value of each classification result to be T or more as the evaluation result." When attempting to design labels under such conditions, a large number of combinations of label items will arise, so by using a system that automatically learns and evaluates these, it is possible to select the optimal combination of label items and use this information in label design.

[0100] 16 is a flowchart showing an example of a processing procedure for selecting an optimum label design under certain conditions. The processing of this flowchart is realized by the functions shown in FIG. 1, that is, by the CPU 1810 reading and executing a program stored in the ROM 1820 or SSD 1860.

[0101] The following processing flow will be explained using an example of label design related to the "company name" label shown in Figure 8(b). Specifically, the "company name" label has attributes of "with title" and "without title," and it is possible to select a label that takes titles into consideration or a label that does not. The "company name" label also has a combination / decomposition attribute, and it is possible to select a combination state: company name (decomposition state: corporate name, legal entity). The parent-child relationship is parent: company name, child: issuer company name, child: destination company name, and so on, and the parent: destination company name, child: buyer company name, child: billing company name, child: shipping company name. Furthermore, the attributes of the company name label are inherited by its children and even its children's children.

[0102] In S2100, the request receiving unit 100 acquires conditions for selecting an optimal label design for learning and evaluation from the user. In this example, the following five conditions are acquired: "Condition 1: Desire to classify 'issuing company name' and 'destination company name'," "Condition 2: Classification may be performed using attributes or child labels of the classified labels," "Condition 3: The minimum support number for each label during learning is 200 or more," "Condition 4: Desire to have the F1 values ​​of both classification results as evaluation results be 80 or more," and "Condition 5: If the F1 value for half of the labels with a certain feature among the combinations of label items is below 60, skip learning of the combination of label items with that feature." These conditions are then sent to the label management unit 102, the list extraction unit 104, the labeled data extraction unit 106, and the analysis unit 209. Then, the process proceeds to S2101.

[0103] In S2101, the label management unit 102 generates a combination of label items based on the acquired conditions. Specifically, from conditions 1 and 2, the following combination of labels can be created: First, there are four possible labels for the "Issuing Company Name": two that consider the presence or absence of honorifics and two that do not, and two that are combined labels and separate labels.

[0104] Furthermore, for "destination company name," if the "destination company name" label is used as the label to be classified, there are four possible cases based on the attribute of whether or not there is a title and the attribute of the bond decomposition, as before. Furthermore, condition 2 allows for a label design using a child label of "destination company name," and if it is divided into three labels, "buyer company name," "billing company name," and "shipping company name," each can have an attribute for whether or not to include a title and an attribute for decomposition, resulting in 4 x 4 x 4 = 64 possible combinations. Also, if two of the three are considered together as one label, for example, if "Buyer company name" and "Billing company name" are considered together as one "Buyer billing company name" label, then there are four possible attributes for each of "Buyer billing company name" and "Shipping company name," giving 4 x 4 = 16 possible combinations. There are three possible combinations, so there are a total of 16 x 3 = 48 possible combinations. Therefore, there are 4 + 64 + 48 = 116 ways to combine the label items for "Destination company name." Therefore, the total number of possible combinations of label items based on Condition 1 and Condition 2 is 4×116=464.

[0105] Next, in order to satisfy condition 3, the analysis unit 209 obtains the number of supports for each label in each label design and sends the information on the number of supports to the label management unit 102. The label management unit 102 attempts to reduce the number of combinations based on the obtained number of supports.

[0106] Specifically, when checking the number of supported labels, it was found that the number of supported labels with honorifics was low for the "Publishing Company Name" label, even when the attribute of whether or not there was a title was taken into consideration. As a result, by ignoring the attribute of whether or not there was a title, there were only two combinations for "Publishing Company Name" alone.

[0107] In addition, if we consider the attribute of whether or not there is a title, the number of supported combinations will be small for "billing company name" alone, so we will not consider the attribute of whether or not there is a title.Furthermore, if we decompose it into "corporate name" and "corporate status," the number of supported combinations will also be small.As a result, by unifying the combined state, there will be only one combination of "billing company name." In addition, the number of supported "recipient company names" will be reduced if they are decomposed into "corporate name" and "legal entity." As a result, by standardizing the combined state, there will be two combinations of "recipient company names." Therefore, the combinations for the "destination company name" are: 4 ways when the "destination company name" is left as is; 4 x 1 x 2 = 8 ways when it is divided into "buyer company name," "billing company name," and "shipping company name"; 4 x 4 = 16 ways when it is divided into "buyer company name" and "billing and shipping company name," 1 x 4 = 4 ways when it is divided into "billing company name" and "buyer's shipping company name," and 2 x 4 = 8 ways when it is divided into "shipping company name" and "buyer's billing company name," for a total of 40 ways. Therefore, when condition 3 is taken into consideration, the number of combinations is 2 x 40 = 80.

[0108] The remaining conditions are not available at this step. Then, one of the combinations of label items is selected, and the process proceeds to S2102.

[0109] In S2102, the output unit 107 generates training data and evaluation data based on the combinations of label items selected one by one, in a manner similar to the flowchart shown in FIG. 7 is that in S1200, instead of obtaining a definition of a learning set from the user, a combination of label items selected by the determination unit is obtained as a learning set from the label management unit 102. Once learning data and evaluation data are created with this label setting, the process proceeds to S2103. The processes from S2102 to S2107, which will be described later, are repeated one by one for the number of combinations of label items.

[0110] In S2103, a learning evaluation unit (not shown) performs learning using the learning dataset output by the output unit 107. After the learning is completed, the learning evaluation unit evaluates the trained learning device using the evaluation dataset. The learning evaluation unit outputs the evaluation result to the evaluation result acquisition unit 208. Then, the process proceeds to S2104.

[0111] In S2104, the evaluation result acquisition unit 208 acquires the evaluation result of the evaluation data, and outputs the acquired evaluation result to the analysis unit 209 and the evaluation result storage unit 222. Then, the process proceeds to S2105.

[0112] In S2105, the analysis unit 209 acquires the evaluation results from the evaluation result acquisition unit 208, analyzes the evaluation results, and calculates an evaluation value. In this example, the F1 value is calculated based on condition 4. In this case, the evaluation is performed based on the object that is originally intended to be classified. For example, in this example, since it is desired to classify "issuing company name" and "recipient company name," even if there is a mistake due to confusion between "buyer company name" and "billing company name," which are child labels of "recipient company name," it can be classified as "recipient company name" and can be evaluated as correct. The evaluation value calculated by the analysis unit 209 is output to and stored in the evaluation result storage unit 222. Then, processing proceeds to S2106.

[0113] In S2106, the label management unit 102 prunes the remaining label item combinations as necessary. Specifically, based on condition 5 in this example, if the F1 value is below "60" for half of the labels in the label item combinations that have a certain characteristic, learning of the label item combination with that characteristic is omitted. For example, of the 80 possible combinations to be evaluated, there are 40 patterns in which the "issuing company name" is decomposed into "corporate name" and "legal entity" as attributes. When the F1 value for "issuing company name" falls below "60" for half of these (20 or more), the pattern in which the "issuing company name" is decomposed into "corporate name" and "legal entity" is deleted from the label item combinations, thereby reducing the number of times steps S2102 to S2107 are performed. Then, the process returns to S2101. If there is no deletion target that meets the condition, the process proceeds to S2107 without performing any processing on the combination of label items in S2106.

[0114] In S2107, a determination unit (not shown) determines whether any valid combinations of label items remain. Here, a valid combination of label items refers to a combination of label items that has not yet been trained or evaluated, among the combinations of label items that take conditions into consideration. If valid combinations of label items still remain (Yes in S2107), one combination is selected from them and notified to the label management unit 102, and the process returns to S2102. On the other hand, if there are no valid label item combinations remaining (No in S2107), the process proceeds to S2108.

[0115] In S2108, the label management unit 102 queries the evaluation result storage unit 222 and acquires the combination of label items that has the best evaluation value that satisfies the conditions. For example, in this example, the "Issuing Company Name" does not take into account honorifics, and "Corporate Name" and "Legal Entity" are combined labels. The "Destination Company Name" is divided into two labels, "Buyer Company Name" and "Billing Address Destination Company Name". The "Buyer Company Name" is then labeled with the presence or absence of honorifics, and "Corporate Name" and "Legal Entity" are combined labels. The combination of label items with the highest F1 value is selected, and this combination is acquired. This completes the process.

[0116] In this way, even in cases where there are many combinations of label items, such as whether it is better to subdivide label items under the conditions presented by the user and have the system learn, or whether it is better to learn using broad label items, or what the attributes of each label should be, it is possible to automatically generate combinations and then automatically obtain the appropriate combination of label items for the desired evaluation value.

[0117] With the above-described configuration, learning data and evaluation data can be output based on the assigned label items, learning and evaluation can be performed, and the results can be analyzed and used for label design.

[0118] (Modification of the second embodiment) In the example shown in the flowchart of FIG. 16, which illustrates an example of a processing procedure for selecting an optimal label design under certain conditions, the learning and evaluation in S2103 was described as being performed after a sufficient number of epochs of learning until convergence. However, evaluation may be terminated before a sufficient number of epochs have been performed. If it is predicted that the evaluation results at the end of learning will be inferior, for example, if the rate of loss decrease during learning is clearly slower than the results obtained when the training data and evaluation data from S2102 to S2107 in this processing procedure are different, learning may be terminated at that point. Terminating the processing early in this way eliminates unnecessary processing and shortens the time required to obtain the final label design results.

[0119] In the above-described embodiments, a character string (text) is used as an example of data to be labeled, but the data is not limited to text and may be an image, video, audio, vector data, etc.

[0120] As described above, according to each embodiment, it is possible to improve the efficiency of label design when assigning labels to data.

[0121] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and that the data may be configured in various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device. Furthermore, the present invention also includes any combination of the above embodiments.

[0122] Other Embodiments The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the present invention may be applied to a system made up of multiple devices, or to an apparatus made up of a single device. The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In other words, all configurations that combine the above-described embodiments and their modifications are included in the present invention.

[0123] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a label management means for managing label items to be assigned to data; an assigning means for assigning the label items managed by the label managing means to the data; a data management means for managing the label items and the data to which the labels are assigned; a storage means for storing information about a label assigned to the data, the label management means has a function of applying a relation attribute, which is a relationship between a plurality of label items, between the label items and managing the relation attribute; The information processing apparatus is characterized in that the assigning means controls so that, to the data to which the label item has been assigned, a label item that has a predetermined relationship with the label item is also assigned. (Configuration 2) The relationship attribute is at least one of a parent-child relationship, a sibling relationship, and a bond-resolving relationship; The parent-child relationship indicates the relationship between a certain label item and a label item that further classifies the label item, The sibling relationship indicates a relationship between label items having the same parent, The combination / decomposition relationship indicates a relationship in which a label item in a combination state can be decomposed into a plurality of decomposition state labels as constituent elements, 2. The information processing device according to configuration 1, wherein the assigning means controls so that, to data to which a child label item has been assigned, a parent label item of the child label item is also assigned. (Configuration 3) The information processing device according to configuration 1 or 2, wherein the label management means allows a plurality of sets consisting of a portion of the label items to be registered as target label groups, and when presenting label items to be assigned to a user, presents label items belonging to a selected target label group as the label items to be assigned. (Configuration 4) the label management means holds a set of at least one target label group as a learning set; 4. The information processing device according to configuration 3, wherein label items to be assigned are presented to the user based on the learning set. (Configuration 5) further comprising a list extraction means for extracting data to be presented to a user; The information processing device according to configuration 4, wherein the list extraction means manages whether or not each of the target label groups has been presented to the user for each unit of the data, and extracts the data that has not yet been presented as a list. (Configuration 6) The information processing device according to configuration 5, wherein when a new label item corresponding to the relational attribute is defined, the list extraction means extracts a list of candidates for data to be labeled from each piece of data based on the relationship of the relational attribute. (Configuration 7) The information processing device according to configuration 6, characterized in that when a new label corresponding to the relational attribute is defined, the assigning means assigns a label to the data extracted by the list extracting means based on the relationship of the relational attribute, automatically or in accordance with a user's selection. (Configuration 8) The information processing device according to configuration 7, characterized in that when a parent label corresponding to at least one or more child labels is newly defined as a label of the parent-child relationship, the assignment means automatically assigns the parent label to each piece of data to which the child label is assigned. (Configuration 9) further comprising a display control means for controlling the content of a display to be presented to a user; The information processing device according to any one of configurations 3 to 8, wherein the display control means presents relationships between label items based on the label items and the target label group managed by the label management means, and accepts input from a user, thereby enabling the user to design labels by adding label items and defining relationship attributes. (Configuration 10) further comprising an analysis means for performing analysis based on the results of learning and evaluation using the labeled data; the analysis means calculates an evaluation value for each label of at least one of the number of supports, recall, and precision for each label item of the training data and the evaluation data; 10. The information processing device according to configuration 9, wherein the display control means displays the value and its attribute for each of the label items determined by the analysis means. (Configuration 11) the analysis means generates at least one of an improvement plan for the label design and a countermeasure plan based on the data based on the calculated values ​​for each of the label items and their attributes; 11. The information processing device according to configuration 10, wherein the display control means displays the generated plan. (Configuration 12) 12. The information processing device according to configuration 11, wherein the label management means generates combinations of label items that satisfy conditions based on conditions related to labels and / or evaluations, and selects from among those combinations. (Method 1) 1. A control method for an information processing device comprising: label management means for managing label items to be assigned to data; label assignment means for assigning the label items managed by the label management means to the data; data management means for managing the label items and the data to which the labels have been assigned; and storage means for storing information relating to the labels assigned to the data, a step of applying a relation attribute, which is a relationship between a plurality of label items, between the label items and having the label management means manage the relation attribute; a step of controlling the data to which the label item has been assigned so that a label item having a predetermined relationship with the label item is also assigned; 1. A method for controlling an information processing device, comprising: (Program 1) 13. A program for causing a computer to function as each of the means according to any one of configurations 1 to 12. [Explanation of symbols]

[0124] 1. Information processing equipment 100 Request receiver 101 Data Management Department 102 Label Management Department 103 Labeling Unit 104 List Extraction Unit 105 Display control unit 106 Labeled Data Extraction 107 Output section 120 Data storage unit 121 Label-related information storage unit

Claims

1. a label management means for managing label items to be assigned to data; an assigning means for assigning the label items managed by the label managing means to the data; a data management means for managing the label items and the data to which the labels are assigned; a storage means for storing information about a label assigned to the data, the label management means has a function of applying a relation attribute, which is a relationship between a plurality of label items, between the label items and managing the relation attribute; The information processing apparatus is characterized in that the assigning means controls so that, to the data to which the label item has been assigned, a label item that has a predetermined relationship with the label item is also assigned.

2. The relationship attribute is at least one of a parent-child relationship, a sibling relationship, and a bond-resolving relationship; The parent-child relationship indicates the relationship between a certain label item and a label item that further classifies the label item, The sibling relationship indicates a relationship between label items having the same parent, The combination / decomposition relationship indicates a relationship in which a label item in a combination state can be decomposed into a plurality of decomposition state labels as constituent elements, 2. The information processing apparatus according to claim 1, wherein said assigning means controls so that, to data to which a child label item has been assigned, a parent label item of the child label item is also assigned.

3. 3. The information processing device according to claim 2, wherein the label management means allows a plurality of sets consisting of a portion of the label items to be registered as target label groups, and when presenting label items to be assigned to a user, presents label items belonging to a selected target label group as the label items to be assigned.

4. the label management means holds a set of at least one of the target label groups as a learning set; The information processing apparatus according to claim 3 , wherein label items to be assigned are presented to a user based on the learning set.

5. further comprising a list extraction means for extracting data to be presented to a user; 5. The information processing device according to claim 4, wherein the list extraction means manages whether or not each of the target label groups has been presented to the user for each unit of the data, and extracts the data that has not yet been presented as a list.

6. 6. The information processing device according to claim 5, wherein when a new label item corresponding to the relational attribute is defined, the list extraction means extracts a list of candidates for data to be labeled from each piece of data based on the relationship of the relational attribute.

7. The information processing device according to claim 6, characterized in that when a new label corresponding to the relational attribute is defined, the assignment means assigns the label to the data extracted by the list extraction means based on the relationship of the relational attribute, automatically or in accordance with a user's selection.

8. 8. The information processing device according to claim 7, wherein when a parent label corresponding to at least one child label is newly defined as a label in the parent-child relationship, the assigning means automatically assigns the parent label to each piece of data to which the child label is assigned.

9. further comprising a display control means for controlling the content of a display to be presented to a user; The information processing device according to any one of claims 3 to 8, characterized in that the display control means presents relationships between label items based on the label items and the target label group managed by the label management means, and accepts input from a user, thereby enabling the user to design labels by adding label items and defining relationship attributes.

10. further comprising an analysis means for performing analysis based on the results of learning and evaluation using the labeled data; the analysis means calculates an evaluation value for each label of at least one of the number of supports, recall, and precision for each label item of the training data and the evaluation data; 10. The information processing apparatus according to claim 9, wherein the display control means displays the value and attribute of each of the label items determined by the analysis means.

11. the analysis means generates at least one of an improvement plan for the label design and a countermeasure plan based on the data based on the calculated values ​​for each of the label items and their attributes; 11. The information processing apparatus according to claim 10, wherein the display control means displays the generated plan.

12. 12. The information processing apparatus according to claim 11, wherein the label management means generates combinations of label items that satisfy conditions based on conditions related to labels and / or evaluations, and selects from among those combinations.

13. 1. A control method for an information processing device comprising: label management means for managing label items to be assigned to data; label assignment means for assigning the label items managed by the label management means to the data; data management means for managing the label items and the data to which the labels have been assigned; and storage means for storing information relating to the labels assigned to the data, a step of applying a relation attribute, which is a relationship between a plurality of label items, between the label items and having the label management means manage the relation attribute; a step of controlling the data to which the label item has been assigned so that a label item having a predetermined relationship with the label item is also assigned; 1. A method for controlling an information processing device, comprising:

14. A program for causing a computer to function as each of the means according to claim 1.

Images