Labeling device, labeling method, and program

Abstract

To provide a labeling device capable of appropriately selecting a label to be assigned to text data.SOLUTION: An extraction unit 201 computes a confidence level for each of multiple candidate labels being candidates for an assigned label to be assigned to text data 101. A label assigning unit 202 computes a sum of confidence levels of anonymization target labels among the candidate labels as a target confidence level, and selects the assigned label to be assigned to a token from among the candidate labels on the basis of the confidence levels of the candidate labels and the target confidence level.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a labeling device, a labeling method, and a program.

Background Art

[0002] In recent years, named entity recognition that extracts named entities such as organization names and personal names from text data has attracted attention. For example, in Patent Document 1, in order to improve the extraction accuracy of a named entity recognition model, which is a machine learning model for extracting named entities, a technique for increasing training data by enabling the use of unlabeled place names as training data is disclosed.

[0003] In addition, as a technique using named entity recognition, there is a text data anonymization technique. In the anonymization technique, named entity recognition is used to assign a named entity label, which is a label corresponding to a named entity, to each element of text data, and the element to which a predetermined named entity label is assigned is anonymized. For example, anonymization is performed by deleting an element (e.g., "Tanaka") to which a named entity label indicating "personal name" is assigned, or replacing it with another element (e.g., "Person A").

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The extraction results of named entity recognition typically include multiple named entity labels and their confidence levels for each element of text data. In anonymization techniques, for example, the named entity label with the highest confidence level is selected for each element, and anonymization is performed according to the selected named entity label. However, for expressions that are used as organization names, personal names, and place names, such as "Hitachi" and "Kawasaki," or expressions that are used as facility names and place names, such as "Kokubunji," the extraction accuracy of named entity recognition becomes low, and there may be multiple named entity labels with similar confidence levels. In such cases, methods such as selecting the named entity label with the highest confidence level may not be able to select the appropriate named entity label, and anonymization may not be performed properly.

[0006] Furthermore, the problems described above can occur not only in named entity recognition but also in other technologies that label text data, such as text classification technologies that assign text classification labels.

[0007] The purpose of this disclosure is to provide a labeling device, a labeling method, and a program that can appropriately select labels to be assigned to text data. [Means for solving the problem]

[0008] A labeling device according to one aspect of the present disclosure is a labeling device that assigns a label to text data according to the content of the text data, comprising: a calculation unit that analyzes the text data and calculates a confidence score representing the likelihood that each of a plurality of candidate labels that are candidates for the assignment label to be assigned to the text data is the assignment label; and an assignment unit that calculates the sum of the confidence scores of a plurality of target labels among the plurality of candidate labels that are subject to processing for a predetermined process, and selects the assignment label from the candidate labels and assigns it to the text data based on the confidence score of each candidate label and the sum. [Effects of the Invention]

[0009] According to the present invention, it becomes possible to appropriately select labels to be assigned to text data. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the configuration of the anonymization device according to the first embodiment of this disclosure. [Figure 2] This is a flowchart illustrating an example of the overall process in the first embodiment of this disclosure. [Figure 3] This figure illustrates an example of the main processing in the first embodiment of this disclosure. [Figure 4] This figure illustrates a reference example of the main processing in the first embodiment of this disclosure. [Figure 5] This figure illustrates an example of the main processing in a second embodiment of the present disclosure. [Figure 6] This figure illustrates a reference example of the main processing in a second embodiment of the present disclosure. [Figure 7] This figure illustrates another example of the main processing in the second embodiment of the present disclosure. [Figure 8] This figure illustrates a reference example of the main processing in other embodiments of the present disclosure. [Modes for carrying out the invention]

[0011] The embodiments of this disclosure will be described below with reference to the drawings.

[0012] (First Embodiment) Figure 1 is a diagram showing the configuration of an anonymization device according to the first embodiment of the present disclosure. The anonymization device 1 shown in Figure 1 has the function of a labeling device that assigns a label to text data according to the content of the text data, and the function of a processing device that performs predetermined processing on the labeled text data.

[0013] In this embodiment, a label is assigned to each of a plurality of elements that constitute text data, namely tokens. More specifically, the label is an entity label indicating an entity expression, and is assigned to each token of the text data. The entity label indicates, for example, "organization name", "person name", "facility name", "place name", "date", "time", "amount of money", or "percentage" as an entity expression.

[0014] In this embodiment, the predetermined process is an anonymization process for anonymizing text data, but any process that performs a process according to the label may be used.

[0015] As shown in FIG. 1, the anonymization device 1 includes a storage unit 10, a processing unit 20, an input unit 30, a display unit 40, and a communication unit 50. The anonymization device 1 is also communicably interconnected with a recording medium 2 and a user terminal 3.

[0016] The storage unit 10 stores a program (computer program) 100 that defines the operation of the processing unit 20. There may be a plurality of programs. Note that the program 100 can be stored in the recording medium 2. The recording medium 2 is a non-temporary storage medium such as a flash memory or a hard disk drive, and is readable by a computer. In FIG. 1, the recording medium 2 is directly connected to the anonymization device 1, but may be connected to the anonymization device 1 via a communication network 4.

[0017] The storage unit 10 also stores various types of information used and generated by the process of the program 100. For example, the storage unit 10 stores text data 101, extraction result information 102, target specification information 103, assigned label information 104, and anonymized text data 105.

[0018] Text data 101 is text data to be processed in the anonymization device 1. The extraction result information 102 is the result of the named entity extraction process that extracts named entities from the text data 101. The target specification information 103 is information that specifies the processing target of a predetermined process, i.e., the anonymization process. In the present embodiment, the target specification information 103 indicates an anonymization target label, which is a named entity label corresponding to a token to be anonymized in the anonymization process, as the target label of the processing target. There are a plurality of anonymization target labels in the present embodiment. The assigned label information 104 is information indicating the named entity label assigned to each token of the text data 101. Note that the assigned label information 104 may indicate the confidence level (described later) of the named entity label in addition to the named entity label. The anonymized text data 105 is the result of the anonymization process.

[0019] The processing unit 20 is a processor such as a CPU (Central Processing Unit), reads the program stored in the storage unit 10, and executes the read program to realize various functions. In the present embodiment, the processing unit 20 realizes a control unit 200, an extraction unit 201, a labeling unit 202, and an anonymization unit 203 by executing a program.

[0020] The control unit 200 controls the entire processing unit 20.

[0021] The extraction unit 201 is a calculation unit that performs a named entity extraction process on the text data 101 and calculates (generates) the extraction result information 102. The extraction unit 201 performs the named entity extraction process using, for example, a learned machine learning model learned for performing the named entity extraction process. The type of the machine learning model and the learning data used for learning are not particularly limited. Further, learning may be performed by the extraction unit 201, or a learned machine learning model learned by another device may be used.

[0022] The label assignment unit 202 generates assigned label information 104 by selecting and assigning an assigned label, which is a named entity label to be assigned to each token in the text data 101, based on the extraction result information 102 and the target designation information 103.

[0023] The anonymization unit 203 is a predetermined processing unit that performs an anonymization process, and in this embodiment, it performs an anonymization process on the text data 101 based on the target designation information 103 and the assigned label information 104 to generate anonymized text data 105.

[0024] Figure 2 is a flowchart illustrating an example of the overall processing performed by the anonymization device 1.

[0025] In the overall processing, first, the control unit 200 acquires the text data 101 and stores it in the storage unit 10 (step S101). The control unit 200 also acquires the target designation information 103 and stores it in the storage unit 10 (step S102). For example, the control unit 200 acquires the text data 101 and target designation information 103 that have been input to the input unit 30 or the user terminal 3. Alternatively, if information indicating the storage location of the text data 101 and target designation information 103 is input to the input unit 30 or the user terminal 3, the control unit 200 may acquire the text data 101 and target designation information 103 from that storage location.

[0026] Next, the extraction unit 201 performs named entity recognition processing on the text data 101 to generate extraction result information 102 and stores it in the storage unit 10 (step S103). Specifically, the named entity recognition processing in this embodiment involves dividing the text data 101 into tokens and calculating a confidence score for each of the multiple named entity labels (hereinafter sometimes referred to as candidate labels) that are candidates for the label to be assigned to the token. In this embodiment, the candidate labels are common to each token. The confidence score is an index value that represents the likelihood of a token being assigned, and is normalized so that the sum of the confidence scores of each candidate label is a predetermined value (in this embodiment, "1").

[0027] The label assignment unit 202 generates and stores the assigned label information 104 in the storage unit 10 by selecting an assigned label, which is a named entity label to be assigned to each token of the text data 101, from among the candidate labels based on the extraction result information 102 and the target designation information 103 (step S104).

[0028] The anonymization unit 203 performs an anonymization process on the text data 101 based on the target designation information 103 and the assigned label information 104 to generate anonymized text data 105 and store it in the storage unit 10 (step S105). In the anonymization process, the anonymization unit 203 anonymizes the tokens in the text data 101 that have been assigned the anonymization target label indicated in the target designation information 103. Token anonymization can be done by, for example, deleting, blacking out, abstracting, dummyizing, or a combination thereof. Abstraction is the process of replacing a token with a higher-level concept, such as replacing the token "dog" with "animal". In abstraction, a token may also be replaced with a term corresponding to the named entity label assigned to that token (for example, the name of the named entity label). Dummyizing is the process of replacing a token representing "date" with another date, for example.

[0029] Then, the control unit 200 displays the anonymous text data 105 on the display unit 40 or the user terminal 3 (step S106) and terminates the process. In step S106, the control unit 200 may also display other information in addition to the anonymous text data 105, such as extraction result information 102, assigned label information 104, or both.

[0030] Figure 3 is a diagram illustrating an example of the main process, which corresponds to steps S103 to S105 in Figure 2. In the example in Figure 3, text data 101 indicates "Tanaka visited Hitachi with Tomita," and the labels to be anonymized are "organization name" and "person's name."

[0031] In this embodiment, named entity labels are represented using the BIO (Begin Inside Outside) method. The BIO method assigns "B (Begin)" to the first token when multiple consecutive tokens represent a single named entity, and to the named entity label for a single token when that token represents a single named entity; "I (Inside)" is assigned to the second and subsequent tokens when multiple consecutive tokens represent a single named entity; and "O (Outside)" is assigned to tokens that are not named entities. In the diagram, "organization name" is represented as "ORG," "person name" as "PER," and "place name" as "LOC." Named entity labels may also be represented using other methods, such as the BIOES (Begin Inside Outside End Single) method.

[0032] In the main processing, first, in step S103, the extraction unit 201 performs named entity recognition processing on the text data 101 "Tanaka visited Tomita and Hitachi," and calculates the confidence level for each candidate label, which is a named entity label, for each of the tokens "Tanaka," "ga," "Tomita," "to," "Hitachi," "o," and "visited," and generates extraction result information 102.

[0033] In step S104, the label assignment unit 202 calculates the target confidence score for each token by summing the confidence scores of the anonymization target labels "B-ORG" and "B-PER" from the candidate labels, and selects a label to assign to the token from among the candidate labels based on the confidence scores of the candidate labels and the target confidence score.

[0034] Specifically, the labeling unit 202 selects the label to be assigned if the candidate label with the highest confidence level among the candidate labels is the label to be anonymized. If the candidate label with the highest confidence level is not the label to be anonymized, the labeling unit 202 determines whether the target confidence level is equal to or greater than the highest confidence level.

[0035] If the confidence level of the target is equal to or greater than the maximum confidence level, the labeling unit 202 selects a new anonymized target label corresponding to the anonymized target label as the assigned label. On the other hand, if the confidence level of the target is less than the maximum confidence level, it selects a candidate label with the maximum confidence level as the assigned label.

[0036] For example, in the case of the token "Tomita," the confidence levels of the candidate named entity labels "B-ORG," "B-PER," "B-LOC," ... "O" are calculated as "0.2," "0.3," "0.4," ... "0.1." In this case, the labeling unit 202 determines whether the target confidence level "0.5 (0.2 + 0.3)," which is the sum of the confidence levels of the anonymized labels "B-ORG" and "B-PER," is greater than or equal to the maximum confidence level "0.4," since the named entity label "B-LOC," which has the highest confidence level of "0.4," is not a label to be anonymized. Here, since the target confidence level "0.5" is greater than or equal to the maximum confidence level "0.4," the labeling unit 202 selects a new anonymized target label corresponding to the anonymized labels "B-ORG" and "B-PER" as the assigned label. The new anonymization target label is, for example, a combination of anonymization target labels with a confidence level greater than 0 (in this embodiment, this would be "B-ORG / B-PER"). Alternatively, the labeling unit 202 may select either the anonymization target labels "B-ORG" or "B-PER" with a confidence level greater than 0, instead of selecting the new anonymization target label "B-ORG / B-PER".

[0037] In the example shown in Figure 3, the confidence levels for the anonymized labels "B-PER" and "B-ORG" are highest for the tokens "Tanaka" and "Hitachi," respectively.

[0038] In step S105, the anonymization unit 203 generates anonymized text data 105 by replacing the tokens "Tanaka", "Tomita", and "Hitachi" from the text data 101 that have been assigned an anonymization target label with an expression corresponding to the anonymization target label assigned to that token.

[0039] Figure 4 is a diagram illustrating a reference example for the main processing. In the example in Figure 4, for each token, the candidate label with the highest confidence level is selected as the assigned label. In this case, the token "Tomita" is assigned the candidate label "B-LOC," which has the highest confidence level of "0.4," and therefore the token "Tomita" is not anonymized despite being a person's name. In contrast, in this embodiment, the token "Tomita" is appropriately anonymized as shown in Figure 3.

[0040] As described above, according to this embodiment, the extraction unit 201 calculates the confidence level for each of the multiple candidate labels that are candidates for the label to be assigned to the text data 101. The label assignment unit 202 calculates the sum of the confidence levels of the labels to be anonymized among the candidate labels as the target confidence level, and selects the label to be assigned to the token from among the candidate labels based on the confidence levels of the candidate labels and the target confidence level. Therefore, since the label to be assigned is selected based on the target confidence level, which is the confidence level of all labels to be anonymized, it is possible to suppress the assignment of inappropriate candidate labels, such as when multiple labels to be anonymized with similar confidence levels are assigned and other candidate labels are assigned. Therefore, it is possible to appropriately select the label to be assigned to the text data 101.

[0041] Furthermore, in this embodiment, the labeling unit 202 selects the anonymized targeting label as the assigned label if the target confidence level is equal to or greater than the maximum confidence level, which is the highest confidence level among the candidate labels. If the target confidence level is less than the maximum confidence level, it selects the candidate label with the highest confidence level as the assigned label. In this case, it becomes possible to select a label to be assigned to the text data 101 more appropriately.

[0042] Furthermore, in this embodiment, the label assignment unit 202 selects the label to be assigned if the candidate label with the highest confidence level is the label to be anonymized, and if the candidate label with the highest confidence level is not the label to be anonymized, and the target confidence level is equal to or greater than the highest confidence level, it selects a new label to be anonymized corresponding to an existing label to be anonymized as the label to be assigned. In this case, it becomes possible to select a label to be assigned to the text data 101 more appropriately.

[0043] Furthermore, in this embodiment, the extraction unit 201 divides the text data 101 into multiple elements, which are tokens, and calculates the confidence level for each of the multiple candidate labels for each token. The label assignment unit 202 selects a label to assign to each token. Therefore, even when assigning labels to each element of the text data, it becomes possible to appropriately select the label to assign.

[0044] (Second embodiment) In the first embodiment, the delimiter of named entity labels was fixed to a single pattern. However, depending on the machine learning model used in the extraction unit 201, the delimiter of named entity labels may be divided into multiple patterns. This embodiment describes the case where the delimiter of named entity labels is divided into multiple patterns. The following mainly describes processing that differs from the first embodiment.

[0045] Figure 5 is a diagram illustrating the main processing in a second embodiment of this disclosure. In the example in Figure 5, text data 101 indicates "Visited Tokyo University Hospital," and the labels to be anonymized are "Organization Name," "Facility Name," and "Person's Name." Note that "Facility Name" is written as "FAC."

[0046] In the example in Figure 5, in step S103, the extraction unit 201 performs named entity recognition processing on the text data 101 "Visited the University of Tokyo Hospital" and calculates the confidence level for each candidate label, which is a named entity label, for each of the tokens "University of Tokyo", "affiliated", "hospital", "to", and "visited", and calculates the extraction result information 102.

[0047] Here, the extraction unit 201 calculates three patterns for each of the tokens "University of Tokyo", "affiliated", "hospital", "to", and "visited", with different delimiters for the named entity labels: Candidate 1: "B-ORG", "O", "O", "O", "O"", Candidate 2: "B-ORG", "I-ORG", "O", "O", "O"", Candidate 3: "B-FAC", "I-FAC", "I-FAC", "O", "O"". In these patterns, the range of tokens to which named entity labels are set differs from one another and overlaps. Also, for the token "University of Tokyo", the confidence level of the named entity label "Organization (ORG)" is "0.5", the confidence level of the named entity label "Facility (FAC)" is "0.5", and the confidence level of the other named entity labels is "0". Furthermore, for the token "Attached," the confidence level for the named entity label "Organization (ORG)" is "0.3," the confidence level for the named entity label "Facility (FAC)" is "0.3," and the confidence level for the named entity label "Not Applicable (O)" is "0.4." Also, for the token "Hospital," the confidence level for the named entity label "Facility (FAC)" is "0.4," the confidence level for "Not Applicable (O)" is "0.6," and the confidence level for the other named entity labels is "0."

[0048] The labeling unit 202 calculates the target confidence value for each token by summing the confidence values ​​of the anonymized labels among the candidate labels, and generates candidate 4 consisting of new anonymized labels (combinations of existing anonymized labels) that have the target confidence value. The labeling unit 202 then selects each candidate label of the candidate with the highest average confidence value of the anonymized labels as the assigned label. In this case, candidate 4 has the highest average confidence value, so the labeling unit 202 selects each candidate label of candidate 4.

[0049] Then, in step S105, the anonymization unit 203 generates anonymized text data 105 by replacing the tokens "University of Tokyo," "affiliated," and "hospital" that have been assigned an anonymization target label from among the tokens of the text data 101 with expressions corresponding to the anonymization target label assigned to those tokens.

[0050] Figure 6 is a diagram illustrating a reference example for the main processing of this embodiment. In the example in Figure 6, each candidate label of candidate 1, which has the highest average value among candidates 1 to 3, is assigned to the token. Therefore, even though "facility" is a label to be anonymized, the tokens "attached" and "hospital" are not anonymized.

[0051] In the example shown in Figure 5, the extraction unit 201 performed named entity recognition using a single machine learning model, but it is also possible to perform named entity recognition using multiple machine learning models.

[0052] Figure 7 illustrates an example of the main process when named entity recognition is performed using multiple machine learning models. In the example in Figure 7, there are two machine learning models, Model 1 and Model 2.

[0053] In the example in Figure 7, the extraction unit 201 generates candidates similar to those in the example in Figure 5 for each machine learning model. The labeling unit 202 generates candidates for each machine learning model consisting of new anonymized labels (combinations of existing anonymized labels). From these candidates, the labeling unit 202 selects each candidate label of the candidate with the highest average confidence score of the anonymized labels as the assigned label.

[0054] As described above, according to this embodiment, even when there are multiple patterns for the delimiter of named entity labels, it becomes possible to appropriately select the label to be assigned to the text data.

[0055] (Third embodiment) In this embodiment, we describe an example in which the confidence level is corrected using the error rate of the label to be anonymized. Below, we will mainly describe a process that differs from the first embodiment.

[0056] There is a tendency for misidentification between named entity labels. For example, organizations and people and places, or places and facilities, are easily confused, while people and phone numbers are less likely to be confused. The extraction unit 201 further calculates an error rate that indicates the likelihood of misidentification between named entity labels. Specifically, the error rate represents the degree to which one named entity label is mistaken for another named entity label. For example, the extraction unit 201 calculates an error rate of "0.1" for misidentifying an "organization name" as a "person's name," and an error rate of "0.04" for misidentifying a "person's name" as an "organization name." The error rate can be calculated, for example, using a machine learning model.

[0057] The extraction unit 201 corrects the confidence level of each candidate label based on the error rate, which is the degree to which the candidate label is mistaken for other labels to be anonymized. Specifically, if the confidence level of named entity label A is C, and the error rates for mistaking named entity label A for other labels to be anonymized L1 to LN are EL1 to ELN, the extraction unit 201 corrects the confidence level C of named entity label A as C' = C × (1 - (EL1 + EL2 + ... + ELN)).

[0058] For example, if the labels to be anonymized are "person's name" and "organization name," and the confidence level for "place name" is 0.8, and the error rate for mistaking a "place name" for a "person's name" and the error rate for mistaking a "place name" for an "organization name" are both 0.1, then the corrected confidence level for "place name" would be 0.8 × (1 - (0.1 + 0.1)) = 0.64.

[0059] According to this embodiment, since the confidence level can be corrected by the error rate, it becomes possible to select the assigned label considering the likelihood of errors between named entity labels, and thus it becomes possible to appropriately select the label to be assigned to text data.

[0060] (Other embodiments) The embodiments of the Disclosure described above are illustrative for illustrative purposes and are not intended to limit the scope of the Disclosure to those embodiments only. Those skilled in the art can implement the Disclosure in various other forms without departing from the scope of the Disclosure.

[0061] For example, labels assigned to text data are not limited to named entity labels. Furthermore, labels may be assigned to the entire text data, rather than to individual tokens. An example of a label assigned to the entire text data is a text classification label used to categorize the content of that text data.

[0062] Figure 8 illustrates an example of the main processing when using text classification labels. In the example in Figure 8, text data 101 is "The government decided on measures to address price increases due to the global situation at a meeting of relevant ministers on the 26th," and the target labels are "Politics" and "Economy."

[0063] In the example shown in Figure 8, in step S103, the extraction unit 201 analyzes the text data 101, "The government decided on measures to address price increases due to the global situation at a meeting of relevant ministers on the 26th," and calculates confidence levels of "0.3," "0.3," and "0.5" for the candidate labels, which are text classification labels "Politics," "Economy," and "Other," respectively. Note that the processing by the extraction unit 201 can be performed using a machine learning model or the like.

[0064] The label assignment unit 202 calculates the target confidence score as a composite value of the confidence scores of the target labels, the text classification labels "politics" and "economics," and selects a label to assign to the text data 101 from among the candidate labels based on the confidence scores of the candidate labels and the target confidence score. In this case, since the target confidence score is greater than the confidence score of "Other," a new target label "politics / economics," which is a combination of the text classification labels "politics" and "economics," is assigned to the text data. [Explanation of Symbols]

[0065] 1: Anonymization device 2: Recording medium 3: User terminal 4: Communication network 10: Storage unit 20: Processing unit 30: Input unit 40: Display unit 50: Communication unit

Claims

1. A labeling device that assigns labels to text data according to the content of the text data, A calculation unit analyzes the text data and calculates a confidence score representing the likelihood that each of the multiple candidate labels that are candidates for the assigned label to be assigned to the text data is the assigned label. A labeling device having a labeling unit that calculates the sum of the confidence scores of a plurality of target labels that are subject to processing for a predetermined process from among the plurality of candidate labels, and selects the label to be assigned from among the candidate labels and assigns it to the text data based on the confidence score of each candidate label and the sum.

2. The labeling device according to claim 1, wherein the labeling unit compares the total value with the maximum confidence level, which is the highest confidence level among the confidence levels of each candidate label, and if the total value is equal to or greater than the maximum confidence level, selects the target label as the label to be assigned, and if the total value is less than the maximum confidence level, selects the candidate label with the maximum confidence level as the label to be assigned.

3. The labeling device according to claim 2, wherein the labeling unit determines whether the candidate label with the highest confidence level is the target label, and if the candidate label with the highest confidence level is the target label, it selects the target label as the labeling label, and if the candidate label with the highest confidence level is not the target label, it performs the comparison and, if the sum is equal to or greater than the highest confidence level, it selects a new target label corresponding to the plurality of target labels as the labeling label.

4. The labeling device according to claim 1, wherein the calculation unit corrects the confidence level of each candidate label based on an error rate that represents the degree to which the candidate label is mistaken for a target label other than the candidate label.

5. The calculation unit divides the text data into multiple elements, and calculates the confidence level for each of the multiple candidate labels for each element. The labeling device according to claim 1, wherein the labeling unit selects the labeling label for each element.

6. The labeling apparatus according to claim 5, wherein the label is a named entity label.

7. The candidate label is set to a range that includes one or more of the elements, The labeling device according to claim 5, wherein, if there are multiple target labels which are candidate labels whose element ranges are different from each other and overlap, the calculation unit calculates the sum value for each element in the range, adds a new target label to the candidate labels which has the sum value as the confidence level, and selects the candidate label which has the highest average confidence level for each element of each candidate label as the assigned label.

8. The labeling device according to claim 5, further comprising a processing unit that performs the predetermined processing on each element of the text data to which the target label has been assigned.

9. The labeling apparatus according to claim 8, wherein the predetermined processing is an anonymization process that anonymizes the elements to which the target label has been assigned.

10. A labeling method using a labeling device that assigns labels to text data according to the content of the text data, The text data is analyzed, and a confidence score is calculated for each of the multiple candidate labels that are candidates for the assigned label to be attached to the text data, representing the likelihood that each of them is the assigned label. A label assignment method comprising: calculating the sum of the confidence scores of multiple target labels from among the multiple candidate labels that are subject to processing for a predetermined process; and selecting the assignment label from among the candidate labels and assigning it to the text data based on the confidence score of each candidate label and the sum of the confidence scores.

11. A program that causes a computer to perform a labeling process that assigns labels to text data according to the content of the text data, The label assignment process described above is: A process of analyzing the text data and calculating a confidence score representing the likelihood that each of the multiple candidate labels that are candidates for the assigned label to be attached to the text data is the assigned label, A program that includes the process of calculating the sum of the confidence scores of a plurality of target labels that are subject to processing for a predetermined process from among the plurality of candidate labels, and selecting the assignment label from among the candidate labels and assigning it to the text data based on the confidence score of each candidate label and the sum.

Images