Named entity output device and program

Abstract

Improve the accuracy of the named entity recognition output. [Solution] The named entity output device comprises: a text input unit that acquires document data; a named entity extraction unit that extracts named entities, which are proper nouns or specific phrases in the field of the document data, from the acquired document data; a named entity verification unit that performs verification including determining whether the extracted named entities are correct or incorrect; a named entity modification unit that modifies the named entity if it is determined in the verification that it is possible to modify the named entity; a response rejection unit that rejects the named entity if it is determined in the verification that it is impossible to modify the named entity; and a named entity output unit that outputs the named entity after the verification and modification.

Description

[Technical Field]

[0001] The present invention relates to a named entity output device and a program. [Background Art]

[0002] Conventionally, techniques for extracting named entities from document information and classifying categories of named entities by using an inference model have been proposed (see, for example, Patent Document 1). [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2022-011533 [Summary of the Invention] [Problems to be Solved by the Invention]

[0004] According to the above-described conventional technology, there is a problem in that span errors and label errors frequently occur in extracted entities, which lowers the accuracy rate of output named entities.

[0005] An object of the present invention is to provide a named entity output device and a program that can improve the accuracy rate of output named entities. [Means for Solving the Problems]

[0006] One aspect of the present invention is a named entity output device comprising: a text input unit for acquiring document data; a named entity extraction unit for extracting named entities, which are proper nouns or specific phrases in the field of the document data, from the acquired document data; a named entity verification unit for performing verification, including determining whether the extracted named entities are correct or incorrect; a named entity modification unit for modifying the named entities if the verification determines that the named entities can be modified; a response rejection unit for rejecting the named entities if the verification determines that the named entities cannot be modified; and a named entity output unit for outputting the named entities that have undergone the verification and modification.

[0007] One aspect of the present invention relates to a computer provided in a named entity output device, Acquiring document data and Extracting proper nouns or specific phrases within the field of the document data from the acquired document data, The process involves verifying the extracted named entities, including determining whether they are correct or incorrect. If, in the verification described above, it is determined that the named entity can be modified, the named entity shall be modified. If, in the verification described above, it is determined that the named entity cannot be modified, the named entity shall be rejected. Outputting the named entity after the verification and correction described above, This is a program to execute [the command / action]. [Effects of the Invention]

[0008] According to the present invention, the accuracy of the output named entities can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overview of the functional configuration of the named entity output system of this embodiment. [Figure 2] This figure shows an overview of the procedure for extracting named entities E using the named entity output system of this embodiment. [Figure 3]This figure shows an example of the configuration of the training data generation function of this embodiment. [Figure 4] This figure shows an example of the procedure for generating training data in this embodiment. [Figure 5] This figure shows an example of error classification in this embodiment. [Figure 6] This figure shows a specific example of the functional configuration of the named entity output system of this embodiment. [Figure 7] This diagram illustrates an example of the operation flow of the named entity output system of this embodiment. [Figure 8] This figure shows an example of document data for this embodiment. [Figure 9] This figure shows an example of a candidate set of named entities in this embodiment. [Figure 10] This figure shows an example of a candidate set of named entities in the case of errors in this embodiment. [Figure 11] This figure shows an example of the process for correcting named entities by the named entity correction unit of this embodiment. [Figure 12] This figure shows an overview of the functional configuration of the modified named entity output system. [Figure 13] This figure shows an example of the functional configuration of a modified named entity output system. [Modes for carrying out the invention]

[0010] The named entity output system 1 of this embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. In all the figures used to illustrate the embodiments, components with the same function are given the same reference numerals, and repeated explanations are omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Also, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on something that has been calculated or processed from XX. "XX" is any element (for example, any information).

[0011] [Embodiment] Hereinafter, an outline of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of an outline of a functional configuration of a named entity output system 1 according to the present embodiment. The named entity output system 1 is a system that extracts a named entity E contained in document data D1, which is information including text, from said document data D1.

[0012] In general, in named entity extraction processing, span errors and label errors of text named entities (also referred to as entities; the same applies hereinafter) occur frequently, which degrades extraction performance. Results of experiments on conventional named entity extraction processing have revealed that about 40% of situations where extraction performance is degraded can be corrected by correcting errors output by the named entity extraction processing, thereby improving extraction performance.

[0013] (Prior Art) Conventionally, named entity extraction techniques for automatically extracting named entities (proper nouns and phrases) from sentences are known. By using this named entity extraction technique, for example, phrases such as material names, functions, and characteristics can be extracted from documents in technical specialized fields such as chemical documents. The extracted named entities can be secondarily used as structured data for search, analysis, and the like.

[0014] In named entity extraction, the objective is to completely extract entities and labels as a set from sentences. As an example, if a chemical document contains the text "The pigment contains pigment violet 19 and pigment orange 71.", it is desirable that "pigment violet 19" and "pigment orange 71" are extracted as entities, "component" is extracted as the label for "pigment violet 19", and "component" is extracted as the label for "pigment orange 71", respectively.

[0015] However, conventional named entity recognition techniques are known to frequently produce extraction errors in both entities and labels. In the example above, an error may occur where "pigment violet 19" is extracted as the entity, but "pigment" is extracted instead (span error), or where "feature" is extracted as the label for "pigment orange 71" instead of "component" (label error). Such span and label errors are one of the factors that impair the performance of named entity recognition, and outputting an incorrect named entity E can lead to a deterioration in the accuracy of subsequent searches and analyses.

[0016] If the language model can autonomously determine the presence and type of errors from the output of named entity recognition and make optimal corrections, the number of recognition errors can be significantly reduced.

[0017] The named entity output system 1 of this embodiment suppresses extraction errors by constructing a language model (LLM_(NER+SC)) that has both of the following capabilities (1) and (2). (1) Language model that learns named entity recognition (LLM_NER) (2) A language model that self-corrects its output (LLM_SC) In the following explanation, these language models will also be referred to as trained models M. Furthermore, the language model that learns named entity recognition (LLM_NER) will be simply referred to as LLM_NER, and the language model that self-corrects its output (LLM_SC) will be simply referred to as LLM_SC.

[0018] LLM_NER learns the correspondence between input text (e.g., text in a specific technical field) and named entities E (e.g., named entities in that technology). When given text, LLM_NER is trained to output named entities E and their labels (also called entity types or tags). More specifically, LLM_NER has a span extraction function that extracts the span of entities from a text, and an entity type estimation function that estimates the type of those entities.

[0019] LLM_SC consists of a named entity verification model (LLM_SV) and a named entity correction model (LLM_SR). In the following description, the named entity verification model (LLM_SV) will also be referred to simply as LLM_SV, and the named entity correction model (LLM_SR) will also be referred to simply as LLM_SR.

[0020] LLM_SV is trained to output whether a named entity E (i.e., the predicted representation by LLM_NER) and its label, outputted by LLM_NER, is correct or incorrect. LLM_SV also outputs the type of error (error classification) if the predicted representation and label are incorrect.

[0021] LLM_SR is trained to output the correct named entity E and label when given the named entity E and label containing errors output by LLM_NER, and the error classification output by LLM_SV.

[0022] Figure 2 is a diagram illustrating the outline of the procedure for extracting named entities E using the named entity output system 1 of this embodiment. (STEP 1) A language model (LLM_NER) trained on named entity recognition extracts named entity E from document data D1. LLM_NER may also generate named entity E based on document data D1. In the following explanation, when the extraction of named entity E and the generation of named entity E are not distinguished, they will be collectively referred to as the extraction of named entity E. (STEP2) The named entity E extracted in STEP1 is validated using a self-correcting language model (LLM_SC) and error detection is performed. (STEP3) If the results of the verification in STEP2 indicate that error correction is possible, the error correction is performed by the self-correcting language model (LLM_SC) of the output.

[0023] In the following explanation, the pre-trained model M used in STEP 1 will also be referred to as the named entity recognition model or the first trained model M1. The pre-trained model M used in STEP 2 will also be referred to as the named entity verification model (LLM_SV) or the second trained model M2. The pre-trained model M used in STEP 3 will also be referred to as the named entity correction model (LLM_SR) or the third trained model M3.

[0024] [Automatic generation of training data] The named entity output system 1 of this embodiment may also include a function to automatically generate training data used in the training phase of the self-correcting language model (LLM_SC) that produces the output results described above.

[0025] Figure 3 shows an example of the configuration of the training data generation function of this embodiment. A named entity recognition model (LLM_NER) is pre-trained independently, and named entity recognition is performed on the dataset used for training to obtain a candidate set of named entities E. (A) Self-verification: Each candidate named entity is compared with the set of named entities in the correct data to obtain the correct / incorrect label and error classification for self-verification. (B) Self-correction: Each candidate named entity is compared with the set of named entities in the correct data, and the most appropriate named entity as a correction for the candidate is obtained from the set of named entities in the correct data.

[0026] Figure 4 shows an example of the procedure for generating training data in this embodiment. For each named entity E included in the candidate set of named entities E, it is compared with the correct data, and based on the comparison result, labels for correctness, error classification, and self-correction are automatically assigned.

[0027] [About error classification] Figure 5 shows an example of error classification in this embodiment. The error classification labels include the following five types.

[0028] (1) Uncorrectable: This occurs when the extracted entity of named entity E is a string that is completely different from the correct answer. For example, when extracting named entity E from document data D1 containing the text "The pigment contains pigment violet 19 and pigment orange 71." as in the example above, the entity should be extracted as "pigment violet 19," but instead "contains" is extracted. The word "contains" is a common verb and is not named entity E.

[0029] (2) Over-extraction (excessive span): This occurs when the label of the extracted named entity E matches the correct label, but the entity is over-extracted. For example, in the example above, the entity should have been extracted as "pigment violet 19," but instead it was extracted as "pigment violet 19 and pigment orange 71."

[0030] (3) Insufficient extraction (under-span): This occurs when the label of the extracted named entity E matches the correct label, but the entities are under-extracted. For example, in the example above, the entity should have been extracted as "pigment violet 19," but instead it was extracted as "pigment." The aforementioned over-extraction and under-extraction are collectively referred to as span errors.

[0031] (4) Label (Label error): This occurs when the entity of the extracted named entity E matches the correct entity, but the label does not match the correct label. For example, in the example above, the label should have been extracted as "component," but it was extracted as "feature." In the following explanation, a labeling error will also be referred to as a labeling mistake.

[0032] (5) Span error + label error: This occurs when the entity of the extracted named entity E is a span error and the label does not match the correct label. For example, in the example above, the entity should have been extracted as "pigment violet 19" and the label as "component," but instead "pigment" and "feature" were extracted.

[0033] Of the error types mentioned above, the four types other than those that are uncorrectable (i.e., over-sampling, under-sampling, incorrect labeling, and a combination of span errors and labeling errors) can be corrected.

[0034] In other words, the types of errors that can correct named entities E include over-extraction of entities, under-extraction of entities, mislabeling, and combinations of over-extraction or under-extraction and mislabeling. In the following explanation, the type of error will also be referred to as the "error classification."

[0035] The named entity output system 1 automatically assigns one of the five types of labels mentioned above to the training data when generating the training data.

[0036] [Specific example of the functional configuration of the named entity output system x] Up to this point, we have described the general overview of the functional configuration of the Named Entity Output System 1. Below, we will explain specific examples of the functional configuration of the Named Entity Output System 1 with reference to the diagrams.

[0037] Figure 6 shows a specific example of the functional configuration of the named entity output system 1 of this embodiment. The named entity output system 1 comprises a named entity output device 10, a text supply device 20, and a display device 30.

[0038] The text supply device 20 is, for example, a file server or a hard disk drive, and stores document data D1. The text supply device 20 supplies the document data D1 to the named entity output device 10. The document data D1 includes, for example, technical documents in a specific technical field.

[0039] The named entity output device 10 extracts named entities E in a specific technical field from the document data D1 supplied by the text supply device 20, and outputs the extracted named entities E to the display device 30.

[0040] The named entity output device 10 is, for example, a computer device such as a personal computer. The named entity output device 10 comprises an arithmetic unit 100 and a storage unit 150. The arithmetic unit 100 includes, for example, a central processing unit (CPU), and operates based on programs and data stored in the storage unit 150, providing various functions. The storage unit 150 is composed of, for example, a hard disk drive or semiconductor memory (flash memory, RAM, ROM), and stores various types of information, such as programs and data read by the arithmetic unit 100. The arithmetic unit 100 may be implemented by a virtual storage device such as a cloud server located outside the named entity output device 10.

[0041] The calculation unit 100 includes, as its functional units, a text acquisition unit 101, a named entity extraction unit 102, a named entity verification unit 103, a response rejection unit 104, a named entity correction unit 105, and a named entity output unit 106.

[0042] The text acquisition unit 101 acquires document data D1 supplied from the text supply device 20. That is, the text acquisition unit 101 (text input unit) acquires document data D1. The text acquisition unit 101 outputs the acquired document data D1 to the named entity recognition unit 102.

[0043] The named entity recognition unit 102 extracts named entities E from the acquired document data D1. Named entities E refer to proper nouns or expressions such as specific phrases within the field of the document data D1.

[0044] In other words, the named entity recognition unit 102 extracts named entities E, which are proper nouns or specific phrases in the field of the document data D1, from the acquired document data D1. The named entity recognition unit 102 outputs the extracted named entities E to the named entity verification unit 103.

[0045] The named entity verification unit 103 performs verification, including determining whether the extracted named entity E is correct or incorrect. If the named entity verification unit 103 determines, as a result of the verification, that an error has occurred in named entity E and that it is impossible to correct the error, it outputs named entity E to the response rejection unit 104. If the named entity verification unit 103 determines, as a result of the verification, that an error has occurred in named entity E and that the error can be corrected, it outputs named entity E to the named entity correction unit 105. If the named entity verification unit 103 determines, as a result of the verification, that no errors have occurred in the named entity E, it outputs the named entity E to the named entity output unit 106.

[0046] The response rejection unit 104 rejects the named entity E if it determines during verification that it is impossible to modify the named entity E. Rejecting the named entity E means not outputting the named entity E from the named entity output device 10 (for example, to the display device 30).

[0047] If the named entity correction unit 105 determines during verification that the named entity E can be corrected, it corrects the named entity E.

[0048] The named entity output unit 106 outputs named entities E that have been verified and determined not to have errors, and named entities E that have been verified and whose errors have been corrected, to the display device 30 as named entity data D2. In other words, the named entity output unit 106 outputs named entities E that have been verified and corrected as named entity data D2 to the display device 30.

[0049] [Operation Flow of Named Entity Output System 1] Figure 7 illustrates an example of the operation flow of the named entity output system 1 of this embodiment. (Step S10) The text acquisition unit 101 acquires document data D1 from the text supply device 20. This document data D1 contains various named entities E.

[0050] Figure 8 shows an example of document data D1 in this embodiment. Document data D1 includes, for example, the sentence shown in the figure: "The pigment contains pigment violet 19 and pigment orange 71, wherein compared to the total mass of pigment violet 19 and pigment orange 71, pigment violet 19 is 30.0% mass % ~ 85.0%." The length of the sentence in document data D1 is arbitrary, and the sentence shown in the figure may be the entirety of document data D1 or an excerpt from document data D1.

[0051] The text acquisition unit 101 outputs the acquired document data D1 to the named entity recognition unit 102.

[0052] More specifically, the text acquisition unit 101 acquires a single sentence or phrase, or multiple sentences or phrases, contained in the document data D1, and outputs a sequence of tokens obtained by dividing them into morphemes or subwords.

[0053] (Step S20) Returning to Figure 7, the named entity recognition unit 102 extracts named entity E from the document data D1.

[0054] More specifically, the named entity recognition unit 102 converts each token in the sequence of tokens into an embedding representation, and based on the converted embedding representation, extracts or generates a set of named entities E that correspond to the target label.

[0055] As described above, the named entity recognition unit 102 extracts named entities E using the trained model M. The named entity recognition unit 102 outputs the extracted named entities E as a set of candidate named entities E to the named entity verification unit 103.

[0056] Figure 9 shows an example of a candidate set of named entities E in this embodiment. In this example, the candidate set of named entities E includes (pigment violet 19, component) and (pigment orange 71, component). In the following description, when a named entity E extracted by the named entity extraction unit 102 is described as (x, y), x represents the named entity E (entity), and y represents the label for that named entity E.

[0057] (Step S30) Returning to Figure 7, the named entity verification unit 103 uses the trained model M to verify named entity E included in the candidate set of named entity E extracted by the named entity extraction unit 102.

[0058] More specifically, the named entity verification unit 103 performs correctness determination and error classification for each named entity E included in the set of named entities E extracted or generated by the named entity extraction unit 102.

[0059] Figure 10 shows an example of a candidate set of named entities E when errors are included in this embodiment. In this example, the candidate set of named entities E includes (pigment, component) and (pigment orange 71, feature). Here, (pigment, component) should be extracted as (pigment violet 19, component). Similarly, (pigment orange 71, feature) should be extracted as (pigment orange 71, component). In this example, (pigment, component) is a span error due to insufficient extraction (insufficient span). (pigment orange 71, feature) is a label error.

[0060] The named entity verification unit 103 determines that the correctness judgment result for (pigment, component) is "incorrect" and the error classification result is "span error" (or more specifically, "insufficient extraction (insufficient span)"), and then associates the correctness judgment result and the error classification result with the extracted named entity E.

[0061] Similarly, the named entity verification unit 103 determines that (pigment orange 71, feature) is "incorrect" and the error classification result is "label error," and associates the correct / incorrect result and the error classification result with the extracted named entity E.

[0062] In other words, the named entity verification unit 103 performs correctness determination and error classification for each named entity E included in the set of named entities E extracted or generated by the named entity extraction unit 102.

[0063] (Step S40) Returning to Figure 7, the named entity verification unit 103 outputs the named entity E to the named entity output unit 106 if the result of the correctness determination by the verification of the named entity E is correct (i.e., no uncorrectable errors, span errors, or label errors have occurred), and proceeds to step S80. If the result of the correctness determination by the verification of the named entity E is not correct, the named entity verification unit 103 proceeds to step S50.

[0064] (Step S50) The named entity verification unit 103 determines whether or not it is possible to correct the incorrect named entity E.

[0065] In other words, the named entity verification unit 103 performs correctness determination and error classification for each named entity E included in the set of named entities E extracted or generated by the named entity extraction unit 102. Error classification includes types of errors that can correct the named entity E and types of errors that cannot correct the named entity E.

[0066] If the result of the correct / incorrect judgment is "incorrect" and it cannot be corrected, the named entity verification unit 103 outputs the target named entity E to the response rejection unit 104 and proceeds to step S60. If the named entity verification unit 103 determines that the result of the correct / incorrect judgment is "incorrect" and obtains an error classification result other than "uncorrectable" (i.e., correctable), it outputs the target named entity E and the verification result to the named entity correction unit 105 and proceeds to step S70.

[0067] (Step S60) The response rejection unit 104 excludes the target named entity E from the output candidates.

[0068] In other words, the response rejection unit 104 excludes from the candidates any named entity E included in the set of candidate named entity E extracted or generated by the named entity extraction unit 102 if the result of the correctness judgment by the named entity verification unit 103 is incorrect and the error classification is deemed irreversible.

[0069] (Step S70) The named entity correction unit 105 corrects the named entity E containing the error using the trained model M. Here, the trained model M has learned the correspondence between error classification and the correct representation of named entity E. The named entity correction unit 105 obtains the correct named entity E from the trained model M by providing the trained model M with the named entity E containing the error and the error classification of the error.

[0070] In other words, the named entity correction unit 105 corrects the named entity E based on the named entity E included in the candidate set of named entity E extracted or generated by the named entity extraction unit 102 and the error classification of the named entity E by the named entity verification unit 103.

[0071] The named entity correction unit 105 outputs the corrected named entity E to the named entity output unit 106 and proceeds to step S80. The named entity correction unit 105 may be configured to recursively correct named entity E. Recursively correcting named entity E means that the corrected named entity E, corrected by the named entity correction unit 105, is provided again to the named entity verification unit 103 to verify the named entity E again.

[0072] Figure 11 shows an example of the process for correcting named entity E by the named entity correction unit 105 of this embodiment. The named entity correction unit 105 may also execute steps S71 and S72 shown in the figure in step S70 described above.

[0073] (Step S71) The named entity correction unit 105 corrects the named entity E in the same manner as in step S70 described above. (Step S72) If the named entity correction unit 105 corrects recursively (Step S72; YES), it returns to step S30. If the named entity correction unit 105 does not correct recursively (Step S72; NO), it proceeds to step S50.

[0074] When recursively modifying named entity E, the named entity modification unit 105 terminates the recursive modification and outputs the modified named entity E at that point to the named entity output unit 106 if the following example conditions are met. (1) If the named entity E is determined to be correct in the correctness determination by the named entity verification unit 103. (2) When the number of recursive modifications for the target named entity E reaches the predetermined upper limit. The named entity correction unit 105 terminates the recursive correction and outputs the corrected named entity E at that point to the response rejection unit 104 if the following example conditions are met. (3) If the named entity E is found to be incorrect in the correctness determination by the named entity verification unit 103 and the error classification is deemed irreversible.

[0075] The named entity output system 1 can improve the accuracy of the output named entity E by recursively verifying and correcting the named entity E repeatedly.

[0076] Furthermore, whether or not to perform recursive correction may be a setting item that can be changed by the user of the named entity output system 1. In this case, the user can set whether to use a first operating mode that performs recursive correction or a second operating mode that does not perform recursive correction.

[0077] (Step S80) Returning to Figure 7, the named entity output unit 106 outputs the named entity E extracted by the named entity extraction unit 102 to the display device 30. The named entity E output by the named entity output unit 106 includes the correct named entity E output by the named entity extraction unit 102 and the named entity E corrected by the named entity correction unit 105.

[0078] In other words, the named entity output unit 106 outputs named entity E that are included in the set of candidate named entity E extracted or generated by the named entity extraction unit 102 and that have been determined to be correct by the named entity verification unit 103, as well as named entity E that have been corrected by the named entity correction unit 105 based on the error classification output by the named entity correction unit 105.

[0079] (Step S90) The arithmetic unit 100 searches the set of candidate named entities E output by the named entity extraction unit 102 in step S20 and determines whether there are candidates for the next named entity E. If there are candidates for the next named entity E (Step S90; YES), the process returns to step S30. If there are no candidates for the next named entity E (Step S90; NO), the series of processes ends.

[0080] [About the types of pre-trained models M] Here, we will explain the types of trained models M used by the named entity extraction unit 102, the named entity verification unit 103, and the named entity correction unit 105 described above. In the named entity output system 1, the trained model M can be one of the following two types.

[0081] (1) Individually trained models M In the named entity recognition unit 102, the named entity verification unit 103, and the named entity correction unit 105, language models with individual parameters may be used. For example, the named entity recognition unit 102 uses the first trained model M1 from the trained model M. The named entity verification unit 103 uses the second trained model M2 from the trained model M. The named entity correction unit 105 uses the third trained model M3 from the trained model M.

[0082] The first learning model M1 is a pre-trained model M that is trained separately from the second learning model M2 used by the named entity verification unit 103. The second learning model M2 is trained using the comparison results obtained by comparing the extraction or generation results of named entities E by the named entity recognition unit 102 using the first learning model M1 with the correct data of named entities E as training data.

[0083] In other words, the named entity extraction unit 102 may extract or generate named entities E using the first learning model M1, which is learned separately from the second learning model M2 used by the named entity verification unit 103.

[0084] Furthermore, the third learning model M3 is a trained model M that has been trained based on the extraction or generation results of named entities E by the named entity recognition unit 102 and the correct data of named entities E, which are the training data.

[0085] In other words, the named entity correction unit 105 corrects the named entity E using the extraction or generation result of the named entity E by the named entity extraction unit 102 and the third learning model M3 which has been trained based on the correct data of named entity E, which is the training data. Note that the third learning model M3 may be a learning model that was trained separately from the first learning model M1.

[0086] According to the named entity output system 1 that utilizes the trained model M configured in this way, if there is an error in the named entity E output by the named entity output device 10, the named entity output device 10 can self-correct by using a combination of multiple types of trained models M. Therefore, the named entity output system 1 can improve the accuracy of the output named entity E.

[0087] (2) Integrated training model M In the named entity recognition unit 102, the named entity verification unit 103, and the named entity modification unit 105, an integrated language model with common parameters may be used. The integrated trained model M refers to a trained model M in which a language model with common parameters has been further trained according to the functions of the named entity recognition unit 102 (extraction or generation of named entities E), the named entity verification unit 103 (verification of named entities E), and the named entity modification unit 105 (modification of named entities E).

[0088] In other words, the named entity extraction unit 102, the named entity verification unit 103, and the named entity correction unit 105 are implemented using a pre-trained model M, which is created by further training a language model with common parameters according to each function.

[0089] By utilizing the integrated, trained model M, the named entity output system 1 can autonomously determine the presence or absence of errors and classify errors based on the extraction results of named entities E within the integrated language model, thereby enabling more accurate correction of named entities E.

[0090] As explained above, the named entity output system 1 allows the named entity output device 10 to self-correct if there is an error in the named entity E output by the device 10. Therefore, the named entity output system 1 can improve the accuracy of the output named entity E.

[0091] [Differentiation] Figure 12 shows an overview of the functional configuration of a modified named entity output system 1a. This modified named entity output system 1a differs from the named entity output system 1 described above in that, when the correct / incorrect judgment result of the named entity verification unit 103 is "incorrect" (i.e., when an error is determined), it can output the basis for the error determination made by the named entity verification unit 103.

[0092] In other words, the named entity output system 1a of this modified version differs from the named entity output system 1 described above in that, in addition to the LLM_SV and LLM_SR described above, the language model (LLM_SC) that self-corrects the output result includes a language model (LLM_SF) that feeds back named entities.

[0093] Figure 13 shows an example of the functional configuration of a modified named entity output system 1a. This modified named entity output system 1a includes a named entity output device 10a. The named entity output device 10a includes a calculation unit 100a instead of the calculation unit 100 described above. The calculation unit 100a further includes a named entity feedback unit 107.

[0094] The named entity feedback unit 107 uses a learned model (LLM_SF shown in Figure 12) that has learned the correspondence between named entity E containing errors and the error classification of named entity E to generate information indicating the basis for the named entity E that the named entity verification unit 103 has determined to be an error. The named entity feedback unit 107 outputs the generated information indicating the basis to the named entity correction unit 105. In other words, the named entity feedback unit 107 feeds back to the named entity correction unit 105 the basis for the error determination of named entity E.

[0095] The named entity feedback unit 107 may have a function to output the basis for the determination of named entity E output by the named entity verification unit 103 in natural language. For example, the named entity feedback unit 107 may have a natural language processing function using artificial intelligence (AI). The named entity feedback unit 107 may also have a prompt function to query the artificial intelligence function located outside the named entity output device 10 for the basis of the error in named entity E in natural language.

[0096] Furthermore, the named entity feedback unit 107 may be configured as a function that the named entity verification unit 103 described above has internally.

[0097] According to the named entity output system 1a of this modified version, which includes a named entity feedback unit 107, the named entity correction unit 105 can correct the named entity E by referring to information on the basis for which the named entity verification unit 103 determined the named entity E to be an error. Therefore, according to the named entity output system 1a of this modified version, the accuracy of correcting the named entity E can be improved.

[0098] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design modifications and the like are also included within the scope of the gist of the present invention. For example, a computer program to implement the functions of each of the above-mentioned devices may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. The term "computer system" here may include hardware such as an operating system and peripheral devices.

[0099] Furthermore, "computer-readable recording media" refers to writable non-volatile memory such as flexible disks, magneto-optical disks, ROMs, and flash memory, portable media such as DVDs (Digital Versatile Discs), and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes volatile memory (such as DRAM (Dynamic Random Access Memory)) within computer systems that act as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines, which retain programs for a certain period of time.

[0100] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit the program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the above program may be intended to implement some of the functions described above. It may also be a so-called differential file (differential program) that can implement the aforementioned functions in combination with programs already recorded in the computer system. [Explanation of Symbols]

[0101] 1...Named entity output system, 10...Named entity output device, 20...Text supply device, 30...Display device, 101...Text acquisition unit, 102...Named entity extraction unit, 103...Named entity verification unit, 104...Response rejection unit, 105...Named entity correction unit, 106...Named entity output unit, D1...Document data, D2...Named entity data, M...Trained model

Claims

1. A text input section for acquiring document data, A named entity extraction unit extracts named entities, which are proper nouns or specific phrases in the field of the document data, from the acquired document data. A named entity verification unit that performs verification including determining whether the extracted named entity is correct or incorrect, If, in the verification described above, it is determined that the named entity can be modified, a named entity modification unit is provided to modify the named entity, In the verification described above, if it is determined that the named entity cannot be modified, the response rejection unit rejects the named entity, A named entity output unit that outputs the named entity after the verification and correction described above, A named entity output device equipped with this feature.

2. The text input unit retrieves a single sentence or phrase, or multiple sentences or phrases, contained in the document data, and outputs a sequence of tokens obtained by dividing them into morphemes or subwords. The named entity output device according to claim 1.

3. The named entity recognition unit converts each token included in the sequence of tokens into an embedding representation, and extracts or generates a set of named entities corresponding to the target label based on the converted embedding representation. The named entity output device according to claim 2.

4. The named entity extraction unit extracts or generates a set of named entities that correspond to the label to be extracted. The named entity verification unit performs correctness determination and error classification for each named entity included in the set of named entities extracted or generated by the named entity extraction unit. If the result of the correct / incorrect judgment is correct, the target named entity is output to the named entity output unit. If the result of the correct / incorrect judgment is incorrect and cannot be corrected, the named entity in question is output to the response rejection unit. If the result of the correct / incorrect judgment is an error classification result other than an error that is incorrect and cannot be corrected, the named entity and the verification result are output to the named entity correction unit. The named entity output device according to claim 1.

5. The named entity verification unit performs correctness determination and error classification for each named entity included in the set of named entities extracted or generated by the named entity extraction unit. The named entity correction unit corrects each named entity included in the candidate set of named entities extracted or generated by the named entity extraction unit, based on the named entity and the error classification of the named entity by the named entity verification unit, according to the named entity output device according to claim 1.

6. The named entity extraction unit, the named entity verification unit, and the named entity correction unit are implemented using a trained model obtained by further training a language model with common parameters according to each function. The named entity output device according to claim 1.

7. The named entity extraction unit extracts or generates named entities using a first learning model that has been learned separately from the second learning model used by the named entity verification unit. The second learning model is trained using the comparison results obtained by comparing the extraction or generation results of named entities by the named entity recognition unit using the first learning model with the correct named entity data as training data. The named entity output device according to claim 1.

8. The named entity correction unit corrects the named entity using the named entity extraction or generation results from the named entity extraction unit and a third learning model that has been trained based on the correct named entity data, which is the training data. The named entity output device according to claim 1.

9. The response rejection unit excludes from the candidates any named entity among the named entity candidates extracted or generated by the named entity extraction unit if the result of the correctness determination by the named entity verification unit is incorrect and the error classification is irreversible. The named entity output device according to claim 1.

10. The named entity output device according to claim 1, wherein the named entity output unit outputs named entities that are included in the set of candidate named entities extracted or generated by the named entity extraction unit and that have been determined to be correct by the named entity verification unit, and named entities that have been corrected by the named entity correction unit based on the error classification output by the named entity correction unit.

11. The named entity verification unit performs correctness determination and error classification for each named entity included in the set of named entities extracted or generated by the named entity extraction unit. The aforementioned error classification includes types of errors that can correct named entities and types of errors that cannot correct named entities. The named entity output device according to claim 1.

12. The types of errors that can correct the named entities include over-extraction of entities, under-extraction of entities, mislabeling, and combinations of over-extraction or under-extraction and mislabeling. The named entity output device according to claim 11.

13. The named entity correction unit outputs the corrected named entity to the named entity verification unit to determine whether the correction result is correct and classify any errors. If the correctness determination is found to be correct, the output is sent to the named entity output unit. If the correct / incorrect judgment is incorrect and the error classification is deemed uncorrectable, the output is sent to the response rejection unit. If the correct / incorrect judgment is incorrect and the error classification is correctable, the output is sent to the named entity correction unit to correct the named entity again. If the named entity to be corrected is determined to be correct by the named entity verification unit, or if the correct / incorrect judgment by the named entity verification unit is incorrect and the error classification is uncorrectable, or if the number of recursive corrections reaches a predetermined upper limit, the output result at that point is sent to the named entity output unit. The named entity output device according to claim 1.

14. The computer equipped with the named entity output device, Acquiring document data and Extracting proper nouns or specific phrases within the field of the document data from the acquired document data, The process involves verifying the extracted named entities, including determining whether they are correct or incorrect. If, in the verification described above, it is determined that the named entity can be modified, the named entity shall be modified. If, in the verification described above, it is determined that the named entity cannot be modified, the named entity shall be rejected. Outputting the named entity after the verification and correction described above, A program to execute.

Images