Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus, a method and a program that suppress accuracy deterioration of token division.SOLUTION: In a unique expression extraction device, a control section extracts a character string from document data, decomposes the extracted character string into token units, and compares between a token number and an input upper limit of a natural language processing model being a unique expression extractor, and a division section divides it into two or more token groups when the token number which is obtained by decomposing the extracted character string exceeds the input upper limit. The two or more token groups are divided to partially overlap the tokens included in the document data on the basis of the limited token number. The limited token number defines a minimum token number in which extraction of a unique expression defined in a given GT is rightly solved, and is a maximum number in the minimum token numbers for each identified unique expression. And, an extraction section performs unique expression extraction for each divided token group, and the control section determines the unique expression extracted in an overlapping portion, and finishes processing.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to information processing technology for natural language processing. [Background technology]

[0002] In recent years, advances in AI technology have led to advances in the field of natural language processing, which uses computers to analyze documents written in human speech and language. Natural language processing technology is expected to be applied in a variety of fields that handle document data, such as document summarization and translation, voice dialogue, and data analysis.

[0003] One application of natural language processing is named entity extraction, which extracts the values ​​of predefined items (named entities) from document data. For example, if a corporate name and expiration date are defined as named entities, the system extracts the character strings corresponding to the corporate name and expiration date from the document data.

[0004] Currently, natural language processing models, such as BERT, which uses Transformers and is the mainstream in natural language processing, break down character strings contained in document data into units called tokens, and use the vectorized tokens as input data. However, because there is a limit to the number of tokens that a natural language processing model can process at one time, when inputting long document data containing tokens that exceed the limit, the document data must be divided into two or more token groups, which must be input and processed separately. If the multiple tokens contained in a single document are simply divided to fit the input limit of the natural language processing model, keywords and context (character strings surrounding the named entity) that are used to distinguish named entities may be lost, resulting in a decrease in the accuracy of named entity identification.

[0005] In Patent Document 1, document data is divided into sections such as chapters, sections, and paragraphs, and named entities are extracted using a natural language processing model for each token group that is expected to retain a certain context. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-64143 Summary of the Invention [Problem to be solved by the invention]

[0007] In Cited Document 1, tokens contained in document data are divided into sections so that the number of tokens processed at one time falls within the upper input limit of the natural language processing model. However, because each divided section is processed separately, there is an issue that the estimation accuracy may decrease due to the loss of context contained in adjacent sections.

[0008] Therefore, an object of the present invention is to prevent a decrease in estimation accuracy in natural language processing when dividing tokens contained in document data. [Means for solving the problem]

[0009] The technology disclosed herein is an information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, the information processing device comprising: a division means for dividing the plurality of tokens obtained by decomposing the input character string into two or more token groups when the number of the plurality of tokens obtained by decomposing the input character string exceeds a predetermined upper limit, wherein a predetermined number of tokens in each of the token groups overlap with other token groups; an extraction means for extracting the named entities for each of the token groups; and a determination means for determining an extraction result of the named entity in the overlapping portion based on an extraction result of the named entity by the extraction means for the overlapping portion with the other token groups. and the determining means determines, among the extraction results of the named entities from each of the two overlapping token groups in the overlapping portion, the extraction result of the named entity from the token group with the larger number of tokens in the overlapping token groups as the extraction result of the named entity in the overlapping portion. It is characterized by: [Effects of the Invention]

[0010] According to the present invention, it is possible to suppress a decrease in estimation accuracy in natural language processing when dividing tokens included in document data. [Brief explanation of the drawings]

[0011] [Figure 1] A block diagram showing an example of the functional and hardware configuration of a named entity extraction device 100. [Figure 2] FIG. 10 is a diagram illustrating an example of document data received by a receiving unit 102. [Figure 3] 1 is a table showing an example of a token acquired by the control unit 101. [Figure 4] Table showing an example of named entities and limit token numbers calculated by the calculation unit 103 [Figure 5] 1 is a flowchart illustrating an example of processing executed by the control unit 101. [Figure 6] 10 is a flowchart illustrating an example of processing executed by the control unit 101 according to the second embodiment. [Figure 7] 10 is a table showing an example of the limit number of tokens calculated by the calculation unit 103 in the third embodiment. [Figure 8] 10 is a flowchart illustrating an example of processing executed by the control unit 101 according to the third embodiment. [Figure 9] 10 is a flowchart illustrating an example of processing executed by the control unit 101 according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0013] [Embodiment 1] In the first embodiment, an example of a system will be described in which character strings are extracted from document data, the extracted character strings are broken down into token units, and named entities are extracted and output from the obtained tokens using a natural language processing model. If the obtained tokens can be input to a natural language processing model at once to extract named entities, the results can be estimated efficiently. However, natural language processing models have an input upper limit, meaning that there is an upper limit on the number of tokens that can be input at one time. Therefore, if a long sentence has a number of tokens that exceeds the input upper limit, it is necessary to divide the token groups into multiple groups and input them into the natural language processing model. However, if the token groups are simply divided to fit the input upper limit of the natural language processing model, keywords and contexts for distinguishing named entities may be lost, which may result in a decrease in the accuracy of named entity extraction.

[0014] Therefore, in the first embodiment, when the number of tokens exceeds the upper limit of input to the natural language processing model, the tokens are first divided into two or more partially overlapping token groups based on the limit number of tokens from which named entities can be correctly extracted. Named entities are then extracted from each divided token group, and the named entities to be extracted from the overlapping portions are determined, thereby preventing a decrease in the accuracy of named entity extraction.

[0015] 1(a) shows a functional block diagram of an example of a named entity extraction device 100 according to this embodiment. The named entity extraction device 100 is an information processing device including a control unit 101, a receiving unit 102, a calculation unit 103, a division unit 104, and an extraction unit 105.

[0016] The control unit 101 is composed of a CPU 111 and the like, and reads out programs and data stored in a ROM 113 into a RAM 112 to execute processes such as named entity extraction.

[0017] The receiving unit 102 receives document data to be subjected to named entity extraction processing through operation of the input device 114 provided in the named entity extraction device 100. The document data may be acquired from the storage device 116, or may be acquired on the network 118 via the network interface 117.

[0018] The calculation unit 103 calculates the limit number of tokens that allows the control unit 101 to correctly extract named entities from document data.

[0019] The dividing unit 104 divides the token group converted from the document data into two or more token groups so that the number of tokens input at one time is less than the upper limit number of tokens that can be input to the natural language processing model.

[0020] The extraction unit 105 extracts named entities from the divided token groups using a natural language processing model. The extracted named entities are stored in the storage device 116 and displayed on the output device 115 such as a display.

[0021] 2(a) shows an example of document data received by the receiving unit 102. The document data is composed of multiple elements of different types, including character strings, symbols, and ruled lines, within a page. Document data used for training and evaluation of natural language processing models is usually provided with ground truth data called GT (Ground Truth). The GT provided to document data 210 defines named entity types (or attributes) for each of the character strings 211, 212, and 213 within the dashed rectangles, such as corporate name, corporate name, and expiration date, to indicate that the character strings are named entities.

[0022] Document data 210 is usually composed of pages and includes symbols and ruled lines, but it may also be composed of multiple pages, or may be data that contains only character information without layout information. In other words, document data may be data in any format as long as character string information can be obtained.

[0023] Fig. 2(b) shows an example of document data received by the receiving unit 102. This document data is similar to Fig. 2(a) and has a similar layout, but differs only in that no GT (correct answer data) is added.

[0024] 3(a) and 3(b) show an example of a token acquired by the control unit 101. The token is represented by an identifier 311, a token string 312, and a GT 313.

[0025] 3(a) shows a token table 310 listing 261 tokens obtained by decomposing character strings extracted from document data 210 to which correct answer data has been assigned into token units through morphological analysis. Each token is assigned a GT in IOB (Inside-Outside-Beginning) format. In this embodiment, three types of named entities are used: corporate name (ORG), personal name (PERSON), and expiration date (DATE), but other types may be defined and used without being limited to these three.

[0026] Since a named entity may be composed of multiple tokens, in such cases, "B-" is added to the GT of the first token of the named entity, and "I-" is added to the GT of the subsequent token. For example, "ABC Co., Ltd." in the character string 211 shown in FIG. 2(a) is a named entity of a corporate name (ORG), but is composed of two tokens, T1_003 ("ABC") and T1_004 ("Co., Ltd."). Therefore, the GT of token T1_003 is assigned "B-ORG" indicating the first token, and the GT of token T1_004 is assigned "I-ORG" indicating the subsequent token. The GT of a token to which no named entity type is assigned is assigned "O" indicating that it is not a named entity. In this embodiment, a GT is assigned to each token in the IOB format as described above, but other methods may be used to assign GTs to named entities that span multiple tokens.

[0027] Fig. 3(b) is a token table 320 listing 264 tokens obtained by decomposing the input character string extracted from the document data shown in Fig. 2(b) into token units through morphological analysis by the control unit 101. Since the table 320 is based on the document data 220 to which no GT is assigned, each token has an identifier 321 and a token character string 322, but no GT.

[0028] 4(a) is a table illustrating an example of a named entity acquired by the calculation unit 103. A named entity is composed of a named entity identifier 411, a character string 412, and a type 413, and the minimum number of tokens 414 indicates the minimum number of tokens surrounding the named entity required to correctly extract the named entity. The minimum number of tokens 414 is calculated by the calculation unit 103 based on multiple tokens to which GT is assigned. The named entity NE_001 corresponds to the character string 211 in the document data 210, and has the character string "ABC Co., Ltd." and the type "Corporate Name (ORG)." Similarly, the named entity NE_002 corresponds to the character string 212 in the document data 210, and has the character string "DEF Co., Ltd." and the type "Corporate Name (ORG)." The named entity NE_003 corresponds to the character string 213 in the document data 210, and has the character string "October 31st" and the type "Expiration Date (DATE)." Named entities NE_004 to NE_006 are named entities defined in different document data, and the types of named entities are set as a person's name, a corporate name, and an expiration date, respectively.

[0029] 4(b) shows the limit number of tokens derived by the calculation unit 103 in the example shown in FIG. 4(a). The limit number of tokens is the largest value among the minimum number of tokens 414 corresponding to all named entities registered in the table 410.

[0030] 5(a) is a flowchart showing an example of a limit token number derivation process executed by the control unit 101 in this embodiment. This flowchart is executed at the time of initialization in the named entity extraction device 100. This process may be configured to acquire a result calculated by a device different from the named entity extraction device 100, which will be described later.

[0031] In S511, the control unit 101 acquires the document data 210 to which the GT has been added from the receiving unit .

[0032] In S512, the control unit 101 extracts an input character string from the acquired document data, and decomposes the extracted input character string into token units by morphological analysis.

[0033] In S513, the control unit 101, using the calculation unit 103, identifies the minimum number of tokens required to correctly extract a named entity defined in the assigned GT. In this step, first, named entities are acquired and stored based on the GT assigned to the tokens of each document data. For the document data 210 to which a GT is assigned, "ABC Co., Ltd.", "DEF Vacuum Co., Ltd.", and "October 31st" are stored as named entities in the table 410. At this time, an identifier 411 is assigned to each acquired named entity, and a named entity character string 412 and a named entity type 413 are associated and stored. Next, named entity extraction is performed on the document data, and the minimum number of tokens required to correctly extract the named entity is identified for the extracted named entity. Specifically, the initial value of the number of tokens before and after the named entity is set to 128, and named entity extraction is performed using that number of surrounding tokens. If the named entity extraction is correct, the number of surrounding tokens is reduced by 1, and named entity extraction is performed again. Named entity extraction is continued while reducing the surrounding tokens until extraction is impossible or an incorrect result is obtained. The identified minimum number of tokens that were successfully extracted is stored in the minimum number of tokens 414 in the table 410. The initial value of the number of neighboring tokens when identifying the minimum number of tokens may be a fixed value, or may be determined based on the number of tokens included in the document data or the results of other document data. Methods for reducing the number of tokens when identifying the minimum number of tokens include reducing the number of tokens by one, searching using a binary search, or any other method that can determine the minimum number of tokens.

[0034] In S514, the calculation unit 103 stores the maximum value among the minimum numbers of tokens for each identified named entity as the limit number of tokens in the table 420. Since the maximum value of the minimum number of tokens in the table 410 is "7", "7" is stored as the limit number of tokens.

[0035] Although the maximum value of the minimum number of tokens is used as the limit here, the smallest number of tokens that is equal to or greater than the minimum number of tokens for a given proportion of named entities may also be used as the limit. For example, if the limit is the number of tokens that are correct for 80% of named entities, then the limit would be "6," which is the minimum number of tokens (414) that is correct for 80% of entities.

[0036] 5(b) is a flowchart showing an example of processing executed in the control unit 101 in this embodiment. This flowchart is executed in response to an instruction to extract named entities from document data in the named entity extraction device 100.

[0037] In S521, the control unit 101 extracts an input character string from the document data, decomposes the extracted input character string into token units, and proceeds to S522. In this step, an input character string is extracted from the document data 220 to which no correct answer data has been assigned, and the extracted input character string is decomposed into token units.

[0038] In S522, the control unit 101 compares the number of tokens with the upper limit of the number of inputs of the natural language processing model, which is a named entity extractor, and if the number of tokens obtained by decomposing the extracted input character string exceeds the upper limit of the inputs, the process proceeds to S523, and if it does not exceed the upper limit of the inputs, the process proceeds to S524. When the upper limit of the inputs of the natural language processing model is 256, the number of tokens stored in the table 410 is 264, which exceeds the upper limit of the inputs, and the process proceeds to S523.

[0039] In S523, the dividing unit 104 divides the tokens included in the document data into two or more token groups with partial overlap based on the token limit identified in S514, and then proceeds to S524. In this step, the token limit stored in table 420 is first obtained. This value may be a preset value. One token group is divided into two or more token groups with overlapping tokens, which is 14 tokens, double the token limit of 7. The reason for doubling the overlapping token limit here is to ensure seven tokens as tokens surrounding named entities. A value obtained by adding the average number of tokens for named entities to this value may also be used. In the example shown in table 320, the document data is divided into token group 1 (T2_001 to T2_256) and token group 2 (T2_243 to T2_264), so that the 14 tokens from T2_243 to T2_256 overlap. When dividing, the number "14" calculated from the limit token number may be set as the lower limit of the number of overlapping tokens, and the division may be performed so as to maximize the number of tokens without increasing the number of divisions. For example, the token group in table 320 may be divided into token group 1 from T2_001 to T2_256, and token group 2 from T2_009 to T2_264. In either case, it is sufficient that the token groups are divided so that the number of tokens after division is equal to or greater than the lower limit calculated from the limit token number.

[0040] In this embodiment, the limit number of tokens is handled as a single numerical value, but it may be calculated separately as the number of tokens before the named entity and the number of tokens after it, and the sum of these may be used as the overlap number.

[0041] In S524, the extraction unit 105 executes named entity extraction for each divided token group. In this step, named entity extraction is executed for each of token group 1 and token group 2, and named entities are obtained. Assume that "GHI Co., Ltd." and "JKL Transport Company" are extracted as the corporate name (ORG) from token group 1, and "March 5th" is extracted as the expiration date (DATE). Assume also that "March 5th" is extracted as the expiration date (DATE) from token group 2. Assume that "March 5th" is extracted from the overlapping parts of token group 1 and token group 2, respectively.

[0042] In S525, the control unit 101 determines the named entity extracted in the overlapping portion and ends the process. Specifically, if the same result is extracted for a common token in the overlapping portion, only one of the results is output; if different results are extracted, the result with the larger number of surrounding tokens is output. Even if one of the results is not detected in the overlapping portion, the result with the larger number of tokens is given priority. In the example shown in S524, "March 5th" is extracted as the expiration date (DATE) in the overlapping portions of both token group 1 and token group 2, so only one result is output. Therefore, the extraction results for table 410 are "GHI Co., Ltd." and "JKL Transport Company" as the corporate names (ORG) and "March 5th" as the expiration date (DATE).

[0043] As described above, when extracting named entities from document data with a large number of characters, dividing the data into two or more overlapping token groups prevents the loss of keyword strings and context surrounding the named entities, thereby minimizing any decline in the accuracy of named entity extraction.

[0044] [Embodiment 2] In the first embodiment, when specifying the minimum number of tokens for each named entity, the number of tokens is gradually reduced to obtain the minimum number of tokens that allows for a correct answer. In contrast, in the present embodiment, the limit number of tokens is obtained based on the degree of association with tokens that appear in the network of the natural language processing model.

[0045] 6 is a flowchart showing an example of processing executed by the control unit 101 in this embodiment. Note that steps S511, S512, and S514 in the flowchart are the same as the steps with the same reference numerals in FIG. 5(a), and therefore their explanation will be omitted here.

[0046] In S611, the calculation unit 103 calculates the minimum number of tokens that will result in correct named entity extraction for each named entity defined in the GT, based on the degree of association with tokens that appear in the network of the natural language processing model. In this step, first, named entities are obtained based on the GT assigned to the tokens of each document data, and stored in table 410 shown in Fig. 4. Table 410 stores named entity identifiers 411, named entity character strings 412, and named entity types 413. Next, for each named entity, the minimum number of tokens 414 that will result in correct named entity extraction is calculated.

[0047] Specifically, first, the tokens contained in the document data are divided into two or more token groups so that the number of tokens before and after the named entity is maximized. Then, the resulting token groups are subjected to extraction of named entities using a natural language processing model with a self-attention mechanism, such as BERT, which uses Transformers. If the named entity is correctly extracted, the relationship between the named entity token and its surrounding tokens is measured by the attention strength that appears in the network of the natural language processing model. The minimum number of surrounding tokens whose attention strength is equal to or greater than a predetermined threshold is determined to be the minimum number of tokens corresponding to the named entity. The calculated minimum number of tokens is stored as the minimum number of tokens 414 in table 410.

[0048] As described above, when extracting named entities from document data with a large number of characters, dividing the data into two or more overlapping token groups prevents the loss of keyword strings and context surrounding the named entities, thereby minimizing a decrease in the accuracy of named entity extraction.Furthermore, the minimum number of tokens can be calculated more easily by calculating it based on information from the network of the natural language processing model.

[0049] [Embodiment 3] In the first embodiment, the shared limit number of tokens is calculated for each named entity. In contrast, in the present embodiment, the limit number of tokens is calculated for each type of named entity.

[0050] An example of the limit number of tokens determined by the calculation unit 103 will be described with reference to table 700 in Fig. 7. The limit number of tokens is composed of a named entity type 701 and a limit number of tokens 702. A value calculated for each named entity type based on the named entity table 410 is entered in 702. In table 700, the limit numbers of tokens for corporate name (ORG), personal name (PERSON), and expiration date (DATE) are defined in 703, 704, and 705, respectively.

[0051] 8A is a flowchart showing an example of processing executed by the control unit 101 in this embodiment. Note that steps S511, S512, and S513 in the flowchart are the same as the steps with the same reference numerals in FIG. 5A, and therefore their explanation will be omitted.

[0052] In S811, the calculation unit 103 determines the limit number of tokens for each type of named entity and ends the process. The value stored in the minimum number of tokens 414 in the table 410 of named entities is obtained, the maximum value is found for each type of named entity, and the limit number of tokens for each type of named entity is saved in the table 700. Although the maximum value is used here, the number of tokens that exceeds the minimum number of tokens for a predetermined percentage of named entities may also be used as the limit number of tokens.

[0053] 8(b) is a flowchart showing an example of processing executed by the control unit 101 in this embodiment. Note that S521, S522, S524, and S525 in the flowchart are the same as the steps with the same names in FIG. 5(b), and therefore descriptions thereof will be omitted.

[0054] In S821, the receiving unit 102 accepts the type (or attribute) of the named entity to be extracted, and proceeds to S521. Here, it is assumed that the user of the named entity extraction device has accepted the extraction of named entities of the type of personal names included in the document data.

[0055] In S822, the dividing unit 104 divides the tokens included in the document data into two or more partially overlapping token groups based on the limit number of tokens corresponding to the type of named entity accepted in S821, and then proceeds to S524. In this step, the limit number of tokens stored in table 700 is obtained based on the type accepted in S821. If multiple types are accepted, the maximum value is used. In this case, since a person's name is accepted in S821, the tokens included in the document data are divided into two or more partially overlapping token groups based on the corresponding limit number of tokens, "2."

[0056] By limiting the maximum number of tokens to the minimum number of tokens corresponding to the type of named entity accepted as the extraction target in this way, it is possible to reduce the number of token group divisions and the number of tokens to be processed in named entity extraction.

[0057] As described above, when extracting named entities from document data with a large number of characters, dividing the data into two or more overlapping token groups prevents the loss of keyword strings and context surrounding the named entity, and suppresses a decrease in the accuracy of named entity extraction. Furthermore, by setting the minimum number of tokens corresponding to the type of named entity to be extracted as the token limit, the number of times the document data is divided can be reduced, thereby reducing the amount of calculation.

[0058] [Embodiment 4] In the first embodiment, all tokens are divided into two or more token groups using a predetermined limit number of tokens. In contrast, in the present embodiment, an example will be described in which the limit number of tokens is updated based on the estimated usage status of named entities.

[0059] 9 is a flowchart showing an example of processing executed by the control unit 101 in this embodiment. This flowchart is executed after named entity extraction is executed on document data in a named entity extraction device and the extracted named entity is used by a user of the device. Note that steps S512 and S513 in the flowchart are the same as the steps with the same reference numerals in FIG. 5(a), and therefore their explanation will be omitted.

[0060] In S911, document data to which a GT has been assigned, which additionally defines the token selected by the device user as a named entity, is acquired, and the process proceeds to S512. Named entities are extracted from the document data, and when the extracted corporate name (ORG) "GHI Co., Ltd." is selected by the user, a GT is assigned to the corresponding token in table 320, which additionally defines the token as a named entity. Specifically, "B_ORG" and "I_ORG" are assigned as GTs to tokens T2_003 and T2_004, respectively, and "O" is assigned as GT to the other tokens, and the resulting document data is acquired.

[0061] In S912, the calculation unit 103 updates the token limit number and terminates the process. Specifically, the results of the processes in S512 and S513 on the document data acquired in S911 are reflected, and the token limit number table 420 is updated based on the named entity table 410 to which the minimum number of tokens corresponding to the named entity selected by the user has been added.

[0062] As described above, when extracting named entities from document data with a large number of characters, dividing the data into two or more overlapping token groups prevents the loss of keyword strings and context surrounding the named entities, thereby minimizing the deterioration of named entity extraction accuracy. Furthermore, by updating the token limit, it is possible to prevent the deterioration of accuracy even for unknown documents.

[0063] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0064] 100 Named entity extraction device 101 Control section 102 Receiving unit 103 Calculation Unit 104 Division 105 Extraction part

Claims

1. An information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a dividing means for dividing the plurality of tokens obtained by decomposing the input character string into two or more token groups when the number of the plurality of tokens obtained by decomposing the input character string exceeds a predetermined upper limit, wherein each of the token groups has a predetermined number of tokens overlapping with other token groups; extraction means for extracting the named entities for each of the token groups; a determination means for determining an extraction result of the named entity in the overlapping portion based on an extraction result of the named entity by the extraction means for the overlapping portion with the other token group; Equipped with the determining means determines, among the extraction results of the named entities from each of the two overlapping token groups in the overlapping portion, the extraction result of the named entity from the token group with the greater number of tokens in the overlapping token groups as the extraction result of the named entity in the overlapping portion; 1. An information processing device comprising:

2. The predetermined upper limit number is the number of tokens that the extraction means can process at one time.

2. The information processing apparatus according to claim 1, wherein:

3. the predetermined number is set based on a limit number of tokens required for the extraction means to extract the named entity; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the limit number of tokens is the smallest number of tokens among the numbers of tokens input when the extraction means correctly extracts a named entity from an input character string to which correct answer data defining the named entity has been added; 4. The information processing apparatus according to claim 3,

5. the limit number of tokens is the smallest number of tokens among the number of tokens whose relevance to tokens corresponding to named entities defined in the correct answer data assigned to the input character string is equal to or greater than a predetermined value in the natural language processing model used by the extraction means; 4. The information processing apparatus according to claim 3,

6. When the correct answer data defines a plurality of named entities, the limit number of tokens is the largest of the plurality of minimum numbers of tokens corresponding to the plurality of named entities.

6. The information processing apparatus according to claim 4, wherein:

7. When the correct answer data defines a plurality of named entities, the limit number of tokens is the smallest number of tokens among the numbers of tokens input when a predetermined percentage of the plurality of named entities are correctly extracted.

6. The information processing apparatus according to claim 4, wherein:

8. further comprising a receiving means for receiving a type of named entity extracted by the extracting means, The limit number of tokens is the smallest number of tokens input when the extraction of the named entity of the type accepted by the accepting means is correct.

4. The information processing apparatus according to claim 3,

9. the correct answer data is obtained by additionally defining a named entity selected by a user from among the named entities extracted from the input character string by the extraction means.

8. The information processing apparatus according to claim 4, wherein the information processing apparatus is a computer.

10. An information processing method for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a step of dividing the plurality of tokens into two or more token groups when the number of tokens obtained by the information processing device decomposing the input character string exceeds a predetermined upper limit, wherein each of the token groups has a predetermined number of tokens overlapping with other token groups; a step of extracting the named entity for each of the token groups by the information processing device; a step of determining an extraction result of the named entity in the overlapping portion based on an extraction result of the named entity in the extraction step for the overlapping portion with the other token group by the information processing device; and the determining step determines, among the extraction results of the named entities from each of the two overlapping token groups in the overlapping portion, the extraction result of the named entity from a token group having a larger number of tokens in the overlapping token groups as the extraction result of the named entity in the overlapping portion; An information processing method comprising:

11. A program for causing a computer to function as the information processing device according to any one of claims 1 to 9.

Images