Information processing device, method for controlling information processing device, and program

Abstract

To solve the problem in a case of simply dividing tokens of a long document in accordance with the upper limit of input of a natural language processing model, in which there is a possibility that the extraction accuracy of named entities may decrease due to the loss of keywords or context used to distinguish the named entities.SOLUTION: When the number of tokens included in a token group obtained by decomposing an input character string exceeds a predetermined upper limit, a control unit 101 of a named entities extraction device 100 replaces a plurality of specific tokens included in the token group with a single token (S403), and extracts named entities from the token group (S404). When the extracted named entities are those compressed by the compression processing, the control unit restores the compressed named entities unique to their states prior to compression (S406).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device for natural language processing, a control method for the information processing device, and a program. [Background technology]

[0002] In recent years, advances in AI technology have led to advances in the field of natural language processing, which uses computers to analyze documents written in human speech and language. Natural language processing technology is expected to be applied in a variety of fields that handle document data, such as document summarization and translation, voice dialogue, and data analysis.

[0003] One application of natural language processing is named entity extraction, which extracts the values ​​of predefined items (named entities) from document data. For example, if a corporate name and expiration date are defined as named entities, the character strings corresponding to the corporate name and expiration date can be extracted from the document data.

[0004] Currently, the mainstream of natural language processing is natural language processing models such as BERT, which uses Transformers. This model breaks down character strings contained in document data into units called tokens, and the input data is the vectorized tokens.

[0005] Generally, there is an upper limit to the number of tokens that a natural language processing model can process at one time. Therefore, when inputting long text data containing tokens exceeding the upper limit, it is necessary to divide the text data into multiple token groups and input and process them separately.

[0006] However, if multiple tokens contained in a single document are divided according to the input limit, the keywords and context (character strings surrounding the named entity) used to distinguish the named entity may be lost, resulting in a decrease in estimation accuracy.

[0007] In Patent Document 1, document data is divided into sections such as chapters, sections, and paragraphs, and named entities are extracted using a natural language processing model for each token group that is expected to retain a certain context. Furthermore, in Patent Document 1, tokens contained in document data are divided into sections so that the number of tokens processed at one time falls within the upper input limit of the natural language processing model. However, because each divided section is processed separately, there is a problem in that the context contained in adjacent sections is lost, which can reduce estimation accuracy.

[0008] Furthermore, in Patent Document 2, when dividing into token groups, previous and subsequent token groups are overlapped to prevent the loss of context contained in adjacent sections. This makes it possible to suppress a decrease in estimation accuracy in natural language processing when dividing tokens contained in document data. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent Publication No. 2021-64143 [Patent Document 2] Japanese Patent Publication No. 2023-072557 Summary of the Invention [Problem to be solved by the invention]

[0010] The technology of Patent Document 2 does not lose context within the range of the overlapping number of tokens, but there is a possibility that the context will be lost in a range greater than that, and in such cases the estimation accuracy may decrease.

[0011] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a mechanism that can suppress a decrease in estimation accuracy in natural language processing. [Means for solving the problem]

[0012] The present invention is an information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, and is characterized by comprising: a compression means for performing a compression process in which, when the number of tokens included in a token group obtained by decomposing the input character string exceeds a predetermined upper limit, a compression means for extracting named entities from the token group; and a restoration means for restoring the compressed named entity to its pre-compression state, if the extracted named entity has been compressed by the compression process. [Effects of the Invention]

[0013] According to the present invention, it is possible to suppress a decrease in estimation accuracy in natural language processing. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a named entity extraction device according to an embodiment of the present invention. [Figure 2] FIG. 4 is a diagram illustrating an example of document data received by a receiving unit. [Figure 3] 10 is a table showing an example of a token acquired by a control unit. [Figure 4] 4 is a flowchart illustrating an example of processing executed by a control unit of the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of a token compression process. [Figure 6] 10 is a flowchart illustrating an example of processing executed by a control unit according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0016] [First embodiment] In the first embodiment, an example of a system will be described that extracts character strings from document data, breaks them down into tokens, and extracts and outputs named entities using a natural language processing model. If named entities can be extracted by inputting all tokens obtained from document data into a natural language processing model at once, results can be estimated efficiently. However, natural language processing models have an input limit, meaning there is an upper limit on the number of tokens that can be input at one time. Therefore, if a long sentence contains tokens whose number exceeds the input limit, it is necessary to divide the token groups into multiple tokens and input them into the natural language processing model. However, simply dividing the token groups to fit the input limit of the natural language processing model may result in the loss of keywords or contexts for distinguishing named entities, which may result in a decrease in the accuracy of named entity extraction. In the first embodiment, specific tokens are compressed from the input tokens to reduce the number of tokens, thereby keeping the number of tokens within the input limit, thereby extracting named entities without dividing them into token groups.

[0017] Fig. 1(a) is a block diagram showing an example of the functional configuration of a named entity extraction device 100 as an information processing device according to an embodiment of the present invention, and Fig. 1(b) is a block diagram showing an example of the hardware configuration of the named entity extraction device 100.

[0018] The named entity extraction device 100 is an information processing device having functional units such as a control unit 101, a receiving unit 102, a calculation unit 103, a division unit 104, and an extraction unit 105. These functional units 101 to 105 are realized by a CPU 111 of the named entity extraction device 100 reading a program stored in a ROM 113 or a storage device 116, etc., into a RAM 112 and executing the program.

[0019] The control unit 101 executes various processes such as named entity extraction. The receiving unit 102 receives (acquires) document data to be subjected to named entity extraction processing by operating the input device 114 provided in the named entity extraction device 100. The document data to be acquired may be stored in the storage device 116, or may be document data on the network 118 (which may be document data managed by a cloud service or the like) via the network interface 117. The control unit 101 breaks down the input character string extracted from the document data acquired by the receiving unit 102 into tokens.

[0020] The calculation unit 103 counts the number of tokens in the document data, calculates the ratio of the number of tokens in the document data to the number of named entities, and so on. The extraction unit 105 uses a natural language processing model to extract named entities from tokens obtained from document data. The extracted named entities are, for example, stored in the storage device 116 and displayed on the output device 115, or output to a device (which may be a cloud service or the like) on the network 118 via the network interface 117.

[0021] The dividing unit 104 divides the tokens obtained from the document data into multiple token groups so that the number of tokens input at one time is less than the upper limit number of inputs to the natural language processing model. Note that the dividing unit 104 is used in the second embodiment described later.

[0022] FIG. 2A shows an example of document data received by the receiving unit 102. As shown in FIG. Document data is composed of multiple elements of different types, including character strings, symbols, and lines, within a page. Document data used for training and evaluation of natural language processing models is usually accompanied by ground truth data called GT (Ground Truth). The GT attached to document data 210 defines named entity types (or attributes) for each of the character strings 211, 212, and 213 within the dashed rectangles, i.e., corporate name, corporate name, and expiration date, to indicate that the character strings are named entities.

[0023] Document data 210 is usually composed of pages and includes symbols and ruled lines, but it may also be composed of multiple pages, or may be data that contains only character information without layout information. In other words, document data may be data in any format as long as character string information can be obtained.

[0024] FIG. 2B is a diagram showing another example of document data received by the receiving unit 102. As shown in FIG. The document data shown in FIG. 2(b) is similar to that shown in FIG. 2(a) and has a similar layout, except that no GT (correct answer data) is added.

[0025] 3(a) and 3(b) are diagrams showing examples of tokens acquired by the control unit 101. FIG. The token is represented by, for example, an identifier 311, a token string 312, and a GT 313. 3(a) is an example of a token table 310 listing 261 tokens obtained by decomposing, into token units, character strings extracted from the document data 210 to which the correct answer data shown in FIG. 2(a) has been assigned using morphological analysis by the control unit 101. Each token is assigned a GT in the IOB (Inside-Outside-Beginning) format.

[0026] In this embodiment, two types of named entities are used, namely, corporate name (ORG) and expiration date (DATE), but other types may be defined and used without being limited to these two types. Since a named entity may consist of multiple tokens, in such cases, "B-" is added to the GT of the first token of the named entity, and "I-" is added to the GT of subsequent tokens.

[0027] For example, the character string 211 "ABC Co., Ltd." shown in Figure 2(a) is a unique entity representation of a corporate name (ORG), but is composed of two tokens, T1_003 ("ABC") and T1_004 ("Co., Ltd."). Therefore, the GT of token T1_003 ("ABC") is assigned "B-ORG," which indicates the first token, and the GT of token T1_004 ("Co., Ltd.") is assigned "I-ORG," which indicates the subsequent token.

[0028] The GT of a token that is not assigned a named entity type is assigned "O" to indicate that it is not a named entity. In this embodiment, as described above, a GT is assigned to each token in the IOB format, but other methods may be used to assign a GT to a named entity that spans multiple tokens.

[0029] FIG. 3(b) is an example of a token table 320 that lists 264 tokens obtained by the control unit 101 decomposing the input string extracted from the document data 220 shown in FIG. 2(b) into token units using morphological analysis. The table 320 is based on the document data 220 to which no GT has been added, and therefore has an identifier 321 and a token string 322 for each token, but no GT.

[0030] Fig. 4 is a flowchart showing an example of processing executed by the control unit 101 of the first embodiment. The processing shown in Fig. 4 and the flowchart in Fig. 5 described later is realized by the CPU 111 of the named entity extraction device 100 reading a program stored in the ROM 113 or the storage device 116, etc., into the RAM 112 and executing the program. Note that the processing of the flowchart shown in Fig. 4 is executed in response to an instruction to extract named entities from document data being issued by the named entity extraction device 100.

[0031] In S401, the control unit 101 decomposes, into token units, an input character string extracted from document data acquired by the receiving unit 102. For example, in this step, an input character string extracted from document data 220 to which no correct answer data is assigned is decomposed into token units. In the case of the document data 220 illustrated in FIG. 2(b), the character string is decomposed into tokens and extracted as shown in table 320 illustrated in FIG. 3(b). Although not shown in FIG. 3(b), "March 5th" in the document data 220 of FIG. 2(b) is decomposed and extracted into four tokens, for example, "3", "month", "5", and "day". Furthermore, "￥2,500" is decomposed and extracted into four tokens, for example, "￥", "2", ",", and "500". Furthermore, "Setagaya-ku, Tokyo" is decomposed and extracted into four tokens, for example, "Tokyo", "to", "Setagaya", and "ward".

[0032] Next, in S402, the control unit 101 compares the number of tokens included in the token group obtained by decomposing the input character string in S401 with the upper limit of the number of inputs of the natural language processing model, which is a named entity extractor, and determines whether the number of tokens exceeds the upper limit of the inputs. For example, the number of tokens stored in the table 320 illustrated in Fig. 3(b) is 264, and it is determined that this exceeds the upper limit of the inputs of the natural language processing model, which has an upper limit of 256.

[0033] Here, if the number of tokens does not exceed the upper limit of input for the named entity extractor (No in S403), the control unit 101 advances the process to S407. In S407, the control unit 101 causes the extraction unit 105 to execute named entity extraction for the tokens obtained by decomposing the input character string in S401, and extracts the corporate name (ORG) and expiration date (DATE). After the processing of S407, the control unit 101 ends the processing of this flowchart (named entity extraction processing).

[0034] On the other hand, if the number of tokens exceeds the upper limit of input to the named entity extractor (Yes in S403), the control unit 101 advances the process to S403. In S403, the control unit 101 extracts specific tokens from the tokens obtained from the input character string and performs compression processing. Here, the processing of compressing specific tokens in S403 will be described in detail with reference to FIG.

[0035] FIG. 5 is a flowchart showing an example of a process for compressing a specific token executed by the control unit 101 of the first embodiment. In S501, the control unit 101 identifies compressible tokens. In this embodiment, compressible character strings include, for example, tokens written in a specific format or regular expressions, such as dates and amounts, and proper nouns such as prefectures, cities, wards, towns, and villages (i.e., proper nouns that match predefined tokens or regular expressions). In the former case, tokens such as year, month, and date, and the digits before and after them are extracted as compressible tokens. In the latter case, information such as prefectures, cities, wards, towns, and villages is compiled into a database and stored in advance in the storage device 116, and the target token is extracted as a compressible token by matching it with each token. Furthermore, if the document data includes table information, multiple tokens included in the table are extracted as compressible tokens.

[0036] Specific examples will be described below. For example, if four tokens, "3," "month," "5," and "day," are extracted in this order in S401 of FIG. 4, these four tokens are identified as compressible tokens in S501. Furthermore, if four tokens, "￥," "2," "," and "500," are extracted in this order, these four tokens are identified as compressible tokens. Furthermore, if four tokens, "Tokyo," "To," "Setagaya," and "Ku," are extracted in this order, these four tokens are identified as compressible tokens. Furthermore, if multiple tokens (such as "item name," "quantity," "unit price," and "amount" in the example of FIG. 2B) are contained within an area having predefined characteristics (in this embodiment, a tabular area, i.e., a table, but not limited to a table), these tokens are extracted as compressible tokens. Note that methods other than the above may also be used to identify compressible tokens. For example, table information within document data may be detected from ruled line information.

[0037] Next, in S502, the control unit 101 determines whether or not the compressible token identified in S501 exists. If there is no compressible token (No in S502), the control unit 101 ends the processing of this flowchart and moves the process to S404 in FIG.

[0038] On the other hand, if there is a compressible token (Yes in S502), the control unit 101 advances the process to S503. In S503, the control unit 101 replaces the character string consisting of the multiple tokens identified in S501 with a single token. In this embodiment, for example, a token representing a date such as the extracted year, month, and day is replaced with a single token "D". A token representing an amount such as ¥ is replaced with a single token "M". A token representing a prefecture, city, ward, town, or village is replaced with a single token "A". In addition, if the information is a table, multiple tokens included in the table (such as "item name", "quantity", "unit price", and "amount" in the example of FIG. 2(b)) are replaced with a single token "T".

[0039] A specific example of the above will be explained below. In the above example, four tokens, "3," "month," "5," and "day," were identified as compressible tokens, and these tokens representing dates are replaced with a single token, "D." As a result, the character string "March 5th" is replaced with "D." Furthermore, four tokens, "￥," "2," "," and "500," were identified as compressible tokens, and these tokens representing amounts are replaced with a single token, "M." As a result, the character string "2,500" is replaced with "M." Furthermore, four tokens, "Tokyo," "To," "Setagaya," and "Ku," were identified as compressible tokens, and these tokens representing prefectures, cities, wards, towns, and villages are replaced with a single token, "A." As a result, the character string "Setagaya Ward, Tokyo" is replaced with "A." Furthermore, multiple tokens included in the table (such as "product name," "quantity," "unit price," and "amount") are replaced with a single token, "T."

[0040] Furthermore, when performing the replacement as described above in S503, the control unit 101 associates the information of the original token with the information after replacement and stores them together with the location information in the storage device 116 so that the token before replacement (state before compression) can be restored. In the above example, the information of the original token "March 5th", the information after replacement "D", and the location information are associated and stored in the storage device 116. Other replaced information is also stored in the same way so that the original information can be restored. In this embodiment, tokens such as "D", "M", "A", and "T" are used for replacement, but other tokens may be used as long as the number of tokens is less than the number of tokens in the string consisting of the identified multiple tokens.

[0041] In the process of S503, compression of a specific token may be repeated until the number of tokens reaches the upper limit of input to the extractor, and compression of the specific token may be terminated when the number of tokens reaches the upper limit of input. Furthermore, in the process of S503, tokens unrelated to the named entity to be extracted by the extraction unit 105 may be compressed, but tokens related to the named entity to be extracted may not be compressed. Specifically, for example, if the type of the named entity to be extracted is a corporate name (ORG) and an expiration date (DATE), tokens representing the corporate name and date may be controlled not to be compressed because they are related to the expiration date (DATE). In the above example, "3," "month," "5," and "day" are tokens representing dates and are related to the named entity to be extracted, "expiration date," so they may not be compressed. Alternatively, tokens related to the named entity to be extracted may be controlled to be compressed only if the number of tokens does not reach the upper limit of input even if tokens unrelated to the named entity to be extracted are compressed. After the above-described process of S503, the control unit 101 ends the process of this flowchart and moves the process to S404 in FIG.

[0042] Returning to the description of the flowchart in FIG. In S404, the control unit 101 causes the extraction unit 105 to perform named entity extraction on the tokens compressed in S403 above, and extracts the corporate name (ORG) and expiration date (DATE). For example, if the input document data is document data 220 in FIG. 2(b), this named entity extraction extracts "JKL Transport Company" as the corporate name (ORG). Furthermore, since the compression process of the specific token in S403 described above has replaced "March 5th," which corresponds to the expiration date, with "D," this named entity extraction extracts "D" as the expiration date (DATE).

[0043] Next, in S405, the control unit 101 determines whether the named entity extracted in S404 above has been compressed in S403 above. If it is determined that the data has not been compressed (No in S405), the control unit 101 ends the processing of this flowchart (named entity extraction processing).

[0044] On the other hand, if it is determined that the named entity extracted in S404 above has been compressed in S403 above (Yes in S405), the control unit 101 proceeds to S406. In the above example, since "D" extracted as the expiration date (DATE) was compressed (replaced) from "March 5th" in S403, the determination in S404 in this example will be Yes.

[0045] In S406, the control unit 101 restores the named entities extracted in S404 above that were compressed in S403 above to the original tokens based on the token information before replacement and its location information stored in the storage device 116. In the above example, "D" extracted as the expiration date (DATE) is restored to the original replacement value "March 5th." After the process of S406, the control unit 101 ends the process of this flowchart (named entity extraction process). Note that the named entity extracted (or restored after extraction) as described above is, for example, stored in the storage device 116 and displayed on the output device 115, or output to a device on the network 118 via the network interface 117.

[0046] As described above, when extracting named entities from document data with a large number of characters, by compressing pre-input token information through substitution and keeping the number of tokens below an upper limit, it becomes possible to extract named entities without dividing tokens into groups as in the past. As a result, it is possible to prevent the loss of keywords and contexts for distinguishing named entities due to token division into groups, and to suppress a decrease in the accuracy of named entity extraction. In this way, in this embodiment, by compressing specific character strings, tokens contained in document data are not divided into groups (i.e., the input document is not divided), and therefore it is possible to suppress a decrease in estimation accuracy in natural language processing.

[0047] Second Embodiment In the first embodiment, named entity processing was performed without dividing the input document data into multiple token groups by compressing part of the input document data. In contrast, in this embodiment, the processing to be performed is selected based on the ratio between the number of tokens in the input document data and the number of named entities to be extracted.

[0048] In the first embodiment, named entities are extracted by compressing specific tokens. However, if the number of tokens in a document is small compared to the number of named entities to be extracted, there may be few tokens that can be compressed. In such cases, it is possible that the number of tokens cannot be kept below the upper limit, resulting in inappropriate named entity extraction. Therefore, in this embodiment, if the ratio obtained by dividing the number of tokens in an input document by the number of named entities to be extracted (i.e., 10) is equal to or greater than a threshold value (e.g., 10), processing similar to that of the first embodiment is performed. On the other hand, if the ratio is less than the threshold value, the tokens in the document are divided into multiple token groups, and named entity extraction is performed for each divided token group. For example, in the case of the document data 220 in FIG. 2(b), the number of tokens is 264 and the number of named entities to be extracted (in the above example, the two entities, namely, the corporate name and expiration date) is 2, so the ratio is 132, i.e., greater than the threshold value of 10. However, if the number of named entities to be extracted is 27 or greater, the ratio falls below the threshold value of 10.

[0049] Fig. 6 is a flowchart showing an example of processing executed by the control unit 101 of the second embodiment. That is, the processing shown in Fig. 4 and the flowchart of Fig. 5 described later is realized by the CPU 111 of the named entity extraction device 100 reading a program stored in the ROM 113 or the storage device 116, etc., into the RAM 112 and executing the program. This flowchart is executed in response to an instruction to extract named entities from document data in the named entity extraction device 100. Note that the same steps as in Fig. 4 are assigned the same step numbers, and their explanations will be omitted here.

[0050] In the second embodiment, if the number of tokens exceeds the upper limit of input to the named entity extractor in S403 (Yes in S403), the control unit 101 advances the process to S601.

[0051] In S601, the control unit 101 uses the calculation unit 103 to calculate the ratio between the number of tokens in the input document and the number of named entities to be extracted, and determines whether the ratio is equal to or greater than a threshold value. Here, if the ratio between the number of tokens in the input document and the number of named entities to be extracted is equal to or greater than the threshold value (Yes in S601), the control unit 101 proceeds to S403 and performs the same processing as in the first embodiment.

[0052] On the other hand, if the ratio between the number of tokens in the input document and the number of named entities to be extracted is less than the threshold (No in S601), the control unit 101 proceeds to S602. Note that it is also possible to determine Yes in S601 and proceed to S403 if the ratio is greater than the threshold, and to determine No in S601 and proceed to S602 if the ratio is equal to or less than the threshold.

[0053] In S602, the control unit 101 performs a division process in which the division unit 104 divides the tokens in the document into multiple token groups. Any method may be used to divide the tokens. For example, the method used in Patent Document 2 is adopted. That is, when the number of tokens included in the token group obtained by decomposing the input character string exceeds a predetermined upper limit, the document data is divided so that they overlap each other based on the number of tokens near the limit at which a predetermined named entity can be extracted. Note that in this embodiment, for example, the method used in Patent Document 2 is adopted, but the present invention is not limited to this. For example, the method used in Patent Document 1 may also be adopted.

[0054] Next, in S603, the control unit 101 uses the extraction unit 105 to perform named entity extraction for each of the divided token groups, extracting the corporate name (ORG) and expiration date (DATE) for each, and then ends the processing of this flowchart. Note that only one of the named entities extracted from the overlapping portions of each token group is output.

[0055] As described above, when the ratio of the number of tokens in a document to the number of named entities to be extracted is small, the accuracy of named entity extraction can be prevented from decreasing by dividing the document into token groups. The control of the first embodiment and the control of the second embodiment may be switched according to a user setting.

[0056] Third Embodiment In the first and second embodiments, named entity extraction is performed using character strings contained in a single document as input document data. In contrast, in this embodiment, named entity extraction is performed from a document spanning multiple pages. The steps of extracting character strings from document data, breaking down into tokens, and extracting and outputting named entities using a natural language processing model are the same as those in the first and second embodiments, and therefore will not be described here.

[0057] In this embodiment, a method in which a user specifies a document in advance is used as a method for selecting documents spanning multiple pages, but other methods may also be used. For example, a method in which page numbers are estimated as named entities from each document data may also be used. If consecutive page numbers exist in the same position, the document is determined to be spanning multiple pages, and the extracted named entities are integrated.

[0058] As described above, the user specifies a document spanning multiple pages, or the page numbers are estimated and named entities are extracted from a document spanning multiple pages, so named entities that exist across pages can be extracted.

[0059] As described above, according to each embodiment, it is possible to suppress a decrease in estimation accuracy in natural language processing.

[0060] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and that the data may be configured in various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device. Furthermore, the present invention also includes any combination of the above embodiments.

[0061] Other Embodiments The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. Furthermore, the present invention may be applied to a system made up of multiple devices, or to an apparatus made up of a single device. The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. In other words, all configurations that combine the above-described embodiments and their modifications are included in the present invention.

[0062] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a compression means for performing a compression process in which, when the number of tokens included in a token group obtained by decomposing the input character string exceeds a predetermined upper limit, a specific plurality of tokens included in the token group are replaced with a single token; extraction means for extracting named entities from the group of tokens; a restoration means for restoring the compressed named entity to its pre-compression state when the extracted named entity has been compressed by the compression process; An information processing device comprising: (Configuration 2) 2. The information processing device of claim 1, wherein the specific token is a proper noun that matches a predefined token or regular expression. (Configuration 3) 2. The information processing apparatus of claim 1, wherein the specific token is a token included in an area having a predefined characteristic. (Configuration 4) 4. The information processing device of configuration 3, wherein the area having the predefined characteristics is an area in a table format. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the compression means performs the compression process until the number of tokens included in the token group reaches a predetermined upper limit number. (Configuration 6) 5. The information processing device according to any one of configurations 1 to 4, wherein the compression means does not compress tokens related to named entities that are the subject of extraction by the extraction means. (Configuration 7) The information processing device described in any one of configurations 1 to 4, characterized in that when the number of tokens included in the token group obtained by decomposing the input character string exceeds a predetermined upper limit, the compression means performs a process of compressing tokens that are not related to the named entity to be extracted by the extraction means, and further, when the number of tokens included in the token group after performing the process exceeds a predetermined upper limit, the compression means performs a process of compressing tokens that are related to the named entity to be extracted. (Configuration 8) 5. The information processing device according to any one of configurations 1 to 4, wherein the input character string is data consisting of a multi-page document. (Configuration 9) a division means for performing a division process in which, when the number of tokens included in the token group obtained by decomposing the input character string exceeds a predetermined upper limit, the input character string is divided so as to overlap with each other based on the number of tokens near a predetermined limit at which named entities can be extracted; a control means for switching between a compression process by the compression means and a division process by the division means in accordance with a ratio between the number of tokens included in the token group obtained by dividing the input character string and the number of named entities to be extracted; 5. The information processing device according to any one of configurations 1 to 4, comprising: (Method 1) A control method for an information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a compression step of performing a compression process in which, when the number of tokens included in a token group obtained by decomposing the input character string exceeds a predetermined upper limit, a specific plurality of tokens included in the token group are replaced with one token; an extraction step of extracting named entities from the group of tokens; a restoration step of restoring the compressed named entity to its pre-compression state if the extracted named entity has been compressed by the compression process; 1. A method for controlling an information processing device, comprising: (Program 1) A program for causing a computer to function as each of the means according to any one of configurations 1 to 4. [Explanation of symbols]

[0063] 100 Named entity extraction device 101 Control section 102 Receiving unit 103 Calculation Unit 104 Division 105 Extraction part

Claims

1. An information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a compression means for performing a compression process in which, when the number of tokens included in a token group obtained by decomposing the input character string exceeds a predetermined upper limit, a specific plurality of tokens included in the token group are replaced with a single token; extraction means for extracting named entities from the group of tokens; a restoration means for restoring the compressed named entity to its pre-compression state when the extracted named entity has been compressed by the compression process; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the specific token is a proper noun that matches a predefined token or regular expression.

3. 2. The information processing apparatus according to claim 1, wherein the specific token is a token included in an area having a predefined characteristic.

4. 4. The information processing apparatus according to claim 3, wherein the area having the predefined characteristics is an area in a tabular format.

5. 5. The information processing apparatus according to claim 1, wherein the compression means performs the compression process until the number of tokens included in the token group reaches a predetermined upper limit number.

6. 5. The information processing apparatus according to claim 1, wherein the compression means does not compress tokens related to the named entity extracted by the extraction means.

7. The information processing device according to any one of claims 1 to 4, characterized in that, when the number of tokens included in the group of tokens obtained by decomposing the input character string exceeds a predetermined upper limit, the compression means performs a process of compressing tokens that are not related to the named entity to be extracted by the extraction means, and further, when the number of tokens included in the group of tokens after performing the process exceeds a predetermined upper limit, the compression means performs a process of compressing tokens that are related to the named entity to be extracted.

8. 5. The information processing apparatus according to claim 1, wherein the input character string is data consisting of a document of multiple pages.

9. a division means for performing a division process in which, when the number of tokens included in the token group obtained by decomposing the input character string exceeds a predetermined upper limit, the input character string is divided so as to overlap with each other based on the number of tokens near a predetermined limit at which named entities can be extracted; a control means for switching between a compression process by the compression means and a division process by the division means in accordance with a ratio between the number of tokens included in the token group obtained by dividing the input character string and the number of named entities to be extracted; 5. The information processing device according to claim 1, further comprising:

10. A control method for an information processing device for extracting named entities from a plurality of tokens obtained by decomposing an input character string, comprising: a compression step of performing a compression process in which, when the number of tokens included in a token group obtained by decomposing the input character string exceeds a predetermined upper limit, a specific plurality of tokens included in the token group are replaced with one token; an extraction step of extracting named entities from the group of tokens; a restoration step of restoring the compressed named entity to its pre-compression state if the extracted named entity has been compressed by the compression process; 1. A method for controlling an information processing device, comprising:

11. A program for causing a computer to function as each of the means according to any one of claims 1 to 4.

Images