Information processor, information processing method and program

Abstract

To appropriately determine the identity of unique expressions contained in object information using a large-scale language model.SOLUTION: An information processor 100 includes: an assignment unit 120 which extracts money amount data contained in an object document 10 and attribute information for identifying the money amount data using an LLM and assigns the extracted attribute information to the extracted money amount data; and determination part 130 which identifies a plurality of money amount data of identical type from the money amount data contained in the object document 10 according to the attribute information and compares the plurality of money amount data to determine whether the plurality of money amount data are identical or not.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program capable of handling linguistic information. [Background technology]

[0002] Conventionally, there are techniques for handling linguistic information. For example, a technique has been proposed in which inconsistent data and corresponding expressions in text are corrected using a factual data database (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-167576 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, in order to detect inconsistencies from factual data contained in text, it is necessary to store factual data indicating a large number of facts in a factual data database. However, it is considered difficult to store all of the factual data necessary for appropriate inconsistency detection in the factual data database.

[0005] The present invention aims to appropriately determine the identity of named entities contained in target information by utilizing a large-scale language model. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that includes an assignment unit that uses a large-scale language model to extract named entities contained in target information and attribute information that can identify the named entities, and assigns the extracted attribute information to the extracted named entities, and a determination unit that identifies multiple named entities of the same type from among the named entities contained in the target information based on the attribute information, and compares the multiple named entities to determine whether the multiple named entities are identical. [Effects of the Invention]

[0007] According to the present invention, it is possible to appropriately determine the identity of named entities contained in target information by utilizing a large-scale language model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 2] FIG. 2 is a diagram showing an example of a target document to be processed by the information processing device. [Figure 3] FIG. 3 is a diagram illustrating an example of attribute information assigned by the assigning unit. [Figure 4] FIG. 4 is a diagram showing a schematic flow of an extraction process when extracting amount data and attribute information of the amount data contained in a target document using LLM. [Figure 5] FIG. 5 is a diagram showing an example of the relationship between input data input to the LLM and output data output from the LLM. [Figure 6] FIG. 6 is a simplified diagram showing a tagged target document output from the LLM in response to an input of the target document. [Figure 7] FIG. 7 is a diagram showing a comparative example of price data and its attribute information included in a tagged target document. [Figure 8] FIG. 8 is a diagram showing an example of a notification when a named entity in which an error has been detected in the determination process is notified to the user. [Figure 9] FIG. 9 is a flowchart showing an example of the determination process. [Figure 10]FIG. 10 is a diagram illustrating an example of use of the information processing device. [Figure 11] FIG. 11 is a diagram illustrating an example of use of the information processing device. [Figure 12] FIG. 12 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 13] FIG. 13 is a diagram showing an example of classification contents obtained by classifying named entities from the viewpoint of information processing technology. [Figure 14] FIG. 14 is a diagram illustrating an example of attribute information related to a named entity. [Figure 15] FIG. 15 is a diagram showing an example of attribute information relating to numerical values ​​(named entities). [Figure 16] FIG. 16 is a diagram showing an example of attribute information related to MONEY (named entity). [Figure 17] FIG. 17 is a diagram showing an example of transition from target information to tagged target information. [Figure 18] FIG. 18 is a diagram showing an example of attribute information relating to a named entity (named entity). [Figure 19] FIG. 19 is a diagram showing an example of attribute information related to PERSON (person's name). [Figure 20] FIG. 20 is a diagram showing an example of transition from target information to tagged target information. [Figure 21] FIG. 21 is a diagram showing an example of transition from target information to tagged target information. [Figure 22] FIG. 22 is a diagram showing an example of transition from target information to tagged target information. [Figure 23] FIG. 23 is a diagram showing an example of attribute information related to ORG (organization name). [Figure 24] FIG. 24 is a diagram showing an example of transition from target information to tagged target information. [Figure 25] FIG. 25 is a diagram showing an example of attribute information related to LOC (place name). [Figure 26] FIG. 26 is a diagram showing an example of attribute information related to COUNTRY (place name). [Figure 27]FIG. 27 is a diagram illustrating an example of attribute information related to time. [Figure 28] FIG. 28 is a diagram showing an example of attribute information related to DATE. [Figure 29] FIG. 29 is a diagram showing an example of attribute information related to TIME. [Figure 30] FIG. 30 is a flowchart showing an example of a determination process using a plurality of assignment results. [Figure 31] FIG. 31 is a diagram showing an example of transition from target information to tagged target information. [Figure 32] FIG. 32 is a flowchart showing an example of a determination process performed after eliminating spelling variations. [Figure 33] FIG. 33 is a flowchart showing an example of a determination process using attributes defined by the LLM. [Figure 34] FIG. 34 is a diagram illustrating an example of graph information that can be realized using LLM. [Figure 35] FIG. 35 is a flowchart showing an example of a determination process using graph information. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Configuration example of information processing device] 1 is a block diagram showing an example of the functional configuration of an information processing device 100. The information processing device 100 can be realized by an information processing device or electronic device such as a server, a personal computer, a smartphone, or a tablet terminal.

[0011] FIG. 2 is a diagram illustrating an example of a target document 10 to be processed by the information processing device 100. The target document 10 is an example of a document including text information 11 and table information 12. While FIG. 2 illustrates a target document 10 including both text information 11 and table information 12, the present embodiment is not limited to this example and can also be applied to a target document including either text information or table information. The target document 10 is an example of a financial statement disclosed by a company. This financial statement is, for example, a securities report. For example, numbers in a securities report are often related between tables and between tables and text. Furthermore, when matching monetary data in a securities report, manual matching is likely to result in human error. Therefore, in this embodiment, a large-scale language model is used to automatically determine the identity of monetary data included in the target document 10, such as a securities report. This improves the accuracy of matching monetary data.

[0012] Here, the text information 11 is information in which various information such as characters, numbers, symbols, etc. are written in a sentence format, and the table information 12 is information in which information such as characters, numbers, symbols, etc. are written in a table format (for example, a table format or a graph format).

[0013] 1, the information processing device 100 includes an acquisition unit 110, an attachment unit 120, a determination unit 130, a recording control unit 140, a storage unit 150, an output control unit 160, and an output unit 170. Each of the acquisition unit 110, the attachment unit 120, the determination unit 130, the recording control unit 140, and the output control unit 160 is realized by, for example, one or more processing circuits such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0014] The acquiring unit 110 accepts target information (for example, target document 10) input by the user, and outputs the target information to the tagging unit 120. This target information is information to be tagged.

[0015] For example, the acquisition unit 110 can be an input device (e.g., a keyboard, a mouse, a recording medium reader, an imaging device, a voice input device, or a scanner) that can input target information including characters, numbers, etc. For example, the voice input device can be a microphone that can input information such as characters, numbers, etc. by voice, a voice input device such as an input device dedicated to voice recognition, etc. For example, the imaging device can be an image acquisition device such as a camera that can capture information such as characters, numbers, etc. and acquire the image information. Note that when image information such as characters, numbers, etc. is acquired by an imaging device, a scanner, etc., it is possible to acquire the information such as characters, numbers, etc. contained in the image information using known character recognition technology. For example, the target information can be read and acquired from a recording medium (e.g., a memory card, a Universal Serial Bus (USB) memory, a Hard Disk Drive (HDD), a Compact Disc (CD), a Digital Versatile Disc (DVD), a Blu-ray (registered trademark) Disc (BD), etc.) that stores a file in which the target information is stored (e.g., a file of the target document 10). Furthermore, for example, a recording medium on which the target information is stored can be connected to the information processing device 100 via wireless or wired communication, and the acquisition unit 110 can read and acquire the target information from the recording medium. Furthermore, if the recording medium is built into the information processing device 100, the acquisition unit 110 can read and acquire the target information from the recording medium. In this way, the acquisition unit 110 is realized by an input interface, a file input device, an imaging device, etc. Note that the file format of the information acquired by the acquisition unit 110 is not particularly limited. For example, any format such as PDF or HTML may be used.

[0016] The assigning unit 120 extracts named entities included in the target information output from the acquiring unit 110 and attribute information that can identify the named entities, and assigns the extracted attribute information to the extracted named entities. That is, the assigning unit 120 tags the named entities included in the target information. The assigning unit 120 then outputs the tagged target information (tagged target document 20 (see FIG. 6 )) to the determining unit 130, the recording control unit 140, and the output control unit 160.

[0017] Here, a named entity refers to a word or phrase with a unique name, and often refers to a proper noun that is specifically limited to nouns. Examples of proper nouns include monetary amounts such as "100 yen," "198 million yen," or "987 US dollars," place names such as "Tokyo," "Osaka," or "Sapporo," or names such as "Yamada Ichiro" or "Tanaka Goro."

[0018] For example, the Message Understanding Conference (MUC) defines the following seven types as named entities: Organization name (ORGANIZATION (ORG)), person's name (PERSON), place name (LOCATION), date expression (DATE), time expression (TIME), monetary amount expression (MONEY), percentage expression (PERCENTAGE)

[0019] Named Entity Recognition (NER), which extracts named entities contained in a target text, is also known as a natural language processing technology. For example, various extraction methods can be used for named entity extraction, such as extraction using dictionary information, rule-based extraction, and machine learning extraction.

[0020] In this embodiment, an example will be described in which a securities report is the target document 10 and monetary expressions (also referred to as monetary data) are extracted as named entities. However, as will be described later, this embodiment can also be applied to cases in which other named entities are extracted from other target documents.

[0021] In addition, in this embodiment, an example of extracting one or more pieces of attribute information related to a named entity will be described. Also, associating attribute information extracted for a named entity with the named entity will be described as tagging. For example, it is possible to tag attribute information (1) to (4) described below as attribute information related to monetary amount data. Note that the attribute information can also be referred to as tag information, metadata, accompanying information, additional information, etc. Note that the method of extracting attribute information will be described in detail with reference to Figs. 4 to 6, etc.

[0022] The determination unit 130 determines whether the named entities tagged with attribute information by the assignment unit 120 are identical, and outputs the determination result to the recording control unit 140 and the output control unit 160. For example, the determination unit 130 identifies multiple named entities of the same type from among the named entities included in the target information based on the attribute information tagged to the named entities. The determination unit 130 then compares the multiple named entities and determines whether the multiple named entities are identical. This determination process will be described in detail with reference to FIG. 7 etc.

[0023] The recording control unit 140 executes recording control to record the target information tagged by the tagging unit 120 and the determination result by the determination unit 130 in the storage unit 150 .

[0024] The storage unit 150 is a storage medium that stores various types of information. For example, the storage unit 150 stores various types of information (e.g., control programs) required for the acquisition unit 110, the assignment unit 120, the determination unit 130, the recording control unit 140, and the output control unit 160 to perform various processes. As the storage unit 150, various storage media such as a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof can be used.

[0025] The output control unit 160 executes output control to cause the output unit 170 to output the target information tagged by the tagging unit 120 and the determination result by the determination unit 130.

[0026] The output unit 170 outputs various types of information based on the control of the output control unit 160. For example, the output unit 170 can be configured with a display unit 171 (see FIG. 8 ) and a sound output unit. The display unit 171 displays various images based on instructions from the output control unit 160. For example, a display panel such as an organic EL (Electro Luminescence) panel or an LCD (Liquid Crystal Display) panel can be used as the display unit 171. The sound output unit outputs various sounds based on instructions from the output control unit 160. For example, one or more speakers can be used as the sound output unit. Note that the output unit 170 is an example of a user interface, and other user interfaces may also be used. For example, the information processing device 100 may output the target information tagged by the tagging unit 120 and the determination result by the determination unit 130 to an external output device (for example, a display device or a sound output device), and cause the output device to output the tagged target information and the determination result.

[0027] 1 shows an example in which the acquisition unit 110, the assignment unit 120, the determination unit 130, the recording control unit 140, the storage unit 150, the output control unit 160, and the output unit 170 are provided in the information processing device 100, but at least one of these may be used as a separate device different from the information processing device 100. For example, by registering at least one of the acquisition device, the storage device, and the output device in advance in the information processing device 100 (for example, by pairing or connecting using a wired or wireless line), it is possible to make the registered device function as the acquisition unit, the storage unit, and the output unit of the information processing device 100. Furthermore, as shown in FIG. 10, the information processing device 100 may be used as a user terminal, or as shown in FIG. 11, the information processing device 100 may be used as a server.

[0028] [Tag format example] 3 is a diagram showing an example of attribute information assigned by the assigning unit 120. In FIG. 3, the data of the amount expression (amount data) included in the target document 10 is <money>Specifically, as shown in a rectangle 200, the following attribute information (1) to (4) is given as attributes of the MONEY tag. (1) type (2) fiscal (3) unit (4) amount

[0029] (1) type is attribute information that indicates the type of monetary data. In other words, it is attribute information that indicates what type of monetary data it is. For example, it can be assigned "sales," "operating profit," "capital," "capital reserve," etc.

[0030] (2) Fiscal is attribute information that indicates the fiscal year of the monetary data. For example, it is possible to assign "previous period" or "current period."

[0031] (3) unit is attribute information indicating the unit of the monetary amount data. For example, "yen" or "million yen" can be assigned. Note that instead of the unit of the monetary amount data, attribute information indicating the type of currency of the monetary amount data (for example, "yen" or "US dollar") may be used.

[0032] (4) amount is attribute information that indicates the integer value of the amount data. For example, if the amount data is "7,118 million yen," it is possible to assign "7118000000."

[0033] The character strings extracted for (1) to (4) above are stored in the "string" within dotted rectangles 201 to 204 shown in rectangle 200. The method for extracting the amount data and attribute information will be described later.

[0034] Note that the attribute information of the MONEY tag shown in Figure 3 is an example, and some of it may be omitted or other attribute information may be added as needed. For example, the above-mentioned (2) may be omitted, and only the above-mentioned attribute information (1), (3), and (4) may be added and used. For example, if the target document contains information on capital, capital reserves, etc., it is possible to add and use only the above-mentioned attribute information (1), (3), and (4).

[0035] Also, for example, identification information "id" to be assigned to the amount data can be assigned as attribute information. For example, a serial number (1, 2, 3, ..., etc.) can be used as this identification information "id."

[0036] Additionally, for example, department-related information "segment" can be added as attribute information. In other words, the attribute information "segment" indicates which business segment within the reporting segment it is. For example, attribute information "segment" can be "consulting business," "business solutions business," "education business," etc.

[0037] Also, for example, information "title" indicating a heading corresponding to a parent concept can be assigned as attribute information. For example, "Financial status and business performance status" can be assigned as attribute information "title."

[0038] Also, for example, information "increase_decrease" (or "up_down") that indicates in detail what type of amount the amount data is can be assigned as attribute information. For example, information such as "total" or "increase ratio" can be assigned as attribute information "increase_decrease" to further classify amount data classified as "type" into more detailed types.

[0039] When extracting each of these attribute information using LLM, it is important to determine the attribute names so that they are consistent with the information that LLM can handle. Therefore, in this embodiment, an example will be shown in which the names of each of the attribute information (1) to (4) described above are used.

[0040] [Example of attribute information extraction] Next, a description will be given of an extraction method for extracting amount data included in the target document 10 and an extraction method for extracting attribute information of the amount data. These extraction processes are executed by the attachment unit 120 shown in FIG.

[0041] For example, the attachment unit 120 can extract amount data and attribute information of the amount data contained in the target document 10 using an AI (Artificial Intelligence) model (e.g., a machine learning model generated by machine learning). The term "learning" used in this embodiment refers to discovering patterns behind a large amount of data based on the data. The AI ​​model generated by learning used in this embodiment is generated using various learning algorithms. For example, various types of content (e.g., text, images) and text describing each of these contents can be read as training data to learn in advance, and this AI model can then be used in the extraction process.

[0042] Examples of AI models that can be used include large language models (LLMs) and multimodal LLMs. Examples of LLMs that can be used include various natural language processing models (e.g., Bidirectional Encoder Representations from Transformers (BERT)), Generative Pre-trained Transformer (ChatGPT), GPT-4, GPT-4 Turbo, GPT-4o (Omni), Bard, and Llama (Large Language Model Meta AI). These are merely examples, and other AI models may also be used. Figure 4 illustrates an example of an extraction process using an LLM 121 to extract amount data and attribute information for the amount data contained in a target document 10.

[0043] [Example of extraction processing using LLM] FIG. 4 is a diagram showing a flow of extraction processing when the LLM 121 is used to extract amount data and attribute information of the amount data contained in the target document 10. In FIG.

[0044] The LLM 121 is realized by the information processing device 100 (see FIGS. 10 and 11) that can execute various extraction processes using the LLM. For example, the attachment unit 120 transmits the target document 10 acquired by the acquisition unit 110 to a server, and the server executes an extraction process of the amount data contained in the target document 10 and the attribute information of the amount data. The attachment unit 120 can receive and use the processing result (tagged target document 20) from the server. In this case, the attachment unit 120 and the server function together as the attachment unit of the information processing device 100. The tagged target document 20 is information in which the amount data extracted from the target document 10 and the attribute information of the amount data are associated with the target document 10. Note that these can be executed by installing a predetermined application in the information processing device 100. Furthermore, for example, if the information processing device 100 is a device with a high computing speed (e.g., a server), the LLM process may be executed in the information processing device 100.

[0045] 5 is a diagram showing an example of the relationship between input data 210 input to the LLM 121 and output data 220 output from the LLM 121. The input data 210 corresponds to a part of a sentence included in the target document 10.

[0046] Here, when ChatGPT, GPT-4, GPT-4 Turbo, or GPT-4o is used as the LLM 121, instruction information (prompts) to be input to these LLMs will be described. For example, when assigning tags using an LLM, as described above, it is important to set attribute names that the LLM 121 can accept in the attribute information. In other words, it is important to set attribute names that are consistent with the information in the LLM 121. Therefore, in this embodiment, the attribute names (1) to (4) described above are set as attribute names that satisfy this condition.

[0047] For example, suppose a user creates sample data that includes one or more sentences before tagging and a sample sentence with the tag added to the sentence. In this case, the sample data for the tag is inserted into a prompt as a tagging example, and a prompt including instructions to tag a target document is input to the LLM along with the target document. The output data (e.g., target document 10, amount data, and attribute information for the amount data) can be used as tagged target information (e.g., tagged target document 20).

[0048] For example, assume that the sample data is a sentence included in the input data 210 shown in Fig. 5(A) and a sentence and attribute information included in the output data 220 shown in Fig. 5(B). In this case, the sentence included in the input data 210 and the sentence included in the output data 220 are inserted into a prompt as tagging examples, and the prompt including an instruction to tag the target document 10 is input to the LLM 121 along with the target document 10, and the corresponding output data (e.g., the target document 10, amount data, and attribute information of the amount data) can be used as tagged target information (e.g., the tagged target document 20).

[0049] Also, for example, monetary data can be extracted as a named entity, and prompts can be input to the LLM to extract (1) type, (2) fiscal, (3) unit, and (4) amount as attribute information of the extracted monetary data, and the corresponding output data (e.g., target document 10, monetary data, and attribute information of the monetary data) can be used as tagged target information (e.g., tagged target document 20). Note that these prompts are merely examples and are not limiting. Other prompts that can extract monetary data contained in target document 10 and attribute information of the monetary data can also be used.

[0050] For example, in the case of LLM, it is possible to process information such as characters, numbers, and symbols in table information 12 written in tabular form. In this case, for example, it is possible to identify and process information such as characters, numbers, and symbols in table information 12 based on each piece of information (e.g., item, cost) written above or to the side of the table in table information 12. For example, because "unit (million yen)" is written in the upper right corner of table information 12 (see FIG. 2), LLM can determine that each number written in table information 12 is related to monetary data related to the currency "yen."

[0051] Note that other extraction methods using machine learning may also be employed. For example, machine learning may be performed using a large amount of text and training data labeled with named entities and their corresponding attribute information to generate a trained model, and the trained model may be used to extract named entities and their corresponding attribute information.

[0052] Alternatively, an extraction method may be used that extracts named entities using dictionary information. For example, named entities to be extracted may be registered in advance as dictionary information (database), and named entities may be extracted by referencing this dictionary information. However, when an extraction method using dictionary information is used, there is a possibility that duplicate personal names and place names may be extracted. For example, the word "Tamachi" may be a personal name "Tamachi" and a place name "Tamachi." Furthermore, since it is difficult to register all numerical values ​​in dictionary information, it is preferable to extract numerical information using other character recognition techniques.

[0053] Alternatively, an extraction method using a rule base to extract named entities may be used. For example, certain rules for extracting named entities may be defined in advance, and named entities may be extracted based on those rules. For example, in the case of the word "Tamachi," a rule may be defined in advance that if "Tamachi" is followed by "san," "kun," or "chan," it is a person's name, and otherwise it is a place name. In other words, it is important to assign a part of speech to each word to be extracted and to define rules based on the surrounding words and parts of speech. In this way, extraction methods that extract named entities using dictionary information or a rule base involve humans devising certain rules and extracting information to be extracted based on whether or not the rules apply. Therefore, it may be difficult to determine all the rules in advance when there is a large amount of information to be extracted, or when the information to be extracted is numerical information. For this reason, the following describes an example of extracting named entities and attribute information using a large-scale language model.

[0054] [Example of output result] Fig. 6 is a simplified diagram showing a tagged target document 20 output from the LLM 121 in response to input of the target document 10. As shown in Fig. 6, a tagged target document 20 is generated in which the amount data extracted by the LLM 121 is associated with corresponding attribute information. Note that in Fig. 6, information corresponding to the text information 11 (see Fig. 2) is shown as text information 21, and information after conversion of the table information 12 (see Fig. 2) into the character information and numeric information contained therein is shown as text information 22.

[0055] Fig. 7 is a diagram showing the amount data and its attribute information included in the tagged target document 20 shown in Fig. 6, enclosed in dotted-line rectangles 31 to 39. The type, fiscal, unit, and amount included in each rectangle 31 to 39 correspond to the attribute information (1) to (4) described above.

[0056] 7, among the monetary data shown enclosed in dotted rectangles 31-39, monetary data identified as multiple monetary data of the same type based on their attribute information are shown connected by arrows C1-C3. That is, arrow C1 indicates dotted rectangles 31 and 34, arrow C2 indicates dotted rectangles 32 and 35, and arrow C3 indicates dotted rectangles 33 and 36. For these multiple monetary data of the same type, the monetary data is compared to determine whether the respective monetary data are the same.

[0057] Specifically, the determination unit 130 identifies the amount data included in the tagged target document 20 based on the attribute information of the amount data included in the tagged target document 20. For example, the determination unit 130 determines the amount data included in the tagged target document 20 based on the attribute information of the amount data included in the tagged target document 20. <money type="…"> …circle< / money> The part containing " " is identified. A known character recognition technique can be used to identify the character string within " ". Next, the determination unit 130 determines whether the identified character string " <money type="…"> …circle< / money> ' to obtain the attribute information of the amount data included in the specified string ' <money type="…"> …circle< / money> The attribute information of the amount data included in " " can be easily identified because each position is uniform.

[0058] Next, the determination unit 130 determines whether the specified character string " <money type="…"> …circle< / money> ', the determination unit 130 identifies the same type of amount data from among the amount data included in the identified character string ' <money type="…"> …circle< / money> Among the attribute information (type, fiscal, unit) included in the attribute information, the determination unit 130 identifies multiple pieces of monetary data whose attribute information matches. That is, the determination unit 130 identifies monetary data of the same type. For example, as shown in FIG. 7, the monetary data within the dotted rectangles 31 and 34 indicated by the arrow C1, the monetary data within the dotted rectangles 32 and 35 indicated by the arrow C2, and the monetary data within the dotted rectangles 33 and 36 indicated by the arrow C3 are identified as monetary data of the same type. Note that in the example shown in FIG. 7, the unit of the attribute information includes both "yen" and "million yen." Therefore, when identifying monetary data of the same type, it may be determined whether the portion of the unit of the attribute information corresponding to the currency (e.g., "yen") matches, or the determination of whether the unit matches may be omitted. Furthermore, if it is possible to identify monetary data of the same type using only one of the attribute information's type and fiscal, only one of them may be used. That is, it is possible to identify monetary data of the same type using at least one piece of attribute information. The value of the "amount" in the attribute information can also be calculated using the "unit" in the attribute information and the tagged amount data. For example, if the "unit" is "yen" and the tagged amount data is "277," then the amount can be calculated as 277. If the "unit" is "million yen" and the tagged amount data is "277," then the amount can be calculated as 277,000,000 (=277×1,000,000). For this reason, extraction of the amount can be omitted in the tagging process using LLM, and the determination unit 130 can calculate the amount using the "unit" in the attribute information and the tagged amount data.

[0059] Next, the determination unit 130 compares the multiple amount data identified as the same type and determines whether these amount data are the same. Specifically, the determination unit 130 compares the amount values ​​of the amount data identified as the same type and determines whether they are the same. For example, the values ​​of the amount data within dotted rectangles 31 and 34 indicated by arrow C1 are the same. The values ​​of the amount data within dotted rectangles 32 and 35 indicated by arrow C2 are the same. On the other hand, the values ​​of the amount data within dotted rectangles 33 and 36 indicated by arrow C3 are different. Therefore, although the values ​​of the amount data indicated by arrows C1 and C2 are the same, it is possible to determine that at least one of the values ​​of the amount data indicated by arrow C3 is incorrect. In this way, it is also possible that the attribute information (type, fiscal, unit) matches, but the amount information does not match. In this case, it is possible to determine that one of the amount information (i.e., amount data) being compared is incorrect. If the attribute information (type, fiscal, unit) matches but the amount information does not match, it is possible that an error has occurred in the tagging process by the tagging unit 120. In this case, the error must be resolved by user verification.

[0060] In this way, by using the attribute information of the amount data, it is possible to automatically perform the matching work of each amount data contained in the target document 10. This eliminates the need for an operator to visually perform the matching work of each amount data contained in the target document 10. It is also possible to improve the accuracy of matching each amount data contained in the target document 10.

[0061] Furthermore, since it is possible to use information (amount) that quantifies the total amount of the amount data as attribute information of the amount data, the determination unit 130 can easily grasp the numerical values ​​to be compared. Therefore, it is possible to omit the process of quantifying the total amount of the amount data to be compared in the comparison process by the determination unit 130. Therefore, it is possible to reduce the calculation process related to the comparison process.

[0062] Furthermore, as the attribute information of the amount data, it is possible to use at least one of the type of amount data, fiscal year information of the amount data, and unit type of the amount data. Therefore, it is possible to perform appropriate comparison processing using a relatively small amount of attribute information. This makes it possible to reduce the amount of calculation processing involved in the comparison processing.

[0063] The recording control unit 140 stores the tagged target document 20 in the storage unit 150. In this case, various information related to the tagged target document 20 can be stored as appropriate. For example, the tagged target document 20 may be stored in the storage unit 150 in its original format (see FIG. 7 ), or named entities and attribute information may be extracted from the tagged target document 20, and the named entities and attribute information may be associated with the tagged target document 20 (or target document 10) and stored in the storage unit 150. Alternatively, only the attribute information of named entities included in the tagged target document 20 may be stored in the storage unit 150. In this way, the various information related to the tagged target document 20 stored in the storage unit 150 can be used as appropriate when the user needs it.

[0064] The output control unit 160 outputs the tagged target document 20 from the output unit 170. In this case, various information related to the tagged target document 20 can be output as appropriate. For example, the tagged target document 20 may be displayed on the display unit 171 of the output unit 170 in its original format (see FIG. 7 ), or named entities and attribute information may be extracted from the tagged target document 20, and the named entities and attribute information may be associated with the tagged target document 20 (or target document 10) and displayed on the display unit 171 of the output unit 170. Alternatively, only the attribute information of the named entities included in the tagged target document 20 may be displayed on the display unit 171 of the output unit 170. Furthermore, various information related to the tagged target document 20 may be output as audio, or may be transmitted to another device and output from that device. In this way, various information related to the tagged target document 20 can be provided as needed by the user by displaying, outputting as audio, transmitting, etc.

[0065] [Example of notification of a named entity spelling error] Fig. 8 is a diagram showing an example of notification when a named entity in which an error has been detected in the determination process by the determination unit 130 is notified to the user. Fig. 8 shows an example in which the tagged target document 20 is displayed on the display unit 171 of the output unit 170, and the portion of the named entity in which an error has been detected in the tagged target document 20 is indicated by being surrounded by triangles E1 and E2, and an example in which audio information S1 indicating that an error has been detected is output from the audio output unit of the output unit 170. Fig. 8 also shows an example in which the portion of the named entity that is the determination target in the tagged target document 20 is surrounded by ellipses J1 to J4. Note that the determination process by the determination unit 130 is the same as the determination process described with reference to Fig. 7. Note that the error notification method shown in Fig. 8 is just an example, and other notification methods may be used.

[0066] Here, in the case of documents with a fixed format, such as securities reports, the position of each item in the text and the flow of each content are almost fixed. Therefore, workers who review target documents such as securities reports often know the location of the parts of the target document where the content to be reviewed is described. Therefore, even if tagged target document 20 is displayed as a notification screen that notifies the worker reviewing a target document such as a securities report of the review results, the worker can easily understand the content to be reviewed. Note that, if page information needs to be displayed, tagged target document 20 with page information of target document 10 added may be displayed. Similarly, if other information that needs to be displayed exists, tagged target document 20 with the necessary information added can be displayed.

[0067] It is also possible to display attribute information in a table format as the output result of the LLM. For example, it is possible to assign an ID (e.g., identification information such as a serial number) to each piece of attribute information, and display a list of attribute information associated with this ID in a table format (e.g., in serial number order). Here, because the LLM processes information incorporating surrounding information, when checking the output result of the LLM, it is important to verify the accuracy by checking the content before and after the original location. For this reason, for example, if only attribute information is displayed as the output result of the LLM, it is expected that it will be difficult to verify why the attribute information was extracted by the LLM. In contrast, in this embodiment, the tagged target document 20 containing the amount data and the attribute information to be evaluated is displayed on the display unit 171, making it possible to verify the accuracy by checking the content before and after the part to be evaluated. This makes it easy to verify why the attribute information was extracted by the LLM.

[0068] [Example of operation of information processing device] 9 is a flowchart showing an example of a determination process in information processing device 100. This determination process is executed based on a program stored in storage unit 150. This determination process is executed when target document 10 is acquired by acquisition unit 110. This determination process will be explained with appropriate reference to FIGS. 1 to 8.

[0069] FIG. 9 shows an example in which, if an error is detected in the determination process, the error is notified to the user.

[0070] In step S501 , the acquisition unit 110 acquires the target document 10 input by the user, and outputs the acquired target document 10 to the attachment unit 120 .

[0071] In step S502, the attachment unit 120 extracts named entities contained in the target document 10 output from the acquisition unit 110, and extracts attribute information related to the extracted named entities. The attachment unit 120 also performs an attachment process to tag the target document 10 with the extracted named entities and the attribute information related to them. The attachment unit 120 then outputs the tagged target document 20 that has been subjected to the attachment process to the determination unit 130, the recording control unit 140, and the output control unit 160. This attachment process is the same as the attachment process described above.

[0072] In step S503, the determination unit 130 executes a determination process to determine whether or not the named entities included in the tagged target document 20 to which attribute information was assigned in step S502 are identical. Then, the determination unit 130 outputs the result of the determination process to the recording control unit 140 and the output control unit 160. This determination process is the same as the determination process shown in FIG.

[0073] In step S504, the output control unit 160 determines whether or not a named entity error was detected by the determination process in step S503. If a named entity error is detected, the process proceeds to step S506. On the other hand, if a named entity error is not detected, the process proceeds to step S505. For example, in the example shown in FIG. 7, an error was detected in at least one of the amount data within the dotted rectangles 33 and 36 indicated by arrow C3, so the process proceeds to step S506.

[0074] In step S505, the output control unit 160 causes the output unit 170 to output information that the named entities included in the target document 10 are the same. For example, the output control unit 160 can cause the display unit 171 of the output unit 170 to display the target document 10 (or the tagged target document 20) and display information that the named entities of the target document 10 are the same. The output control unit 160 may also cause the audio output unit to output audio information that the named entities of the target document 10 are the same.

[0075] In step S506, the output control unit 160 causes the output unit 170 to output, in an identifiable manner, the named entities in which an error has been detected among the named entities contained in the target document 10. For example, as shown in Fig. 8, the output control unit 160 can cause the display unit 171 of the output unit 170 to display the tagged target document 20 and add triangles E1 and E2 to the named entities in which an error has been detected to indicate that an error has been detected. The output control unit 160 may also cause the audio output unit to output audio information S1 indicating that the named entity in the tagged target document 20 is incorrect.

[0076] In step S507, the recording control unit 140 stores the tagged target document 20 to which the attribute information has been assigned in step S502 in the storage unit 150. In this case, the recording control unit 140 may store in the storage unit 150 the tagged target document 20, the target document 10 acquired in step S501, and the attribute information (see FIG. 7) assigned to the corresponding tagged target document 20 in association with each other.

[0077] [Examples of using information processing equipment] 1 shows an example in which the acquisition process, assignment process, determination process, etc. are executed in the information processing device 100. In this case, the user terminal used by the user U1 can be used as the information processing device 100. An example of use in this case is shown in FIG.

[0078] 10 is a diagram showing an example of how the information processing device 100 is used. For example, information processing devices and electronic devices such as a personal computer, a smartphone, a tablet terminal, etc. can be used as the information processing device 100. For example, when a user U1 inputs a target document 10 into the information processing device 100, a tagged target document 20 corresponding to the target document 10 and a determination result for the tagged target document 20 are displayed on the display unit 171.

[0079] Furthermore, all or part of each process, such as the acquisition process, the assignment process, and the determination process, may be executed by another device. In this case, an information processing system is configured by the devices that execute part of each process. For example, at least part of each process can be executed by a device that user U1 can use (e.g., a smartphone, a tablet terminal, a personal computer), various information processing devices such as a server that can be connected via a predetermined network such as the Internet, and various electronic devices. For example, a device different from the user terminal used by user U1 can be used as information processing device 100. An example of use in this case is shown in FIG. 11.

[0080] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing device 100 may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0081] FIG. 11 is a diagram showing an example of use of the information processing device 100. For example, the information processing device 100 can be an information processing device such as a server or an electronic device. The user terminal 180 is, for example, an information processing device or an electronic device such as a personal computer, a smartphone, or a tablet terminal. The network NW1 is a network such as a public line network or the Internet. The user terminal 180 and the information processing device 100 are connected to the network NW1 by a communication method using wireless communication or a communication method using wired communication, or by both methods.

[0082] For example, when user U1 inputs target document 10 into user terminal 180, user terminal 180 transmits the target document 10 to information processing device 100. When information processing device 100 receives the target document 10, it performs an assignment process, a determination process, etc. on the target document 10 and transmits the processing results to user terminal 180. User terminal 180 displays the processing results, that is, a tagged target document 20 corresponding to the target document 10 and the determination results for the tagged target document 20, on display unit 171.

[0083] [Example of pre-processing and post-processing] The above has shown an example of performing the assignment process, determination process, etc. using the input target document 10. Here, when the target document 10 contains a large amount of text information, a large amount of table information, etc., it is possible to reduce the computational load related to the calculation processes such as the assignment process, determination process, etc. by performing a predetermined preprocessing. Therefore, Fig. 12 shows an example of performing the assignment process, determination process, etc. after performing the preprocessing on the target information to be processed, and then performing the postprocessing after performing each of these processes.

[0084] [Configuration example of information processing device] Fig. 12 is a block diagram showing an example of the functional configuration of the information processing device 400. Note that the information processing device 400 is a partial modification of the information processing device 100 shown in Fig. 1, with the addition of a pre-processing unit 410 and a post-processing unit 420, and apart from these additions, the information processing device 400 is common to the information processing device 100. For this reason, parts common to the information processing device 100 are assigned the same reference numerals as those in the information processing device 100, and descriptions thereof will be omitted.

[0085] The preprocessing unit 410 performs predetermined preprocessing on the target information output from the acquisition unit 110, and outputs the target information after preprocessing to the annotation unit 120. For example, if the target information includes both text information and table information, the text information and table information can be separated, and the combination of the separated text information and table information can be used as the target information for annotation processing, determination processing, etc. For example, in the case of an HTML file, a table tag is attached to the beginning of the table information, making it possible to recognize the table information. In this way, known segmentation processing can be used to segment the text information and table information.

[0086] Furthermore, for example, when dividing text information, it is possible to divide it based on semantic chunks. In other words, if a sentence contains a meaningful structure, it is preferable to divide it without destroying that structure. For example, it is possible to divide it based on paragraphs, chapters, or the like of the sentence. For dividing these sentences, known language processing can be used.

[0087] Furthermore, for example, in the case of an HTML file, attribute information included in the HTML data (for example, attribute information related to font type, size, color, etc.) is considered to be very long and difficult to handle. Therefore, in the case of an HTML file, the preprocessing unit 410 may perform preprocessing to temporarily delete unnecessary parts (attribute information included in the HTML data).

[0088] The post-processing unit 420 performs predetermined post-processing on the tagged target information output from the tagging unit 120, and outputs the tagged target information after the post-processing (e.g., tagged target document 20 (see FIG. 4)) to the determination unit 130, the recording control unit 140, and the output control unit 160. The post-processing unit 420 performs post-processing corresponding to the pre-processing performed by the pre-processing unit 410. For example, if the pre-processing unit 410 performs a division process to divide text information and table information, the post-processing unit 420 performs a combination process to combine the divided text information and table information. For example, if the pre-processing unit 410 performs a division process to divide text information, the post-processing unit 420 performs a combination process to combine the divided text information. For example, if the pre-processing unit 410 performs a pre-processing to delete unnecessary parts (e.g., attribute information included in HTML data), the post-processing unit 420 performs a process to restore the deleted parts. For these restoration processes, known processing methods can be used.

[0089] [Examples of using other named entities] The above example shows how to extract and use monetary expressions (monetary data) as named entities. However, as mentioned above, it is also possible to extract other named entities and determine errors. Therefore, the following example shows how to extract other named entities and perform the determination process.

[0090] [Example using organization name] First, an example in which an organization name (ORG) is extracted and used as a named entity will be described.

[0091] For example, the assigning unit 120 assigns the following to the data of the organization (organization data) included in the target document 10: <org>It is possible to add a tag called "ORG tag". For example, <org type="String" name="String" en_name="String"> [Organization Data]< / org> " can be used as the tag format (i.e., the expression within " "). In this tag format, [organization data] is the extracted organization name. For example, in the case of ABC Co., Ltd., "ABC Co., Ltd." or "(ABC Co., Ltd.)" is stored in [organization data]. In addition, the following attribute information (O1) to (O3) can be assigned as attributes of the ORG tag. (O1)type (O2) name (O3)en_name

[0092] (O1) type is attribute information that indicates the type of organizational data. In other words, it is attribute information that indicates what type of organization the organizational data is. For example, in the case of ABC Co., Ltd., it is possible to assign "KK". Note that KK is information that indicates a corporation.

[0093] (O2) name is attribute information indicating the name of the organization data. For example, in the case of ABC Co., Ltd., it is possible to assign "ABC."

[0094] (O3) en_name is attribute information indicating the English notation of the organization data. For example, in the case of ABC Co., Ltd., it is possible to assign "ABC".

[0095] The attribute information of the ORG tag shown here is an example, and some of it may be omitted as necessary, or other attribute information may be added.

[0096] Furthermore, when performing error determination processing, the determination unit 130 identifies organizational data of the same type based on the attribute information of the organizational data. Specifically, the determination unit 130 identifies multiple organizational data whose attribute information (e.g., (O1) type, (O2) name, (O3) en_name) matches. The determination unit 130 then compares the multiple organizational data identified as the same type and determines whether these organizational data are the same. For example, the determination unit 130 compares the [organization data] of the organizational data identified as the same type and determines whether they are the same.

[0097] [Example of using a person's name] First, an example in which a person's name (PERSON) is extracted and used as a named entity will be described.

[0098] For example, the assigning unit 120 assigns the following to the data of a person's name (personal name data) included in the target document 10: <person>It is possible to add a tag called "PERSON tag" to a person. For example, <person family_name="String" given_name="String" role="String" en_name="String"> [Personal name data]< / person> " can be used as the tag format (i.e., the expression within " "). In this tag format, [person's name data] is the extracted person's name. In addition, the following attribute information (P1) to (P4) can be assigned as attributes of the PERSON tag. (P1)family_name (P2)given_name (P3)role (P4)en_name

[0099] (P1) family_name is attribute information indicating the family name and surname for a person's name. For example, in the case of Yamada Taro, "Yamada" can be assigned.

[0100] (P2) given_name is attribute information indicating the name of a person. For example, in the case of Yamada Taro, it is possible to give "Taro."

[0101] (P3) role is attribute information that indicates the role of a person's name. For example, in the case of the CEO of ABC Co., Ltd., it is possible to assign "CEO." Also, in the case of a section manager of ABC Co., Ltd., it is possible to assign "Section Manager."

[0102] (P4) en_name is attribute information indicating the English spelling of personal name data. For example, in the case of Yamada Taro, it is possible to assign "TARO YAMADA." It is preferable to separate en_name into first and last names.

[0103] The attribute information of the PERSON tag shown here is an example, and some of it may be omitted as necessary, or other attribute information may be added.

[0104] Furthermore, when performing error determination processing, the determination unit 130 identifies personal name data of the same type based on the attribute information of the personal name data. Specifically, the determination unit 130 identifies multiple personal name data whose attribute information (e.g., (P1) family_name, (P2) given_name, (P3) role, (P4) en_name) matches. The determination unit 130 then compares the multiple personal name data identified as the same type and determines whether these personal name data are identical. For example, the determination unit 130 compares the [personal name data] of the personal name data identified as the same type and determines whether they are identical. Details of the process for determining the identity of personal name data will be described later.

[0105] [About named entities] The above examples show how monetary expressions (monetary data), organization names (ORG), and people's names (PERSON) are extracted and used as named entities. However, as mentioned above, it is also possible to extract and use other named entities. Therefore, below we will explain named entities classified from the perspective of information processing technology.

[0106] FIG. 13 is a diagram showing an example of classification contents of named entities (NE) from the viewpoint of information processing technology. This classification contents includes those described in the MUC. Furthermore, "Other" is set assuming that domain-specific expressions will be included in named entities in the future. Note that the domain shown in this embodiment is a term meaning each field, each area, etc.

[0107] 13 shows an example in which named entities are broadly classified (major classifications) into numerical values, names, and time (Time Exp), and then further classified (minor classifications) into numerical values, names, and time. Named entities corresponding to each of these classifications will be described in detail with reference to FIGS. 14 to 16.

[0108] Fig. 14 is a diagram showing an example of attribute information related to a named entity. The classifications shown in Fig. 14 correspond to the major classifications shown in Fig. 13. Fig. 14 shows an example in which a rectangle in which corresponding attribute information is written is placed in the lower right part of a rectangle in which a named entity is written. Similarly, Fig. 15 and subsequent figures also show examples in which a rectangle for attribute information is placed in the lower right part of a rectangle for a named entity.

[0109] FIG. 14 shows an example in which id, type, and ENTITY are used as the attribute information that is the basis of a named entity.

[0110] An ID is identification information for identifying named entities contained in the target information. For example, a serial number (1, 2, 3, ..., etc.) can be used as an ID. For example, in one target information (target document), it is possible to assign serial numbers to named entities in the order they are written. Note that IDs can be used or omitted as needed.

[0111] "type" is attribute information indicating the type (classification) of a named entity. That is, it is attribute information indicating what type of expression the corresponding named entity is. Details of "type" will be explained in FIG. 15 and subsequent figures.

[0112] ENTITY is information that indicates the content of the corresponding named entity. For example, ENTITY means an independent and unique object (also called, for example, substance, existence, entity, essence, or main body) that can be distinguished from other named entities. ENTITY can also be used, for example, when matching multiple named entities. In other words, ENTITY is used when verifying the identity of multiple named entities. Note that ENTITY is set for each named entity, so it is a variable. For this reason, ENTITY is written in uppercase in the rectangle located at the bottom right of the named entity (within the rectangle) in Figure 14.

[0113] As shown in the bottom rectangle of Figure 14, for example, when targeting a named entity representing a number, ENTITY can be set to "amount." When targeting a named entity with a name, ENTITY can be set to "name." When targeting a named entity representing a time, ENTITY can be set to "standard time."

[0114] [Attribute information for each named entity] As mentioned above, a number is a named entity. It is possible to use id, type, and ENTITY as attribute information for this number. However, in this case, ENTITY is used as the attribute information for matching numbers, and amount is used as attribute information for matching numbers. In other words, it is possible to use id, type, and amount as attribute information for a number. As mentioned above, amount is attribute information that indicates a number. For example, if a number is monetary data, and the monetary data is "7,118,000,000 yen," then "7,118,000,000" can be used as the amount.

[0115] As described above, a general proper noun is a named entity. In FIG. 14, a general proper noun will be described as "with name." id, type, and ENTITY can be used as attribute information for this general proper noun (with name). In this case, however, ENTITY is set as name as attribute information for matching general proper nouns. That is, the attribute information for a general proper noun can be id, type, and name. As described above, name is attribute information indicating a general proper noun. For example, when a general proper noun is a name, if the name is "Yamada Taro," then "Yamada Taro" can be used as the name.

[0116] As described above, a time expression is a named entity. In FIG. 14, the time expression will be described as "time." id, type, and ENTITY can be used as attribute information of this time expression. In this case, however, ENTITY is set as standard time (st) as attribute information for matching the time expression. That is, the attribute information of the time expression can be id, type, and st. As described above, this standard time is attribute information indicating the time expression.

[0117] [Numerical attribute information] FIG. 15 is a diagram showing an example of attribute information related to numerical values ​​(named entities). Here, as shown in FIG. 14, an example is shown in which id, type, and amount are used as basic attribute information for numerical values. FIG. 15 also shows an example in which numerical values ​​(named entities) are classified (subclassified) into three categories: MONEY, PERCENTAGE, and others. In other words, this shows an example in which MONEY, PERCENTAGE, and others belong to a lower hierarchy of numerical values. Note that MONEY has the same content as the amount data described above.

[0118] [About MONEY attribute information] As mentioned above, MONEY (amount data) is a named entity. id, type, and amount can be used as attribute information of this MONEY. However, as mentioned above, unit and currency can also be added as attributes of MONEY. In this case, the attribute information of MONEY can be id, type, amount, unit, and currency.

[0119] As described above, unit is attribute information indicating the unit of the monetary amount data. For example, "yen" or "million yen" is assigned as the unit.

[0120] Currency is attribute information that indicates the type of currency. For example, a currency code is assigned as currency. For example, for Japanese yen, "JPY" is assigned as currency.

[0121] [About PERCENTAGE and other attribute information] PERCENTAGE and others, which correspond to a lower level (subcategory) of numerical values, are named entities. id, type, and amount can be used as attribute information for PERCENTAGE and others. Note that for PERCENTAGE and others, if additional attributes are required to confirm identity for each document or domain, attribute information can be expanded or added as appropriate.

[0122] Specifically, since PERCENTAGE is a numerical value, id, type, and amount can be used as attribute information for PERCENTAGE. Also, as mentioned above, if additional attributes are required to confirm identity for each document or domain, attribute information can be expanded or added as appropriate.

[0123] "Other" in Figure 15 means a numeric expression other than MONEY or PERCENTAGE. Therefore, id, type, and amount can be used as attribute information for "Other." Also, as mentioned above, if additional attributes are needed to confirm identity for each document or domain, attribute information can be appropriately expanded or added, or type names (for example, MONEY or PERCENTAGE) can be appropriately expanded or added.

[0124] [About MONEY attribute information] FIG. 16 is a diagram showing an example of attribute information related to MONEY (named entity). Here, as shown in FIG. 15, an example is shown in which id, type, amount, unit, and currency are used as basic attribute information of MONEY. Also, FIG. 16 shows an example in which MONEY is classified (subclassified) into two categories: MONEY_IR and MONEY_Other. In other words, this shows an example in which MONEY_IR and MONEY_Other belong to a lower layer than MONEY.

[0125] [MONEY_IR attribute information] MONEY_IR is information related to IR (Investor Relations) within MONEY (amount data). As such, MONEY_IR (amount data) is a proper expression because it is included in MONEY (amount data). IR is also known as "investor relations."

[0126] Basic attribute information (id, type, amount, unit, currency) can be used as attribute information for MONEY_IR. However, as described above, fiscal, segment, and up_down may be added as attribute information required within an IR document. In this case, the attribute information for MONEY_IR (amount data) can be id, type, amount, unit, currency, fiscal, segment, and up_down. Note that if additional attributes are required to confirm identity for each document or domain, the attribute information can be expanded or added as appropriate. The following shows an example of adding attribute information for fiscal, segment, and up_down.

[0127] As described above, fiscal is attribute information that indicates the fiscal year of the amount data. For example, "previous fiscal year," "current fiscal year," "current consolidated fiscal year," etc. can be assigned as fiscal.

[0128] As mentioned above, "segment" indicates which business segment within the reporting segment. For example, "consulting business," "business solutions business," "education business," etc. can be assigned as a segment.

[0129] As described above, up_down is attribute information that indicates in detail what type of amount it is. For example, if the amount data "sales" classified as "type" increases, "up" or "increase" can be assigned as up_down. Also, for example, if the amount data "sales" classified as "type" decreases, "down" or "decrease" can be assigned.

[0130] [MONEY_Other attribute information] MONEY_Other is information about the amount data other than IR within MONEY (amount data). As such, MONEY_Other (amount data) is an expression included in MONEY (amount data), and is therefore a named entity. The basic attribute information of MONEY (amount data) (id, type, amount, unit, currency) can be used as attribute information for this MONEY_Other. Note that attribute information required for domains other than IR can be expanded and added as appropriate for each domain. For example, if additional attributes are required to confirm identity for each document or domain, attribute information can be expanded and added as appropriate.

[0131] These are representative examples for ease of explanation, and other attribute information that may be assumed within the IR domain includes information about stocks, information about numbers (for example, number of employees, number of affiliated companies, number of businesses), etc. Other attribute information may also be adopted.

[0132] [Example of the relationship between input data and output data] 17 is a diagram showing an example of the relationship between input data 600 input to the LLM 121 (see FIG. 4) and output data 601 output from the LLM 121. The input data 600 corresponds to a portion of a sentence contained in the target information (e.g., a securities report). The output data 601 shows an example in which numerical values ​​(MONEY and PERCENTAGE (see FIG. 15)) are extracted as named entities.

[0133] For example, if the target information is written only in Japanese yen, Japanese yen can be identified without using currency="JPY". For this reason, the attribute information "currency" can be used only when dealing with multiple exchange rate information. Similarly, the attribute information "up_down" can be used only when comparing the current period with the previous period. Also, for example, instead of using the attribute information "up_down", the attribute information "type" can be set to "sales increase".

[0134] Also, Fig. 17 shows an example in which id is omitted. Also, Fig. 17 shows an example in which type, segment, fiscal, unit, amount, and up_down are used as attribute names of MONEY that are acceptable to the LLM 121. Also, Fig. 17 shows an example in which type and amount are used as attribute names of PERCENTAGE that are acceptable to the LLM 121.

[0135] 5, we will now explain the instruction information (prompt) to be input to ChatGPT, GPT-4, GPT-4 Turbo, or GPT-4o when using these LLMs as the LLM 121. It is preferable to use attribute information required for named entities included in the target information as attribute names that the LLM 121 can accept.

[0136] 5, for example, assume that a user creates sample data including one (or more) sentences before tagging and sample sentences with tags added to the sentences. In this case, the sample data for the tags is inserted into a prompt as a tagging example, and a prompt including an instruction to tag target information is input to the LLM together with the target information. The corresponding output data (e.g., the target document, MONEY, and PERCENTAGE attribute information) can be used as tagged target information (e.g., output data 601).

[0137] For example, assume that the sample data is a sentence included in input data 600 shown in Fig. 17(A) and a sentence and attribute information included in output data 601 shown in Fig. 17(B). In this case, the sentence included in input data 600 and the sentence included in output data 601 are inserted into a prompt as tagging examples, and a prompt including an instruction to tag target information is input to LLM 121 together with the target information, and the corresponding output data (e.g., the target document and attribute information of MONEY and PERCENTAGE) can be used as tagged target information (e.g., output data 601).

[0138] Also, for example, it is possible to extract MONEY and PERCENTAGE as named entities, extract type, segment, fiscal, unit, amount, and up_down as attribute information of that MONEY, and input a prompt to the LLM to extract type and amount as attribute information of that PERCENTAGE, and use the corresponding output data (e.g., the target document, attribute information of MONEY, and PERCENTAGE) as tagged target information (e.g., output data 601). Note that these prompts are merely examples and are not limiting. Other prompts that can extract MONEY and PERCENTAGE included in the input data 600 and the attribute information of that MONEY and PERCENTAGE may also be used.

[0139] [About attribute information with names] Fig. 18 is a diagram showing an example of attribute information related to named entities (named entities). Here, as shown in Fig. 14, an example is shown in which id, type, and name are used as basic attribute information for named entities. Fig. 18 also shows an example in which named entities (named entities) are classified (subclassified) into four categories: PERSON, ORG, LOC, and others. In other words, this shows an example in which PERSON, ORG, LOC, and others belong to a lower level than named entities.

[0140] As mentioned above, named entities are named entities. It is preferable to use only attribute information that can be understood from the context (meaning the context, contextual relationship, circumstances, background, and situation of the target information) for named entities. For example, it is linguistically known that the most attributes of a given object (e.g., a person, organization, or place name) are described when it first appears in a document. That is, due to the characteristics of language, when people write a document, they first summarize and introduce all the necessary information (e.g., detailed information about the entity (proper noun, element)) about the given object (e.g., a person, organization, or place name). After that, they often only describe the name of the object, without providing any detailed information about the object. In other words, when a given object (e.g., a person, organization, or place name) is mentioned for the second or subsequent time, detailed information about the object (e.g., a person, organization, or place name) is often omitted because it is redundant.

[0141] As such, the first named entity has a lot of attribute information, and the same named entity that appears in subsequent contexts is often expressed by name only, so attribute information with a name that appears from the second time onwards can basically be limited to id and name.

[0142] [About PERSON (person's name) attribute information] FIG. 19 is a diagram showing an example of attribute information related to PERSON (person's name). As described above, PERSON (person's name) is a named entity. Furthermore, since PERSON (person's name) is a name, it has the attributes id, type, and name. However, in this example, the attribute information "type" is determined to be an inappropriate attribute for PERSON and is therefore omitted.

[0143] As attribute information of PERSON, it is possible to use id, name, title, given_name, family_name, and dob (date of birth) in addition to title, given_name, family_name, and dob. In addition, other attribute information (for example, nickname (see Figures 20 and 21)) may be used.

[0144] "name" is attribute information indicating the name corresponding to PERSON (person's name). For example, in the case of Yamada Taro (Yamada Taro's hiragana spelling), "Yamada_Taro" can be assigned. For example, in the case of Paul Samantha (English: Paul Samantha) or Sam Pauli (English: Sum Pauli's nickname), "paul_samantha" can be assigned. By placing an "_ (underscore)" between the characters indicating the family name and the characters indicating the given name, it is possible to clearly distinguish between the characters indicating the family name and the characters indicating the given name. It is preferable to use a prompt for "name" that instructs the system to use unique notation for identical people in the same target information. This can improve the accuracy of determining the identity of names, as described below. It is also preferable to use a prompt for other names (e.g., "CITY") that instructs the system to use unique notation for identical people in the same target information.

[0145] A title is attribute information that indicates the title, role, position, etc. of a PERSON (person's name). For example, Dr., Prof., CEO, etc. are assigned as attribute information of a title. Furthermore, when an organizational position is stored in the attribute information of a title, it is possible to add information about the organization name in which the position is held by adding 'org' as new attribute information. Note that instead of adding new attribute information 'org', the organization name may be included in the attribute information of a title. Furthermore, a role may be used instead of a title (see Figures 20 and 21).

[0146] As described above, given_name is attribute information that indicates the name of a PERSON (person's name). For example, in the case of Yamada Taro, "Taro" can be given.

[0147] Family_name is attribute information that indicates the family name and surname of a person (person's name). For example, in the case of Yamada Taro, "Yamada" can be assigned.

[0148] "dob" is attribute information indicating the date of birth of a person (person's name). For example, date information in the Gregorian calendar is assigned as attribute information of "dob."

[0149] [About PERSON_Wiki attribute information] PERSON_Wiki (Wikipedia (registered trademark)) is a named entity belonging to PERSON, and therefore has the attributes id, type, name, title, given_name, family_name, and dob. PERSON_Wiki is also a Wikipedia tag for information about people. For this reason, when attribute information (for example, information that can be understood from the context) such as born_loc (place of birth), education, and dod (date of death: for deceased people, the date of death) first appears in the target information, PERSON_Wiki is extracted and assigned as the person's attribute information.

[0150] [PERSON_Other attribute information] Since PERSON_Other is a named entity that belongs to PERSON, it has the attributes id, name, title, given_name, family_name, and dob. However, if other attribute information is required when performing matching processing, it is possible to add other required attribute information as appropriate.

[0151] [Example of the relationship between input data and output data] 20 to 22 are diagrams showing examples of the relationship between input data 610, 620, and 630 input to the LLM 121 (see FIG. 4) and output data 611, 621, and 631 output from the LLM 121. FIG.

[0152] 20 corresponds to a portion of a sentence written in English contained in target information (for example, a document introducing a living person). Output data 611 shows an example in which PERSON (belonging to the category with a name (see FIGS. 18 and 19)) is extracted as a named entity.

[0153] FIG. 20 also shows an example in which id, dob, nickname, role, and name are used as attribute names of PERSON that the LLM 121 accepts.

[0154] Input data 620 shown in Fig. 21 corresponds to a portion of a sentence written in Japanese that is included in target information (for example, a document introducing a living person). The contents of input data 610 and input data 620 are the same. Output data 621 shows an example in which PERSON (belonging to the category with name (see Figs. 18 and 19)) is extracted as a named entity.

[0155] 21 shows an example in which id, dob, en_writing, nationality, role, and name are used as attribute names of PERSON that are acceptable to the LLM 121. Note that en_writing is an attribute that indicates English writing, and nationality is an attribute that indicates nationality.

[0156] Input data 630 shown in Fig. 22 corresponds to a portion of a sentence written in Japanese that is included in target information (for example, a document introducing a historical figure). Output data 631 shows an example in which PERSON (which belongs to the named entity category (see Figs. 18 and 19)) is extracted as a named entity.

[0157] 22 shows an example in which id, name, dob, dod, nationality, and role are used as PERSON attribute names that are acceptable to the LLM 121. Although Fig. 20 to Fig. 22 show examples in which PERSON attribute names are arranged in different orders, it is preferable that the attribute names in the target information to be judged be arranged in the same order.

[0158] 5, we will now explain the instruction information (prompt) to be input to ChatGPT, GPT-4, GPT-4 Turbo, or GPT-4o when using these LLMs as the LLM 121. It is preferable to use attribute information required for named entities included in the target information as attribute names that the LLM 121 can accept.

[0159] 5, for example, assume that a user creates sample data including one (or more) sentences before tagging and sample sentences with tags added to the sentences. In this case, the sample data for the tags is inserted into a prompt as a tagging example, and a prompt including an instruction to tag target information is input to the LLM together with the target information. The corresponding output data (e.g., target information, attribute information of a person) can be used as tagged target information (e.g., output data 611, 621, 631).

[0160] For example, assume that the sample data is a sentence included in input data 610 shown in Figure 20(A) and a sentence and attribute information included in output data 611 shown in Figure 20(B). In this case, the sentence included in input data 610 and the sentence included in output data 611 are inserted into a prompt as tagging examples, and a prompt including an instruction to tag target information is input to LLM 121 along with the target information, and the corresponding output data (e.g., a target document, attribute information of a person) can be used as tagged target information (e.g., output data 611). The same applies to the examples shown in Figures 21 and 22.

[0161] Also, for example, a prompt to extract PERSON as a named entity and extract id, dob, nickname, role, and name as attribute information of the extracted PERSON can be input to the LLM, and the corresponding output data (target information, attribute information of PERSON) can be used as tagged target information (e.g., output data 611). The same applies to the examples shown in FIGS. 21 and 22. Note that these prompts are merely examples and are not limiting. Other prompts that can extract PERSON included in the input data 610 and the attribute information of that PERSON may also be used.

[0162] [About ORG (organization name) attribute information] FIG. 23 is a diagram showing an example of attribute information related to ORG (organization name). As mentioned above, ORG (organization name) is a unique entity. Furthermore, since ORG (organization name) is a name, it has the attributes id, type, and name. FIG. 23 also shows an example in which ORG (organization name) is classified (subclassified) into four categories: COMPANY, UNIVERSITY, PUBLIC SECTOR, and others. In other words, this shows an example in which COMPANY, UNIVERSITY, PUBLIC SECTOR, and others belong to the lower hierarchy of ORG (organization name).

[0163] As attribute information for ORG (organization name), formal_name, abb (abbreviation), founded, and address can be added, and id, type, name, formal_name, abb, founded, founder, and address can be used.

[0164] The formal_name is attribute information indicating the title, role, position, etc. of ORG (organization name). For example, "ABC Co., Ltd." is given as attribute information of the formal_name.

[0165] abb is attribute information indicating the abbreviation of ORG (organization name). For example, in the case of "ABC Co., Ltd.", "ABC" or "ABC" is given as attribute information of abb.

[0166] "founded" is attribute information that indicates the founding date of ORG (organization name). For example, if "ABC Co., Ltd." was founded on January 1, 2000, then "January 1, 2000" is assigned as the "founded" attribute information. "founder" is attribute information that indicates the person (founder) who founded ORG (organization name). For example, if "ABC Co., Ltd." was founded by Ota Jiro, then "Ota Jiro" is assigned as the "founder" attribute information.

[0167] "Address" is attribute information that indicates the address of ORG (organization name). For example, in the case of "ABC Co., Ltd.", which was established in Kita-ku, Sapporo, Hokkaido, etc., "Kita-ku, Sapporo, Hokkaido, etc." is assigned as attribute information of Address.

[0168] [Company attribute information] Since COMPANY is the name of an organization, it has the attributes id, type, name, formal_name, abb, founded, and address. Figure 23 also shows an example in which CEO and industry are added as attribute information for COMPANY. Therefore, id, type, name, formal_name, abb, founded, address, CEO, and industry can be used as attribute information for COMPANY. Note that these attribute information items may be added or deleted as appropriate for each domain. For example, information listed in company information (e.g., number of employees, capital) may be used as attribute information. For example, in the case of IR data for a company, a point on the time axis is determined, so capital, number of employees, etc. can be added as attribute information.

[0169] [About UNIVERSITY attribute information] Since UNIVERSITY is an organization name, it has the attributes id, type, name, formal_name, abb, founded, and address. Also, Figure 23 shows an example in which president (chairman, chancellor) is added as attribute information for UNIVERSITY. Therefore, id, type, name, formal_name, abb, founded, address, and president can be used as attribute information for UNIVERSITY. Note that these attribute information may be added or deleted as appropriate for each domain.

[0170] [About PUBLIC SECTOR attribute information] Since the PUBLIC SECTOR is the name of an organization, it has the attributes id, type, name, formal_name, abb, founded, and address. Also, FIG. 23 shows an example in which head (head of a public institution) is added as attribute information of the PUBLIC SECTOR. Therefore, it is possible to use id, type, name, formal_name, abb, founded, address, and head as attribute information of the PUBLIC SECTOR. Note that these attribute information may be added or deleted as appropriate for each domain.

[0171] [Other attribute information of ORG] Others of ORG (organization name) have the basic attribute information of ORG (organization name). That is, Others of ORG (organization name) have the attributes id, type, name, formal_name, abb, founded, and address. Note that new organizations and their attributes may be defined as appropriate depending on the target information (target document). Furthermore, these attribute information may be added or deleted as appropriate for each domain.

[0172] [Example of the relationship between input data and output data] FIG. 24 is a diagram showing an example of the relationship between input data 640 input to the LLM 121 (see FIG. 4) and output data 641 output from the LLM 121.

[0173] The input data 640 corresponds to a portion of a sentence contained in the target information (for example, a document introducing an existing company). The output data 641 shows an example in which PERSON (belonging to a named entity (see FIGS. 18 and 19)) and COMPANY (belonging to an ORG (organization) also having a name (see FIGS. 18, 19, and 23)) are extracted as named entities.

[0174] 24 also shows an example in which "id, type, formal_name, en_name, founder, industry, name," "id, name," etc. are used as COMPANY attribute names that are acceptable to the LLM 121. Also, an example in which id and name are used as PERSON attribute names that are acceptable to the LLM 121 is shown. Note that FIG. 24 also shows an example in which attribute names for which attribute information cannot be obtained are omitted.

[0175] 5, we will now explain the instruction information (prompt) to be input to ChatGPT, GPT-4, GPT-4 Turbo, or GPT-4o when using these LLMs as the LLM 121. It is preferable to use attribute information required for named entities included in the target information as attribute names that the LLM 121 can accept.

[0176] 5, for example, assume that a user creates sample data including one (or more) sentences before tagging and sample sentences with tags added to those sentences. In this case, the sample data for the tags is inserted into a prompt as a tagging example, and a prompt including an instruction to tag target information is input to the LLM together with the target information. The corresponding output data (e.g., the target information, attribute information of the PERSON and COMPANY) can be used as tagged target information (e.g., output data 641).

[0177] For example, assume that the sample data is a sentence included in input data 640 shown in Figure 24(A) and a sentence and attribute information included in output data 641 shown in Figure 24(B). In this case, the sentence included in input data 640 and the sentence included in output data 641 are inserted into a prompt as tagging examples, and a prompt including an instruction to tag target information is input to LLM 121 together with the target information, and the corresponding output data (e.g., the target document, attribute information of PERSON and COMPANY) can be used as tagged target information (e.g., output data 641).

[0178] Also, for example, it is possible to extract COMPANY as a named entity, and input a prompt to the LLM to extract "id, type, formal_name, en_name, founder, industry, name" or "id, name" as attribute information of the extracted COMPANY, and use the corresponding output data (target information, attribute information of COMPANY) as tagged target information (e.g., output data 641). Also, for example, it is possible to extract PERSON as a named entity, and input a prompt to the LLM to extract "id, name" as attribute information of the extracted PERSON, and use the corresponding output data (target information, attribute information of PERSON) as tagged target information (e.g., output data 641). Note that these prompts are merely examples and are not limiting. Other prompts that can extract COMPANY and PERSON included in the input data 610 and the attribute information of the COMPANY and PERSON may also be used.

[0179] [About LOC (place name) attribute information] FIG. 25 is a diagram showing an example of attribute information related to LOC (place name). As described above, LOC (place name) is a name, and therefore has attributes of id, type, and name. Note that these pieces of attribute information may be added or deleted as appropriate for each domain. FIG. 25 also shows an example in which LOC (place name) is classified (subclassified) into four categories: COUNTRY, MOUNTAIN, RIVER, and others. In other words, this shows an example in which COUNTRY, MOUNTAIN, RIVER, and others belong to the lower layer of LOC (place name).

[0180] [Country (place name) attribute information] FIG. 26 is a diagram showing an example of attribute information related to COUNTRY (place name). As described above, COUNTRY (place name) is a name, and therefore has attributes of id, type, and name. These pieces of attribute information may be added or deleted as appropriate for each domain. FIG. 26 also shows an example in which COUNTRY (place name) is classified (subclassified) into two categories: REGION (with CITY belonging to a lower level) and others. In other words, this shows an example in which REGION, CITY, and others belong to a lower level of COUNTRY (place name).

[0181] For example, in Japan, the REGION type can be "Prefecture". Also, in the United States, the REGION type can be "State". Also, in Japan, the CITY type can be "Ward, City, Town, Village". Also, in the United States, the CITY type can be "County". For example, if "Big Apple" is used to write New York, then " <loc id="”0001”," type="”city”" name="”new_york”"> Big Apple< / loc> This makes it possible to improve the accuracy of the matching process.

[0182] In addition, the postal code used in each region can identify a unique place in the world. Therefore, by adding "postal_code" as attribute information of CITY, it becomes possible to easily identify the place name.

[0183] [About time attribute information] FIG. 27 is a diagram showing an example of attribute information related to time. As described above, time is a proper noun, and therefore has attributes of id, type, and standard time. These attribute information may be added or deleted as appropriate for each domain. FIG. 27 also shows an example in which time is classified (subclassified) into two categories: DATE and TIME. In other words, this shows an example in which DATE and TIME belong to a lower layer of time.

[0184] [About DATE attribute information] FIG. 28 is a diagram showing an example of attribute information related to DATE. As described above, DATE is a proper noun, and therefore has attributes of id, type, and standard time. Note that these attribute information may be added or deleted as appropriate for each domain. FIG. 28 also shows an example in which DATE is classified (subclassified) into four categories: YEAR, MONTH, DAY, and DOW. In other words, this shows an example in which YEAR, MONTH, DAY, and DOW belong to the lower hierarchical level of DATE. Note that DOW (Day of Week) means the day of the week.

[0185] [About TIME attribute information] FIG. 29 is a diagram showing an example of attribute information related to TIME. As described above, TIME is a proper noun, and therefore has the attributes id, type, and standard time. Note that these pieces of attribute information may be added or deleted as appropriate for each domain. FIG. 29 also shows an example in which TIME is classified (subclassified) into three categories: HOUR, MINUTES, and SECONDS. In other words, this shows an example in which HOUR, MINUTES, and SECONDS belong to the lower layer of TIME. Note that the time used in each region is ultimately calculated based on standard time. Therefore, by using standard time as the attribute information for each of TIME, HOUR, MINUTES, and SECONDS, it becomes possible to easily match dates, times, etc.

[0186] [Method for determining the identity of named entities] Here, a method for determining the identity of named entities will be described. Specifically, by matching multiple named entities included in the target information, it is possible to determine the identity of those named entities.

[0187] First, named entities contained in the target information are extracted, and their attribute information is extracted. Then, from the extracted attribute information, named entities (or named entities presumed to be identical) with matching preset attribute information (e.g., type, other attribute information) are extracted. Then, the ENTITY of the attribute information of multiple named entities extracted after being determined to match (or presumed to be identical) with the preset attribute information is compared, and if the ENTITY to be compared matches, the compared named entities are determined to be identical. On the other hand, if the ENTITY to be compared does not match, the compared named entities are determined to be erroneous. For example, if the named entity is monetary amount data, the process is similar to the example shown in FIG. 7, etc.

[0188] 20 to 22 and 24, if the named entity is a PERSON (person's name), a named entity with matching preset attribute information (e.g., "dob" or other attribute information) is extracted from the extracted attribute information. Then, the "name" attribute information of multiple extracted personal names that are determined to match (or are presumed to be identical) is compared, and if the names to be compared match, the named entities to be compared are determined to be identical. On the other hand, if the names to be compared do not match, the named entities to be compared are determined to be in error. For example, as shown in FIGS. 20 and 21, the name "paul_samantha" is assigned to Paul Samantha, Sam Pauli, and Sum Pauli, and therefore it is possible to determine that these names belong to the same person (Paul Samantha). Also, for example, as shown in FIG. 22, the name "Yamada_Taro" is given to Yamada Taro and Yamada Taro, so it is possible to determine that these names belong to the same person (Yamada Taro).

[0189] For example, if the named entity is a city, the system extracts named entities with matching preset attribute information (e.g., type, postal code, and other attribute information) from the extracted attribute information. Then, the system compares the names of multiple city attribute information items extracted because the preset attribute information is determined to match (or is presumed to be identical). If the names match, the system determines that the named entities are the same. On the other hand, if the names do not match, the system determines that the named entities are erroneous. For example, the Italian city of "Venice" could be written as "Venezia," "Venice," "Venice," or "Venice." Even in such cases, by storing a unified Italian spelling of "Venezia" (or English spelling of "Venice") in the name, it is possible to determine that the spellings "Venezia," "Venice," "Venice," and "Venice" refer to the same city (Venezia).

[0190] It is also conceivable that different names may be assigned to the same named entity. In this case, the degree of match (or similarity) of attribute information other than the name may be determined, and named entities whose degree of match is equal to or exceeds a certain standard may be treated as possibly being the same named entity. For example, it is possible to notify the user that different names may be assigned but that they may be the same named entity. For example, as in the example shown in FIG. 8, it is possible to assign some information (for example, a triangle mark) to named entities that may be the same but have different names assigned.

[0191] [Example of using multiple output results] However, it is known that currently, there is a problem of inconsistency in the output results output from LLMs. For example, even if the same input data and prompts are input into an LLM multiple times, different data may be output as the output result. Therefore, it is important to input the same input data and prompts into an LLM multiple times and identify and use the output results (output data) with high reliability (credibility) from among the multiple output results. For example, it is possible to input the same input data and prompts into an LLM multiple times and, if the same output results (output data) are output above a threshold, adopt the same output data that is output above that threshold as the output data with high reliability (credibility). An example of this is shown in Figure 30.

[0192] [Example of operation of information processing device] Fig. 30 is a flowchart showing an example of the determination process in the information processing device 100. This determination process is a partial modification of the determination process shown in Fig. 9. Specifically, steps S511, S513, and S519 to S523 correspond to steps S501 to S507. Therefore, some explanations of the parts corresponding to Fig. 9 will be omitted. This determination process will be explained with appropriate reference to Figs. 1 to 29.

[0193] In step S512, the tagging unit 120 sets a variable N to 0. The variable N is a value indicating the number of times the tagging process has been performed.

[0194] In step S514, the tagging unit 120 stores the tagged target information that has been tagged in step S513 in association with the variable N. For example, the storage unit 150 can store the tagged target information.

[0195] In step S515, the assignment unit 120 sets the variable N to N+1.

[0196] In step S516, the assigning unit 120 determines whether the assigning process in step S513 has been executed M times. If the assigning process in step S513 has been executed M times, the process proceeds to step S517. On the other hand, if the assigning process in step S513 has not been executed M times, the process returns to step S513.

[0197] Here, M is a value indicating the number of processes required to evaluate the reliability (belief) of the assignment process in step S513. For example, a value of 10 or more can be set as M. However, M is not limited to this and can be set appropriately based on experiments, simulations, etc.

[0198] In step S517, the attachment unit 120 compares the attachment results of the M tagging processes executed in step S513. For example, the attachment unit 120 acquires the M pieces of tagged target information (M pieces of tagged target information) held in step S514. The attachment unit 120 then selects a predetermined number (e.g., two or more) of tagged target information from among the M pieces of tagged target information as comparison targets, compares the predetermined number of pieces of tagged target information to determine whether the sentences and characters included in the predetermined number of pieces of tagged target information to be compared are identical. This comparison process can employ known character recognition technology. The attachment unit 120 also performs this identity determination process (comparison process) for all combinations of the M pieces of tagged target information. The attachment unit 120 then calculates the number of pieces of tagged target information that have the same sentences and characters as the number of pieces of tagged target information with identical attachment results.

[0199] Note that these identity determination processes (comparison processes) are merely examples, and other processes may be executed. For example, the assigning unit 120 extracts tag portions of named entities tagged in M ​​pieces of tagged target information. For example, when the named entity is monetary amount data, the assigning unit 120 extracts the tag portion of " <money type="…"> …circle< / money> " is identified as the tag portion. Then, the assignment unit 120 compares the extracted tag portions for each of the M pieces of tagged target information to determine whether the tag portions are identical. This comparison process can employ known character recognition technology. The assignment unit 120 also performs this identity determination process (comparison process) for all combinations of the M pieces of tagged target information. Then, the assignment unit 120 calculates the number of tagged target information pieces that have the same tag portions, and sets this as the number of tagged target information pieces that have the same assignment result.

[0200] In step S518, the assigning unit 120 determines whether the number of identical assignment results among the M assignment results compared in step S517 is equal to or greater than a threshold value TH1. If the number of identical assignment results is equal to or greater than the threshold value TH1, the process proceeds to step S519. On the other hand, if the number of identical assignment results is less than the threshold value TH1, the assigning unit 120 outputs the assignment results of the M assignment processes to the output control unit 160, and the process proceeds to step S524.

[0201] Here, the threshold value TH1 is a value that serves as a reference when evaluating the reliability (belief) of the assignment process in step S513. This threshold value TH1 can be set to, for example, a value of M / 2 or more. For example, when M is 10, a value of 5 or more can be set. However, the threshold value TH1 is not limited to this and can be set appropriately based on experiments, simulations, etc.

[0202] In step S519, the determination unit 130 performs a determination process to determine whether the named entities included in the tagged target information are identical or not, using the tagging results determined to be identical in step S517 from among the tagging results of the M tagging processes performed in step S513. In this case, since the tagging results have been determined to be identical, it is possible to perform the determination process using one of the tagging results determined to be identical. Note that this determination process is similar to step S503 shown in FIG. 9, and therefore description thereof will be omitted here.

[0203] In step S524, the output control unit 160 causes the output unit 170 to output the granting results of the M granting processes executed in step S513. For example, the output control unit 160 causes the display unit 171 of the output unit 170 to display the granting results of the M granting processes stored in step S514 in a comparable manner. In this case, if there are many display contents of the granting results of the M granting processes, they may be displayed by scrolling or the like, or the display areas (windows) of the granting results of each granting process may be displayed overlapping each other, or other display modes may be used. In these cases, different parts of the display contents of the granting results of the M granting processes may be displayed in a distinguishable manner (for example, by adding color or making them bold).

[0204] The user can select an adopted grant result from among the grant results of the M grant processes displayed in this way. For example, the user can select an adopted grant result using an operating member (e.g., a mouse, a keyboard, a touch panel, or a voice input device). In this way, the grant result selected by the user as the adopted grant result may be displayed in a selected state so as to be distinguishable from other grant results (e.g., by adding color or making it bold).

[0205] 30 illustrates an example in which, when it is determined in step S518 that the number of identical grant results is equal to or greater than the threshold value TH1, the grant results of M grant processes are output in step S524, and the user is allowed to select the grant result to be adopted by operation. However, this is not limiting. For example, when it is determined in step S518 that there are multiple groups of identical grant results (the number of which is equal to or greater than the threshold value TH2), the grant results of the grant processes may be output for each group in step S524, and the user may be allowed to select the grant result to be adopted (for each group). For example, when M is 20, the threshold value TH2 can be set to a value of approximately 2 to 5. Then, for example, when it is determined that there are groups of identical grant results (for example, a first group of three, a second group of four, and a third group of five), the grant results of the grant processes may be output for each group (for example, the first to third groups), and the user may be allowed to select the grant result to be adopted (for example, any of the first to third groups). The threshold value TH2 is not limited to this, and can be set appropriately based on experiments, simulations, or the like.

[0206] In step S525, the output control unit 160 determines whether or not a selection operation has been performed to select an adopted assignment result from among the assignment results of the M assignment processes output in step S524. For example, it is possible to determine whether or not a decision operation (e.g., a selection operation of the decision button) has been performed by the user to determine the content of the selection operation after the assignment results of the M assignment processes have been output from the output unit 170 in step S524. If a selection operation to select an assigned result to be adopted has been performed, the process proceeds to step S526. On the other hand, if a selection operation to select an assigned result to be adopted has not been performed, the operation of the determination process ends.

[0207] In step S526, the assignment unit 120 generates a prompt for reflecting the assignment result selected in step S525 (the assignment result to be adopted).

[0208] For example, assume that the target information acquired in step S511 and the tagging result (tagged target information) selected in step S525 are used as sample data. In this case, the sample data of the tag is inserted into a prompt as a tagging example, and a prompt including an instruction to tag the target information is input to the LLM 121 together with the target information, and the output data in response thereto can be used as tagged target information. Note that this prompt is merely an example and is not limiting. Each of the prompts described above may also be employed.

[0209] Here, the LLM outputs output data that it believes to be correct for all input data and prompts. However, as mentioned above, it is currently known that there is a problem of inconsistency in the output results from the LLM. Furthermore, the correctness (correctness) of the output data from the LLM depends on the user who uses the LLM. Therefore, it is preferable for the user to decide which of the multiple output data from the LLM to adopt.

[0210] It is also possible to execute the determination process for multiple pieces of target information (e.g., multiple securities reports). In this case, it is also possible to execute the tagging process for the other pieces of target information in step S513 using the prompt generated in step S526. In this way, it is possible to issue an additional command (additional instruction) to the LLM 121 so that tags in the style desired by the user are assigned.

[0211] In this way, the same input data and prompts can be input multiple times into the LLM, and from among the multiple output results, it is possible to identify and use the output result (output data) with the highest reliability (credibility). This makes it possible to perform identity determination processing using the output result (output data) with the highest reliability (credibility) even if the output results output from the LLM are inconsistent. In other words, it is possible to improve the accuracy of the identity determination processing.

[0212] [Example of summarizing variations in attribute information] As mentioned above, it is currently known that there is a problem of inconsistency in the output results from LLMs. For example, even when input data and prompts are entered into an LLM, different attribute names may be output as the output results. For example, it is expected that output results such as "sales increase amount," "sales increase amount," and "sales increase value" may be output for the same named entity as types of attribute information indicating the amount of increase in sales. This phenomenon can be called spelling variation (spelling variation) of attribute information. An example of this spelling variation is shown in Figure 31(A).

[0213] Figure 31 is a simplified diagram showing a portion of tagged target information 700 output from LLM 121 in response to input of target information. Figure 31(A) shows an example in which spelling variations occur in tagged target information 700. Note that Figure 31(A) only shows the attribute information in which spelling variations occur. Specifically, this shows an example in which the attribute information "type" of monetary data (named entities) included in tagged target information 700 is given multiple different names (sales increase, sales increase amount, sales increase value) despite being an attribute indicating the same content.

[0214] Figure 31(B) shows an example of consolidating the notation variations (sales increase, sales increase amount, sales increase value) shown in Figure 31(A). That is, it shows an example of unifying multiple different names (sales increase, sales increase amount, sales increase value) into one name (sales increase). This method of consolidating notation variations will be described in detail with reference to Figure 32.

[0215] As shown in Figure 31(A), when there are spelling variations in the same target information, it is possible to unify the names of multiple attributes that have spelling variations. That is, among the multiple attribute names assigned to named entities included in the same target information, it is possible to unify the names of attributes that can be determined to indicate the same attribute to one of the names. For example, when the same named entity (amount of increase in sales) included in the same target information is assigned multiple attribute names (sales increase, sales increase amount, sales increase value), it is possible to unify the names to one of the names. Therefore, an example of unifying the names to one of the names is shown in Figure 32.

[0216] [Example of operation of information processing device] Fig. 32 is a flowchart showing an example of the determination process in the information processing device 100. Note that this determination process is a partial modification of the determination process shown in Fig. 9. Specifically, steps S531, S532, and S535 to S539 correspond to steps S501 to S507. Therefore, some explanations of the parts corresponding to Fig. 9 will be omitted. Also, this determination process will be explained with appropriate reference to Figs. 1 to 31.

[0217] In step S533, the assigning unit 120 executes a detection process to detect spelling variations in the names of each piece of attribute information included in the tagged target information to which attribute information has been assigned in step S532 (e.g., the tagged target information 700). That is, the assigning unit 120 executes a determination process to determine whether or not a spelling variation has occurred in the names of each piece of attribute information included in the tagged target information to which attribute information has been assigned. If a spelling variation is detected, the process proceeds to step S534. On the other hand, if a spelling variation is not detected, the process proceeds to step S535.

[0218] Specifically, the tagging unit 120 extracts tags of the same category from the tags that have been assigned to the tagged target information. For example, the tag for the amount data (MONEY) includes the following: <MONEY type=…< / person> < / org> < / money> are assigned, so that tags to which these are assigned can be extracted as tags of amount data (MONEY).

[0219] Next, the assignment unit 120 acquires the notation of the attribute information "type" included in each tag of the extracted same category. Then, the assignment unit 120 extracts notations that are partially identical and partially different from the acquired notations of "type." For example, in the example shown in FIG. 31(A), the notations (sales increase, sales increase amount, sales increase value) of the attribute information "type" of the monetary data (named entity) included in the tagged target information 700 are partially identical ("sales increase") but differ in other parts ("minutes," "amount," "value"). Therefore, the assignment unit 120 extracts the sales increase, sales increase amount, and sales increase value from the notations of the attribute information "type" of the monetary data (named entity) included in the tagged target information 700. In this case, the assignment unit 120 detects the sales increase, sales increase amount, and sales increase value that have notation variations from the names of the attribute information included in the tagged target information 700.

[0220] In step S534, the assignment unit 120 performs a spelling variation consolidation process to consolidate the spelling variations detected in step S533. For example, if the spellings for which spelling variations have been detected are synonyms, paraphrases, or the like, it is considered possible to consolidate the spellings for which spelling variations have been detected into one spelling. On the other hand, if the spellings for which spelling variations have been detected have completely different meanings, it is considered impossible to consolidate the spellings for which spelling variations have been detected into one spelling. Therefore, in this embodiment, an example is shown in which the spelling variation consolidation process is performed when the spellings for which spelling variations have been detected are synonyms, paraphrases, or the like.

[0221] For example, for each spelling in which spelling variations were detected in step S533, a prompt (including tagged target information as needed) inquiring whether the spellings mean the same thing can be input to the LLM 121, and the spelling variations can be consolidated based on the output result. For example, if the output result from the LLM 121 indicates that the spellings in which spelling variations were detected mean the same thing, the spelling variations can be resolved by inputting a prompt to the LLM 121 instructing it to unify the spelling variations (sales increase, sales increase amount, sales increase value) into one of the spellings (for example, sales increase).

[0222] Furthermore, for example, for each spelling variation detected in step S533, a prompt (including tagged target information as needed) inquiring whether the spelling will mean the same thing even if swapped is input to the LLM 121, and the spelling variations can be consolidated based on the output result. For example, if the output result from the LLM 121 indicates that the spelling will mean the same thing even if swapped, the spelling variations (sales increase, sales increase amount, sales increase value) can be resolved by inputting a prompt to the LLM 121 instructing it to unify the spelling variations (sales increase, sales increase amount, sales increase value) into one of the spellings (for example, sales increase amount).

[0223] 30, the tagging process may be executed multiple times, and spelling variations may be consolidated based on the tagging results. For example, if spelling variations are detected in step S533, one or more new tagging processes may be executed for the same target information, and it may be confirmed whether the spellings included in the tagging results (the spellings in which spelling variations are detected) are the same or have been swapped. If the spellings included in the tagging results (the spellings in which spelling variations are detected) are the same or have been swapped, the spelling variations may be consolidated. In this case, the spelling variations may be resolved by inputting a prompt to the LLM 121 to unify the spelling variations (sales increase, sales increase amount, sales increase value) into one of the spellings (e.g., sales increase amount).

[0224] Note that other prompts may be used as prompts to instruct the user to unify variations in notation (sales increase, sales increase amount, sales increase value) into one notation (e.g., sales increase). For example, a prompt may be used to instruct the user to unify attribute information "type" relating to the amount of increase in sales to "sales increase." Also, for notations that mean the same thing, such as sales increase, sales increase amount, and sales increase value, a prompt may be used to instruct the user to unify all attribute information "type" to "sales increase." In this way, using a prompt that imposes some kind of constraint often allows the LLM 121 to refine the output results.

[0225] In this way, when a spelling variation occurs, by eliminating the spelling variation, it is possible to improve the search accuracy when searching for attribute information of the same type, which in turn makes it possible to improve the accuracy of error detection processing, for example.

[0226] The spelling variation detected in step S533 may be displayed on the display unit 171 to notify the user, and the spelling variation may be resolved based on some operation by the user. For example, the user may check the spelling variation (e.g., sales increase, sales increase amount, sales increase value) displayed on the display unit 171 and subjectively decide to unify it to "sales increase." In this case, the spelling variation can be resolved by inputting a prompt to the LLM 121 instructing to unify the spelling variation (sales increase, sales increase amount, sales increase value) to "sales increase."

[0227] In this way, for the tagged target information for which spelling variation consolidation processing has been performed, it is determined again in step S533 whether or not a spelling variation has been detected (i.e., whether or not the spelling variation has been resolved). If it is determined that the spelling variation has not been resolved, the process proceeds to step S534, where the spelling variation consolidation processing is performed again.

[0228] In this way, if there is a variation in the notation of attribute information in the output result from the LLM, it is possible to resolve the variation and use it. As a result, even if the output result from the LLM is inconsistent, it is possible to perform the identity determination process using an output result (output data) with high reliability (credibility). In other words, it is possible to improve the accuracy of the identity determination process.

[0229] In the above example, we have shown how to resolve orthographic variations by detecting orthographic variations in the attribute information "type" of a named entity and then performing a process to consolidate the orthographic variations. However, it is also possible to similarly detect orthographic variations in other attribute information of named entities (e.g., fiscal, unit, etc.) and perform a process to consolidate the orthographic variations to resolve orthographic variations.

[0230] [Example of generating attribute names for named entities using LLM] The above shows an example of tagging, where attributes related to named entities are set in advance and then tagged using those attributes. It is possible to add attributes that have not yet been defined and assign appropriate attribute names to those attributes by asking various questions to the LLM. It is also possible to input target information into the LLM, inquire about the types of named entities contained in the target information, and then have the LLM define tag names, attribute names, etc. appropriate for the target information based on the output results. Therefore, below is an example of generating attribute names, etc. related to named entities using the LLM.

[0231] It is known that humans tend to minimize the content of descriptions of objects. For example, human attributes always include information about birth (e.g., date of birth, place of birth). However, when humans generate samples of attribute information for named entities (e.g., names) contained in target information, they often discard (delete) attributes (e.g., date of birth, place of birth) that are not relevant to the analysis and verification of the target information in order to prevent the assignment of a large amount of attribute information. It is possible that these discarded attributes may be needed for some processing. Therefore, by tagging named entities using LLM, it is possible to assign various attribute information to named entities depending on the target information.

[0232] For example, a query about the existence of human attributes can be input to the LLM, and the resulting output (attributes) can be used for tagging. In this case, the LLM lists the human attributes it knows, and the information corresponding to those attributes is assigned as attribute information. This makes it possible to properly assign attributes to specific named entities (e.g., human names). However, depending on the type of attribute, attribute information may not be stored. For example, consider a case where company attributes are listed and information corresponding to those attributes is assigned as attribute information. In this case, "COMPANY" is stored as the attribute information "type," but since no information corresponding to the attribute information "founder" exists in the target information, it is possible that the founder information will not be assigned to the attribute information "founder." In this way, when attributes are generated by the LLM, attribute information may not be assigned to the generated attributes. However, for attributes that can be assigned, attribute information appropriate to that attribute is assigned.

[0233] Also, for example, it is conceivable to input a query into the LLM as to what category of named entities is contained in the target information, and then use the output result (named entities) to input a query into the LLM as to what attributes exist for that named entity, and then use the output result (attributes) to perform tagging. For example, suppose a query is input into the LLM as to what category of named entities is contained in the target information, and the output result indicates that monetary amount data is included. In this case, it is conceivable to input a query into the LLM as to what attributes exist for that monetary amount data, and then use the output result (attributes) to perform tagging.

[0234] Also, for example, suppose you input a query into LLM asking what categories of named entities are included in the target information, and the output shows that COMPANY (company name) is included. In this case, you could input a query into LLM asking what attributes exist for that COMPANY, and use the output results (attributes) to tag the information. For example, attributes for COMPANY could include shareholders, business partners, subsidiaries, relationships, clients, etc.

[0235] Alternatively, for example, a simplified prompt may be used, for example, by inputting target information and a prompt instructing the LLM to tag named entities contained in the target information with attributes related to the named entities, and using the output result.

[0236] It is generally known that when humans write text, the spelling of a named entity often changes between the first and subsequent spellings (especially the last spelling). For example, when writing the musical instrument violin in Japanese, it is conceivable that the musical instrument violin might be written as "ビイルン" ("biorin") at the beginning of the sentence and as "ヴァイオリン" ("violin") at the end. Therefore, by using LLM to assign a large number of attributes (attribute information) to named entities, it is possible to use this large amount of attribute information to appropriately extract named entities to be compared. This also makes it possible to appropriately perform identity determination of the named entities to be compared. For example, by comparing the spelling of "ビイルン" with the spelling of "ヴァイオリン," it is possible to discover and point out differences in the spelling. This makes it possible to perform identity determination of named entities on inconsistent output, such as text written by humans, and to generate consistent and appropriate output.

[0237] Although the above example shows a prompt being generated using the term "named entity," the prompt may be generated using the term "proper noun" instead of the named entity.

[0238] [Example of operation of information processing device] Fig. 33 is a flowchart showing an example of the determination process in the information processing device 100. Note that this determination process is a partial modification of the determination process shown in Fig. 9. Specifically, steps S541 and S543 to S548 correspond to steps S501 to S507. Therefore, some explanations of the parts corresponding to Fig. 9 will be omitted. Also, this determination process will be explained with appropriate reference to Figs. 1 to 32.

[0239] In step S542, the assignment unit 120 executes a generation process using the LLM to generate attributes related to named entities included in the target document. For example, as described above, it is conceivable to input a query to the LLM as to what attributes exist for named entities included in the target document, and obtain the output results (attributes).

[0240] In step S543, the assignment unit 120 extracts named entities included in the target document 10 output from the acquisition unit 110, and extracts attribute information related to the extracted named entities. In this case, the assignment unit 120 extracts attribute information related to the attributes generated in step S542.

[0241] In this way, instead of using prompts using sample data generated by a user, it is possible to have the LLM generate some or all of the sample data (or equivalent data) and use the generated prompts. In other words, it is possible to have the LLM perform some or all of the prompt generation process that would otherwise be performed manually by a user. For example, it is possible to input all of the target information into the LLM, query which named entities (or proper nouns) are contained in the target information, and then have the LLM define tag names, attribute names, etc. appropriate for the target information based on the query results. This increases the reliability (credibility) of the output results from the LLM, and enables the identity determination process to be performed using these highly reliable (credible) output results (output data). In other words, it is possible to improve the accuracy of the identity determination process.

[0242] [Example of generating a graph of named entities using LLM] The above shows an example of using LLM to assign attribute information to named entities. Here, target information is input into LLM, and graph information about the named entities contained in the target information can be output from LLM. This graph information can be, for example, a graph that represents the relationships between various types of information in a network format. For example, a knowledge graph, which systematically links various types of information (knowledge) and represents them in a graph structure, can be used.

[0243] Figure 34 is a diagram showing an example of graph information that can be realized using LLM. The graph information shown in Figure 34(A) is an example of a knowledge graph in which Company A and individuals (Yamada Taro, Okawa Jiro, Tanaka Saburo) are represented as points (nodes), the relationships between them are connected by lines (links, edges), and information expressing those relationships is represented on the lines.

[0244] For example, named entities, entities, etc., can be represented by points. Furthermore, for example, a verb in a sentence becomes information indicating the relationship between one or more nouns related to that verb. Therefore, for example, based on the nouns and verbs in a sentence, the relationship between nouns and verbs can be represented by lines. That is, the relationship connecting one named entity or entity to another can be represented by a line. For example, Figure 34(B) shows an example of graph information related to the sentence, "The culprit is a friend of Mr. A, who complained to his neighbor about stepping on his cat's tail." Note that the modification relationships in this sentence contain some unclear parts, so it is expected that it will be difficult to confirm the relationships between the literary objects appearing in the sentence alone. Therefore, by using the graph information shown in Figure 34(B), it is possible to easily confirm the relationships between the literary objects appearing in the sentence.

[0245] Here, the LLM is equipped with a language for drawing pictures. For example, it is possible to generate various shapes such as squares and triangles based on the output results from the LLM. For example, it is possible to input target information and instruction information (prompts) to the LLM to extract named entities contained in the target information and generate graph information that connects the relationships between the extracted named entities with lines, and then use the output data (graph information related to the target information) in response to this.

[0246] For example, a predetermined diagram in Mermaid notation can be used as this graph information. For example, when ChatGPT, GPT-4, GPT-4 Turbo, GPT-4o, or the like is used, these can handle Mermaid notation, and therefore a predetermined diagram in Mermaid notation can be output as graph information. Note that these prompts are merely examples and are not limiting. Other prompts capable of generating graph information indicating the relationships between named entities contained in the target information may also be used. For example, a prompt indicating that the relationships between named entities contained in the target information should be output in a predetermined diagram in Mermaid notation may be used. Furthermore, for example, in addition to generating the graph information, a prompt instructing the LLM to extract attribute information for each named entity and assign it to each named entity, as described above, may be input to the LLM. In this case, as described below, output data from the LLM (e.g., graph information related to the target information, attribute information for each node (named entity)) can be used in the determination process.

[0247] Furthermore, it is possible to adopt a data format such as a tuple as graph information. For example, in the tuple data format, it is possible to extract cases where the objects are the same based on three elements: subject, verb, and object, and generate graph information showing the relationships between the subjects and objects based on the extraction results.

[0248] For the output data (graph information) generated by the LLM in this way, it is possible to generate a graph from the output data (graph information) of the LLM using, for example, a predetermined drawing application (for example, turtle, which is a standard feature of Python).It is also possible to use, for example, an image generation AI (for example, DALL-E2) that can generate images based on input text information.

[0249] In this way, the assignment unit 120 functions as a graph information generation unit that generates graph information. That is, the assignment unit 120 inputs the target information acquired by the acquisition unit 110 and instruction information (prompt) to the LLM 121 to extract named entities contained in the target information and generate graph information that connects the relationships between the extracted named entities with lines, and acquires corresponding output data (graph information related to the target information). In this case, the output control unit 160 can display graph information on the display unit 171 based on the output data (graph information related to the target information). For example, the output control unit 160 can display graph information on the display unit 171 using graph theory related to graphs consisting of a set of nodes (nodes, vertices, points) and a set of edges (branches, edges, lines). For example, graph information such as that shown in FIGS. 34(A) and 34(B) can be displayed.

[0250] Furthermore, for example, when the target information is a securities report, each monetary data item is assigned to a node, and the relationships between these monetary data items can be connected by lines. In this case, the monetary data item at a higher level (e.g., total sales) will match the total amount of each monetary data item at a lower level (e.g., sales for a reportable segment, other sales items) belonging to that level. Therefore, for example, it can be determined whether the monetary data item at a higher level matches the total amount of each monetary data item at a lower level belonging to that level, and if they match, it can be determined that the monetary data being evaluated is correct. On the other hand, if they do not match, it can be determined that there is an error in one of the monetary data items being evaluated.

[0251] In this way, by using LLM, it is possible to convert the relationships between named entities contained in the target information into a graph and discover inconsistencies within the target information. Furthermore, by displaying a graph showing the relationships between named entities contained in the target information, it is possible to make it easier for users to visually and intuitively grasp the named entities. Furthermore, by displaying a graph showing the relationships between named entities contained in the target information, it becomes easier to track the named entities contained in the target information and to explain those named entities.

[0252] [Example of operation of information processing device] Fig. 35 is a flowchart showing an example of the determination process in the information processing device 100. Note that this determination process is a partial modification of the determination process shown in Fig. 9. Specifically, steps S551 and S553 to S557 correspond to steps S501 and S503 to S507. Therefore, some explanations of the parts corresponding to Fig. 9 will be omitted. Also, this determination process will be explained with appropriate reference to Figs. 1 to 34.

[0253] In step S552, the assignment unit 120 executes a generation process using the LLM to generate graph information indicating the associations between named entities contained in the target document. For example, as described above, it is conceivable to input a query to the LLM as to what attributes exist for named entities contained in the target document, and obtain the output results (attributes).

[0254] In step S553, the assignment unit 120 uses the graph information generated in step S552 to execute a determination process for determining whether or not the named entities included in the graph information are identical. For example, when the target information is a securities report, as described above, it is possible to determine whether or not the monetary data being the subject of the determination is correct based on whether or not the monetary data in the upper layer matches the total amount of the monetary data in the lower layers belonging to it.

[0255] Furthermore, it is assumed that identical named entities contained in the same target information will basically become the same node. Therefore, if the same named entity contained in the same target information exists as different nodes, it is possible to determine that one of those named entities is incorrect. In this case, as an estimation method for estimating that different nodes are the same named entity contained in the same target information, it is possible to adopt, for example, a known estimation method for estimating the similarity between objects (nodes). For example, as a similarity estimation method, it is possible to adopt an estimation method based on distance, an estimation method based on similarity, etc. As a distance-based estimation method, it is possible to adopt an estimation method based on Euclidean distance, Manhattan distance, Chebyshev distance, etc. As a similarity-based estimation method, it is possible to adopt cosine similarity, Jaccard coefficient, Dice coefficient, etc.

[0256] For example, consider a case where the same named entity (which should be the same) included in the same target information is represented by different nodes (first node, second node). In this case, it is determined whether the attribute information (e.g., type) assigned to each of the first node and the second node is the same. If this attribute information is the same, it is possible that the first node and the second node are the same named entity. In this case, it is possible to determine the degree of agreement between the first node and the second node in the graph using the above-described method for estimating the similarity of each node. Furthermore, for example, it is possible to determine whether the relationship between the first node and other nodes connected thereto in the graph matches, and the relationship between the second node and other nodes connected thereto in the graph matches, and if these relationships match (or are similar above a predetermined standard), it is possible to determine that the first node and the second node are the same named entity. In this way, it is possible to determine whether each node in the graph is the same named entity based on the degree of agreement (or similarity) between each node. Furthermore, if the same named entity (which should be the same) contained in the same target information is in different nodes (first node, second node), it can be determined that the named entity corresponding to either the first node or the second node is incorrect.

[0257] In this way, graph information about named entities included in the target information can be output from the LLM, and the graph information can be used to perform identity determination processing, thereby improving the accuracy of identity determination processing.

[0258] [Configuration example and effects of this embodiment] The information processing devices 100 and 400 utilize an LLM (an example of a large-scale language model) to extract monetary data (an example of a named entity) and attribute information (e.g., type, fiscal, unit, amount) that can identify the monetary data from a target document 10 (an example of target information), and include an assignment unit 120 that assigns the extracted attribute information to the monetary data. The information processing devices 100 and 400 also include a determination unit 130 that identifies multiple monetary data of the same type from the monetary data contained in the target document 10 based on the attribute information, and compares the multiple monetary data to determine whether the multiple monetary data are identical. The information processing method according to this embodiment includes these processes. The program according to this embodiment is a program that causes a computer to execute these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each function executable by each information processing device. The information processing devices 100 and 400 shown here may be configured as a single device or multiple devices. Furthermore, instead of the information processing devices 100 and 400, an information processing system may be configured with a plurality of devices capable of executing the processes realized by the information processing devices 100 and 400. The target document 10 is, for example, a disclosure document (e.g., a securities report) disclosed by a company.

[0259] According to this configuration, it is possible to use LLM to extract attribute information of the amount data included in the target document 10, and to use this attribute information to appropriately determine the identity of the amount data. Furthermore, since it is possible to use LLM to perform a determination process using the attribute information assigned to the target document 10, it is possible to reduce the computational load involved in the determination process. Furthermore, since it is possible to perform a determination process using the attribute information assigned to the target document 10 using LLM, it is possible to improve the determination accuracy of the determination process.

[0260] The assignment unit 120 extracts the target document 10 (an example of target information), amount data (an example of a named entity), and attribute information (e.g., type, fiscal, unit, amount) related to the amount data from the target document 10, inputs a prompt (an example of instruction information) that instructs the assignment of the extracted attribute information to the extracted amount data into an LLM (an example of a large-scale language model), and obtains the output result from the LLM, a tagged target document 20 in which attribute information has been assigned to the amount data.

[0261] According to this configuration, by inputting the target document 10 and a predetermined prompt into the LLM, it is possible to obtain from the LLM a tagged target document 20 in which attribute information is added to the amount data. This makes it possible to appropriately extract the amount data and the attribute information of that amount data using the LLM.

[0262] Target document 10 (an example of target information) includes both text information 11 and table information 12. Addition unit 120 adds attribute information related to amount data (an example of a named entity) to the position of the amount data in text information 11 to generate text information 21, and adds attribute information related to the amount data to the position of the amount data in text information 22 (character information and numerical information) related to converted table information 12, which is obtained by converting table information 12 into the character information and numerical information contained in that table information 12.

[0263] For example, as described above, workers who check target documents such as securities reports often know the location of the part of the target document where the content to be checked is described. Therefore, by displaying tagged target document 20 to workers who check target documents such as securities reports, the content to be checked can be easily understood.

[0264] The information processing devices 100 and 400 further include an output control unit 160 that displays, on the display unit 171, a determination result obtained by using attribute information added to text information 11 and attribute information added to text information 22 (character information and numerical information) related to table information 12, superimposed on the text information 21 to which the attribute information has been added and the text information 22 (character information and numerical information) to which the attribute information has been added. For example, as shown in Fig. 8, the determination results (ovals J1 to J4, triangles E1 and E2) are superimposed and displayed on the text information 21 to which the attribute information has been added and the text information 22 to which the attribute information has been added.

[0265] For example, as described above, workers reviewing target documents such as securities reports often know the location of the portion of the target document where the content to be reviewed is described. Therefore, by displaying the tagged target document 20 with the judgment results (ovals J1-J4, triangles E1 and E2) superimposed, workers reviewing target documents such as securities reports can easily understand the content to be reviewed. Furthermore, by displaying the tagged target document 20, it is possible to verify the accuracy by checking the content before and after the portion to be reviewed. Therefore, it is possible to easily verify why the extracted attribute information was output by the LLM.

[0266] The assignment unit 120 extracts the type (type) of amount data (amount expression) in the target document 10 (an example of target information), the fiscal year information (fiscal) of the amount data, the unit type (unit) of the amount data, and the total numerical value (amount) of the amount data as attribute information.

[0267] According to this configuration, it is possible to appropriately extract attribute information that can improve the accuracy of the determination process by the determination unit 130.

[0268] The judgment unit 130 uses at least one of the type of monetary data (monetary expression), fiscal information of the monetary data, and unit type of the monetary data to identify multiple monetary data of the same type from the monetary data, compares the total amount of the numerical values ​​of the multiple monetary data, and judges whether the multiple monetary data are the same.

[0269] According to this configuration, it is possible to identify multiple amount data of the same type using one or multiple attribute information, and the results of this identification can be used to appropriately determine whether the multiple amount data are the same.

[0270] The attachment unit 120 inputs the target document 10 (an example of target information) and the above-mentioned prompt (an example of instruction information) into an LLM (an example of a large-scale language model) a plurality of times to obtain a plurality of output results. Then, the attachment unit 120 compares the plurality of output results and, based on the comparison results, determines an output result to be used in the determination process by the determination unit 130 from among the plurality of output results. For example, as shown in FIG. 30 , when tag attachment processes are performed M times, if the same attachment result among the M attachment results is equal to or greater than the threshold value TH1, the attachment unit 120 determines the same attachment result as the output result to be used in the determination process by the determination unit 130.

[0271] With this configuration, the same input data and prompts can be input to the LLM multiple times, and from among the multiple output results, it is possible to identify and use the output result (output data) with high reliability (credibility). As a result, even if the output results output from the LLM are inconsistent, it is possible to perform identity determination processing using the output result (output data) with high reliability (credibility). In other words, it is possible to improve the accuracy of the identity determination processing.

[0272] The attachment unit 120 compares multiple output results from an LLM (an example of a large-scale language model), and if there are many output results with the same content relative to a threshold value TH1, the output result with the same content is used as the output result to be used in the determination process by the determination unit 130. For example, as shown in Fig. 30, when tag attachment processes are executed M times, the attachment unit 120 compares the M attachment results, and if the number of identical attachment results among the M attachment results is equal to or greater than the threshold value TH1, the identical attachment result is used as the output result to be used in the determination process by the determination unit 130.

[0273] With this configuration, the same input data and prompts can be input into the LLM multiple times, and from among the multiple output results, it is possible to identify and use the output result (output data) that has the highest reliability (believability) based on threshold value TH1.

[0274] If there are multiple pieces of attribute information with spelling variations among the extracted attribute information, the assigning unit 120 uses an LLM (an example of a large-scale language model) to perform a process of unifying the spellings of the multiple pieces of attribute information with spelling variations. For example, as shown in Figure 31(A), if there is spelling variation in the attribute information "type", the assigning unit 120 uses an LLM (an example of a large-scale language model) to perform a process of unifying the spellings of the multiple pieces of attribute information "type" with spelling variations. As a result, the spelling variation of the attribute information "type" is resolved as shown in Figure 31(B).

[0275] With this configuration, if there is a variation in the notation of attribute information in the output result from the LLM, it is possible to resolve the variation and use it. As a result, even if the output result output from the LLM is inconsistent, it is possible to execute the identity determination process using an output result (output data) with high reliability (credibility). In other words, it is possible to improve the accuracy of the identity determination process.

[0276] The assignment unit 120 inputs the target information in which attribute information has been assigned to the named entity and a prompt (instruction information) that instructs the unification of the spelling of multiple pieces of attribute information in which spelling variations have occurred into an LLM (an example of a large-scale language model), and obtains, as an output result from the LLM, the target information in which the spelling of multiple pieces of attribute information in which spelling variations have occurred has been unified.

[0277] According to this configuration, if there is a variation in the notation of attribute information in the output result from the LLM, it is possible to resolve the variation in notation using the LLM by using a predetermined prompt.

[0278] The assignment unit 120 queries an LLM (an example of a large-scale language model) for named entities contained in a target document 10 (an example of target information) and attributes related to those named entities, and extracts attribute information related to the named entities and attributes based on the query results from the LLM.

[0279] This configuration allows the LLM to generate some or all of the sample data (or equivalent data) generated by the user, and then use the generated data to generate prompts. In other words, the LLM can perform some or all of the prompt generation process, which would otherwise be performed manually by the user. For example, the entire target information can be input into the LLM, and a query can be made to determine which named entities (or proper nouns) are contained in the target information. Based on the query results, the LLM can define tag names, attribute names, and other attributes appropriate for the target information. This increases the reliability (credibility) of the output results from the LLM, and enables the identity determination process to be performed using these highly reliable (credible) output results (output data). In other words, the accuracy of the identity determination process can be improved.

[0280] The information processing devices 100 and 400 include an attachment unit 120 (an example of a generation unit) that uses an LLM (an example of a large-scale language model) to extract multiple named entities (e.g., numerical values, names, and time) included in a target document 10 (an example of target information) and generates graph information representing the relationships between the multiple named entities; and a determination unit 130 that identifies multiple named entities of the same type from among the named entities included in the target document 10 based on the graph information and compares the multiple named entities to determine whether the multiple named entities are identical. An information processing method according to this embodiment includes each of these processes. A program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, a program according to this embodiment is a program that causes a computer to realize each function that can be executed by each information processing device. The information processing devices 100 and 400 shown here may be configured by a single device or multiple devices. Instead of the information processing devices 100 and 400, an information processing system may be configured by multiple devices that can execute each of the processes realized by the information processing devices 100 and 400. The target document 10 is, for example, a disclosure document (for example, a securities report) disclosed by a company.

[0281] This configuration allows the LLM to output graph information about named entities included in the target information, and the graph information can be used to perform identity determination processing, thereby improving the accuracy of identity determination processing.

[0282] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0283] Each process described in this embodiment is executed based on a program that causes a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to execute each process described in this embodiment.

[0284] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments. [Explanation of symbols]

[0285] 100, 400 information processing device, 110 acquisition unit, 120 assignment unit, 130 determination unit, 140 recording control unit, 150 storage unit, 160 output control unit, 170 output unit, 171 display unit, 180 user terminal, 410 pre-processing unit, 420 post-processing unit, NW1 network

Claims

1. an attribute unit that uses a large-scale language model to extract named entities included in the target information and attribute information that can identify the named entities, and assigns the extracted attribute information to the extracted named entities; a determination unit that identifies a plurality of named entities of the same type from among the named entities included in the target information based on the attribute information, and compares the plurality of named entities to determine whether the plurality of named entities are the same; An information processing device comprising:

2. The assigning unit inputs the target information and instruction information instructing the target information to extract the named entities and the attribute information related to the named entities from the target information and assign the extracted attribute information to the extracted named entities to the large-scale language model, and acquires the target information in which the attribute information is assigned to the named entities, which is an output result from the large-scale language model. The information processing device according to claim 1 .

3. the target information includes both text information and table information; The assigning unit adds the attribute information related to the named entity to a position of the named entity in the text information, and adds the attribute information related to the named entity to a position of the named entity in converted table information obtained by converting the table information into character information and numeric information included in the table information. The information processing device according to claim 1 .

4. an output control unit that displays a determination result determined using the attribute information added to the text information and the attribute information added to the converted table information, superimposed on the text information to which the attribute information has been added and the converted table information to which the attribute information has been added; The information processing device according to claim 3 .

5. the named entity is a monetary amount expression, The assigning unit extracts, as the attribute information, the type of the monetary expression in the target information, fiscal year information of the monetary expression, the type of unit of the monetary expression, and the total amount of the numerical value of the monetary expression. The information processing device according to claim 1 .

6. The determination unit uses at least one of the type of monetary expression, the fiscal year information of the monetary expression, and the type of unit of the monetary expression to identify a plurality of monetary expressions of the same type from among the monetary expressions, and compares the total amounts of the numerical values ​​of the monetary expressions related to the plurality of monetary expressions to determine whether the plurality of monetary expressions are the same. The information processing device according to claim 5 .

7. The target information is a disclosure document disclosed by a company.

7. The information processing device according to claim 1.

8. The granting unit The target information and the instruction information are input to the large-scale language model a plurality of times, and the output results obtained a plurality of times are acquired; determining an output result to be used in the determination process by the determination unit from among the plurality of output results based on a comparison result obtained by comparing the plurality of output results; The information processing device according to claim 2 .

9. the assigning unit compares the plurality of output results, and if there are a large number of output results with the same content based on a threshold, the output result with the same content is set as the output result to be used in the determination process by the determining unit. The information processing device according to claim 8 .

10. When there are a plurality of pieces of attribute information having spelling variations among the extracted attribute information, the assigning unit uses the large-scale language model to perform a process of unifying the spellings of the plurality of pieces of attribute information having spelling variations. The information processing device according to claim 1 .

11. The assigning unit inputs the target information in which the attribute information is assigned to the named entity and instruction information instructing to unify the spellings of the plurality of attribute information in which spelling variations occur to the large-scale language model, and acquires, as an output result from the large-scale language model, the target information in which the spellings of the plurality of attribute information in which spelling variations occur are unified. The information processing device according to claim 10.

12. The assignment unit queries the large-scale language model for the named entities included in the target information and attributes related to the named entities, and extracts attribute information related to the named entities and the attributes based on a query result from the large-scale language model. The information processing device according to claim 1 .

13. a generation unit that extracts a plurality of named entities included in the target information using a large-scale language model and generates graph information that represents the relationships between the plurality of named entities; a determination unit that identifies a plurality of named entities of the same type from among the named entities included in the target information based on the graph information, and compares the plurality of named entities to determine whether the plurality of named entities are the same; An information processing device comprising:

14. an assignment process of extracting named entities included in the target information and attribute information capable of identifying the named entities using a large-scale language model, and assigning the extracted attribute information to the extracted named entities; a determination process of identifying a plurality of named entities of the same type from among the named entities included in the target information based on the attribute information, and comparing the plurality of named entities to determine whether the plurality of named entities are the same; An information processing method including:

15. an assignment step of extracting named entities included in the target information and attribute information capable of identifying the named entities using a large-scale language model, and assigning the extracted attribute information to the extracted named entities; a determination step of identifying a plurality of named entities of the same type from among the named entities included in the target information based on the attribute information, and comparing the plurality of named entities to determine whether the plurality of named entities are the same; A program that causes a computer to execute the following.

Images