Information processing device, control method of information processing device, and program

Abstract

To efficiently perform tag attachment in a document and a semi-standard document having a different layout based on the trading company.SOLUTION: An information processing device which performs processing which attaches a unique expression tag to a document image comprises: reception means which receives an input document image; selection means which selects a reference document image for comparing with the input document image, from among a document image group to which the unique expression tag has been already attached; extraction means which extracts a character block corresponding to a character string to which the unique expression tag has been attached in the reference document image selected by the selection means from the input document image; and attachment means which attaches the unique expression tag to the character string showed by the extracted character block, on the basis of a similarity degree between a feature amount vector of the character string shown by the extracted character block and the feature amount vector of the character string to which the unique expression tag has been attached in the reference document image.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a technique for creating a training data set for a machine learning model. [Background technology]

[0002] Annotation tools that generate learning data by assigning correct answer information (tags) to be learned to target data are used as programs for creating learning data used in machine learning such as named entity extraction. A large amount of documents are collected to generate learning data for highly accurate named entity extraction, but the collected documents may include standardized documents generated from the same template. If a target document to which a new tag is to be added is created from the same template as the standardized document, it is possible to automatically complete the tagging work by reusing the tag information of the standardized document and applying it to the target document as is. There is a need to be able to automatically assign tags to semi-standardized documents that are the same type of document but have different layout details depending on the document or trading company. Here, when the target document is a semi-standardized document, the number of rows in the table or the length of the character string near the tag is different, so it is not possible to automatically assign tags of other semi-standardized documents to which tags have already been assigned. In this regard, Patent Document 1 discloses a technology that automatically sets an area to be extracted to a file name for tables and character strings in a document to be processed that are in the same positional relationship as those in a document scanned in the past when filing a semi-standardized document with a different layout. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-128727 A Summary of the Invention [Problem to be solved by the invention]

[0004] When the technology of Patent Document 1 is applied to tagging a semi-formulated document, the target document is tagged based only on the positional relationship between the target document and the semi-formulated document to which the tag has been added. This may result in a tagging error, such as a tag not being added when the tag should be added, or an incorrect tag being added. When such a tagging error or tagging error occurs, the user must manually re-add the correct tag, which places a burden on the user. The present disclosure has been made in view of the above problems, and aims to efficiently tag semi-formulated documents. [Means for solving the problem]

[0005] The information processing device according to the present disclosure is an information processing device that performs a process of assigning named entity tags to document images, and is characterized in that it comprises an accepting means for accepting an input document image, a selecting means for selecting a reference document image to be compared with the input document image from a group of document images to which named entity tags have already been assigned, an extracting means for extracting from the input document image a character block corresponding to a character string to which the named entity tag has been assigned in the reference document image selected by the selecting means, and an assigning means for assigning a named entity tag to a character string represented by the extracted character block based on the similarity between a feature vector of the character string represented by the extracted character block and a feature vector of a character string to which the named entity tag has been assigned in the reference document image. Effect of the Invention

[0006] According to the technology of the present disclosure, it becomes possible to efficiently assign tags even to semi-standardized documents. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of an information processing device. [Diagram 3] FIG. 4 is a diagram showing an example of a display screen. [Figure 4]11 is a flowchart showing a process for generating learning data. [Diagram 5] 11 is a flowchart showing a process of reading an input document. [Figure 6] 11 is a flowchart showing a process of searching for a similar document. [Figure 7] FIG. 13 is a diagram showing an example of a similar document list. [Figure 8] 11 is a flowchart showing a reference document selection process. [Figure 9] FIG. 4 is a diagram showing an example of tag information. [Figure 10] 11 is a flowchart showing a tagging process. [Figure 11] FIG. 4 is a diagram showing an example of tag frame information. [Figure 12] 11 is a flowchart showing a maximum similar character block extraction process. [Figure 13] 11 is a flowchart showing a tag frame drag start process. [Figure 14] FIG. 13 is a diagram showing an example of moving tag frame information. [Figure 15] 13 is a flowchart showing a tag frame drop process. [Figure 16] 13 is a flowchart showing a storage process. [Figure 17] 13 is a flowchart showing a closing process. [Figure 18] 11A to 11C are diagrams for explaining a process of extracting similar strings within a block. [Figure 19] 11 is a flowchart showing a tag assignment process of a tag assignment unit. [Figure 20] 11 is a flowchart showing a process of extracting a character string within a most similar block. [Figure 21] FIG. 11 is a diagram showing an example of most similar string information. [Figure 22] 11 is a flowchart showing a process of extracting similar strings within a block. [Figure 23] 11 is a flowchart showing a reference document selection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure. In addition, each process (step) in the flow chart is indicated by a reference number beginning with "S".

[0009] [First embodiment] Hereinafter, a first embodiment of the present disclosure will be described with reference to the accompanying drawings. Fig. 1 is a block diagram showing a hardware configuration of an information processing device 100 according to the first embodiment. The information processing device 100 includes a GPU 101, a CPU 102, a ROM 103, a RAM 104, a storage device 105, a network interface 106, an output interface 107, an input interface 108, and a control device 109. Each of the above devices is connected via a system bus 121.

[0010] The GPU 101 is a processor that performs efficient calculations by processing data in parallel, and performs high-speed calculation processing instead of the CPU 102. The CPU 102 reads out programs and data stored in the ROM 103 to the RAM 104 and performs various processes described below. The ROM 103 stores fixed data such as programs and data executed by the CPU 102. The RAM 104 is used as a main memory of the CPU 102 and a temporary storage area such as a work area. The storage device 105 is a large-capacity storage device that stores document data and the like. The network interface 106 is an interface that connects to a network, and the information processing device 100 can be connected to a network such as the Internet. The output interface 107 is an interface that outputs data and the like to an external output device 110. The input interface 108 is an interface that accepts data, commands, and the like from an external input device 111. The control device 109 is a device that performs various controls. The information processing device 100 has been described above as one device, but may be composed of multiple devices. For example, the storage device 105 may be an external storage server.

[0011] FIG. 2 is a block diagram showing a functional configuration of the information processing device 100 according to the present embodiment. The information processing device 100 includes a learning data generation unit 210, a tag assignment unit 220, a storage unit 230, a learning unit 260, and a named entity extraction unit 270. The learning data generation unit 210 generates learning data 232 in which a named entity tag is assigned to each character string of document data 231 stored in the storage unit 230, and stores the learning data in the storage unit 230. The tag assignment unit 220 has a function of efficiently assigning tags to document data 231 to which a named entity tag has not been assigned and which has been read by the learning data generation unit 210. Specifically, the tag assignment unit 220 acquires tag information from learning data 232 to which a named entity tag has been assigned and which has been generated from the same template, and assigns tags to character strings of the document data 231. The learning unit 260 learns the learning data 232 to construct an inference model 281, and stores data of the inference model 281 in the storage unit 230. The memory unit 230 is realized by the storage device 105, and stores data of document data 231, training data 232, and inference model 281, and provides these data to each functional unit. The named entity extraction unit 270 extracts named entities from the document data using the inference model 281.

[0012] 2 is realized by the CPU 102 executing a program read from the ROM 103 to the RAM 104 and a program such as an application loaded from the storage device 105 to the RAM 104. The execution results of each process are held in the RAM 104, while the data of the document data 231, the learning data 232, and the inference model 281 are stored in the storage device 105.

[0013] <Explanation of named entity learning data> First, the named entity tag of the learning data 232 generated by the information processing device of this embodiment will be described. The named entity tag is assigned in the IOB2 format, which is generally used in the field of named entity extraction technology. In the IOB2 format, the B-<arbitrary character string> tag is assigned to the first unit of a string that means a named entity, and the I-<arbitrary character string> tag is assigned to the second unit and thereafter, and if it is not any named entity, the O tag is assigned. Then, the region of the same type of I tag that follows the B tag is regarded as the string of the same tag. For example, when an organization name "ORG" tag is assigned to "ABC Inc." of the string "To:ABC Inc.", the B-ORG tag is assigned to the fourth character "A", the I-ORG tag is assigned to the fifth character "B" to the eleventh character ".", and the O tag is assigned to the first to third characters "To:". In this way, "ABC Inc." is expressed as a named entity ORG. Note that the tag assignment format is not limited to the IOB2 format, and may be the IOB1 format or the IOE1 format.

[0014] <Explanation of tagging process> Next, an operation of the learning data generating unit 210 and the tag assigning unit 220 in the information processing device 100 of this embodiment, in which the learning data generating unit 210 assigns tag information assigned to existing learning data to the document data 231 to generate learning data 232, will be described.

[0015] FIG. 3 is a diagram showing an example of a UI screen for performing a tagging operation. Four areas are displayed on the UI screen. The left area is called a tool pane 301, the center area is called an editing pane 302, the lower right area is called a tagger search pane 303, and the upper right area is called a tagger preview pane 304. The tool pane 301 has a tag tool button 310 for each named entity tag to be assigned to the input document. The operator can select a character string of the input document image displayed in the editing pane 302 and select a tag tool button 310 according to the type of named entity tag to be assigned, thereby assigning a tag to the selected character string. The editing pane 302 displays an input document image to which a named entity tag is to be assigned. The tagger search pane 303 can search for tagged learning data (a group of tagged document images) similar to the document to be tagged displayed in the editing pane 302, and display a list of the data. The tagger preview pane 304 displays a preview of the learning data (reference document) selected by the operator from the list in the tagger search pane 303, and the operator can confirm the document content (reference document image) of the selected learning data and the assigned tags.

[0016] 4 is a flowchart showing the overall process of the information processing device 100 generating the learning data 232. A program of the information processing device 100 according to this flow is stored in the ROM 103, read into the RAM 104, and executed by the CPU 102. In S401, the information processing device 100 determines an operation by an operator and performs a process corresponding to the operation.

[0017] Hereinafter, the processing will be described in accordance with the order of the operator's operations with reference to Figs. 3 and 4. Details of each processing will be described later. First, when the operator presses the "Open" button 320, the CPU 102 of the information processing device 100 performs an input document reading process (S402). Next, when the operator inputs a document folder name 330 and then presses the "Search" button 332, the CPU 102 of the information processing device 100 performs a search process (S403) for documents (hereinafter also referred to as "similar documents") whose similarity to the input document is equal to or greater than a predetermined value. When the operator selects one document from the similar document list 333, the CPU 102 of the information processing device 100 performs a reference document selection process (S404). When the reference document selection process (S404) is executed, the document data (reference document image) selected by the operator is displayed in the area 304.

[0018] When the operator confirms that the document is a reference document from the display of the reference document image and the assigned tag displayed in the area 304 and presses the "tag assignment" button 340, the CPU 102 of the information processing device 100 executes tag assignment processing (S405). If there is a tag for which no character string to which a tag can be assigned is found in the input document image by the tag assignment processing, a tag (tag frame) with only a frame and no character string is displayed. When the operator starts dragging the tag frame by operating the mouse, etc., the CPU 102 of the information processing device 100 executes tag frame drag start processing (S406). When the operator drops the dragged tag frame onto the character string in the input document image (drag and drop), the CPU 102 of the information processing device 100 executes tag frame drop processing (S407). When the operator presses the "save" button 321, the CPU 102 of the information processing device 100 executes save processing (S408). Finally, when tagging is complete and the operator presses the "Close" button 322, the CPU 102 of the information processing device 100 performs a close process (S409) and ends the process for the input document image. When the process for the input document image is ended, learning data 232 is generated. If the operator performs another operation, such as pressing the tag tool button 310, the CPU 102 of the information processing device 100 performs other processes (S410). Here, the learning data generation unit 210 shown in the functional configuration of the information processing device 100 performs an input document reading process, a saving process, and a closing process. The tag assignment unit 220 performs a similar document search process, a reference document selection process, a tag assignment process, a tag frame drag start process, and a tag frame drop process.

[0019] 5 is a detailed flowchart of the input document reading process (S402) executed by the CPU 102 of the information processing device 100. First, the CPU 102 of the information processing device 100 stores learning data of a document currently being tagged in the storage device 105 by a storage process (S408) described later. In S501, the CPU 102 of the information processing device 100 displays a screen for accepting a document to which a named entity tag is to be assigned on the display control device 112, and the process proceeds to S502. In S502, the CPU 102 of the information processing device 100 determines whether a document (input document) has been accepted on the displayed document acceptance screen. If the document has been accepted on the document acceptance screen, the process proceeds to S503. If the document has not been accepted on the document acceptance screen, the process of the flowchart shown in FIG. 5 ends.

[0020] In S503, the CPU 102 of the information processing device 100 reads the accepted document, and the process proceeds to S504. In S504, the CPU 102 of the information processing device 100 determines whether an error occurred when reading the accepted document. If an error occurred, the process proceeds to S505. If no error occurred, the process proceeds to S506.

[0021] In S505, CPU 102 of information processing device 100 performs error processing, and the processing of the flowchart shown in Fig. 5 ends. In S506, CPU 102 of information processing device 100 displays the contents of the read input document as an input document image in edit pane 302, and the processing proceeds to S507. In S507, CPU 102 of information processing device 100 turns OFF the changed flag stored in RAM 104, and the processing of the flowchart shown in Fig. 5 ends. If the changed flag is ON, it indicates that there has been a change in the tag setting assigned to the input document image. If the changed flag is OFF, it indicates that there has been no change in the tag setting.

[0022] 6 is a detailed flowchart of the similar document search process (S403) executed by the CPU 102 of the information processing device 100. First, in S601, the CPU 102 of the information processing device 100 determines whether the input document to which tags are to be added in S402 has been opened and is being edited, and whether a folder for searching for documents similar to the input document has been set. The folder for searching for documents similar to the document being edited is set by the user inputting a folder name in the area 330 of FIG. 3. If the input document is not being edited in S601 or a folder for searching for documents similar to the input document has not been set, the process of the flowchart shown in FIG. 6 ends. If the document to which tags are to be added is being edited in S601, and a folder for searching for documents similar to the input document has been set, the process proceeds to S602.

[0023] In S602, the CPU 102 of the information processing device 100 searches for learning data from the set document search folder and determines whether the learning data is found. If the check box 331 in FIG. 3 is ON, subfolders of the folder specified in the area 330 are also searched recursively. If the learning data is not found, the process of the flowchart shown in FIG. 6 ends. If the learning data is found, the process proceeds to S604. In S604, the CPU 102 of the information processing device 100 calculates a feature vector of the input document, and the process proceeds to S605. The document feature vectorization process uses, for example, the Doc2Vec method of natural language processing.

[0024] The loop process from S605 to S611 is performed for the number of pieces of found learning data. In S606, the CPU 102 of the information processing device 100 calculates the feature vector of the found learning data, and the process proceeds to S607. In S607, the CPU 102 of the information processing device 100 calculates the similarity of the feature vectors of the input document and the learning data, and the process proceeds to S608. In S608, the CPU 102 of the information processing device 100 determines whether the found learning data is similar to the input document. If it is determined that the found learning data is similar to the input document, the process proceeds to S609. If it is determined that the found learning data is not similar to the input document, the process proceeds to S610. The similarity is calculated using the cosine similarity of the feature vectors. That is, if the feature vectors of the text A and the text B are VecA and VecB, respectively, the cosine similarity can be calculated as the inner product of VecA and VecB / (norm of VecA*norm of VecB). The cosine similarity takes a value from -1 to 1, with 1 representing the most similar (identical). In the determination in S608, if the similarity is equal to or greater than a predetermined value, it is determined that the learning data is similar to the input document. In S609, the CPU 102 of the information processing device 100 adds the found learning data to a similar document list stored in the RAM 104, and the process proceeds to S610. FIG. 7 shows an example of the similar document list. The similar document list is composed of the document name of the learning data and the folder name in which the document is stored. In S610, the CPU 102 of the information processing device 100 determines whether the processes from S606 to S609 have been performed on all the learning data found in S602. If the processes from S606 to S609 have been performed on all the learning data, the process proceeds to S611. If the processes from S606 to S609 have not been performed on all the learning data, the process returns to S605. In S611, the CPU 102 of the information processing apparatus 100 displays the similar documents list in the area 333 in FIG. 3, and the process of the flowchart shown in FIG. 6 ends.

[0025] FIG. 8 is a detailed flowchart of the reference document selection process (S404) executed by the CPU 102 of the information processing device 100. First, in S801, the CPU 102 of the information processing device 100 reads a similar document (reference document) selected from the area 333 displaying the similar document list, and the process proceeds to S802. In S802, the CPU 102 of the information processing device 100 performs an error determination process. If an error occurs, the process proceeds to S803. If no error occurs, the process proceeds to S804. In S803, the CPU 102 of the information processing device 100 performs error processing. When the error processing is completed, the process of the flowchart shown in FIG. 8 ends. In S804, the CPU 102 of the information processing device 100 displays the reference document image and the tag information added, which are the contents of the similar document selected in S801, in the tag source preview pane 304. This enables a comparison between the input document and the reference document. When the process of S804 is completed, the process of the flowchart shown in FIG. 8 ends.

[0026] FIG. 9 shows an example of tag information added to a reference document. The tag information is composed of a block ID 901, a block area 902, a tag ID 903, a character 904, a tag 905, and a character area 906. The tag information is added for each character. The block ID 901 is an identifier that uniquely identifies the character block to which the tag is added. The original document to which the tag is added is composed of blocks for each area such as characters, images, and tables, and the tag information is attached to the character block to which the tag is added. The block area 902 is the coordinates of the upper left and lower right points of the area of ​​the block to which the tag is added. The tag ID 903 is an identifier that uniquely identifies the tag of the named entity string. In the IOB2 format, the B-<arbitrary character string> tag is added to the first character of the named entity string, and the I-<arbitrary character string> tag is added to the second and subsequent characters, but the same tag ID is set for the first character to the last character of one named entity string. The character 904 is the tagged character, and the tag 905 is a tag in the IOB2 format. A character area 906 is an area occupied by tagged characters 904. The tag is displayed by surrounding the character string to which the tag is set with a solid line frame as in area 341 in Fig. 3, and displaying the tag name at the top as in area 342.

[0027] FIG. 10 is a detailed flowchart of the tag assignment process (S405) executed by the CPU 102 of the information processing device 100. The loop process from S1001 to S1011 is repeated for the number of tags. In S1002, a character block of the input document image in the vicinity of the tag area of ​​each tag included in the reference document image displayed in the tag source preview pane 304 is searched for, and the process proceeds to S1003. The vicinity of the tag area means that the area of ​​the block overlaps even a little with an area obtained by expanding the area occupied by the character string to which the tag is assigned by a predetermined length in all directions. The area occupied by the character string to which the tag is assigned is a circumscribed rectangular area including all the character areas 906 of each tag information for multiple characters having the same tag ID 903. In S1003, the CPU 102 of the information processing device 100 determines whether or not a character block is found in the vicinity of the tag area of ​​the reference document image in the input document image. If a character block is found, the process proceeds to S1004. If a character block is not found, the process proceeds to S1009.

[0028] In S1004, the CPU 102 of the information processing device 100 extracts a character block of a character string that is most similar to the tag character string of the reference document image, and the process proceeds to S1005. The details of the extraction process will be described later. In S1005, the CPU 102 of the information processing device 100 judges whether the similarity of the extracted character block is equal to or greater than a predetermined value. If the similarity is equal to or greater than the predetermined value, the process proceeds to S1006. If the similarity is less than the predetermined value, the process proceeds to S1009. In S1006, the CPU 102 of the information processing device 100 assigns the tag of the reference document image to the extracted character block, and the process proceeds to S1007. In S1007, the CPU 102 of the information processing device 100 displays the assigned tag, and the process proceeds to S1008. In S1007, similar to the tag display of the reference document image in S804, the character string to which the tag is set is surrounded by a solid-line frame as displayed in the editing pane 302 of FIG. 3, and the tag name is displayed at the top. In S1008, the CPU 102 of the information processing device 100 sets the changed flag to ON, and the process proceeds to S1011.

[0029] In S1009, the CPU 102 of the information processing device 100 creates a tag (tag frame) that is only a frame with no character string, and the process proceeds to S1010. In S1010, the CPU 102 of the information processing device 100 displays the tag frame, and the process proceeds to S1011. The tag frame is a character block with an empty character string created, and tag frame information is added to the block. FIG. 11 shows an example of tag frame information. The tag frame information is obtained by adding a source block ID 1101 and a source tag ID 1102 to the tag information. In the tag frame information added to the tag frame generated in S1009, the character 904 is empty, the tag 905 is the same as the tag information of the reference document image, and the character area 906 is an area obtained by adding the character areas of each character of the tag information of the reference document image. In addition, the source block ID 1101 and the source tag ID 1102 can calculate a feature vector of the tag string of the reference document image, similar to the block ID 901 and the tag ID 903 of the reference document image. The tag frame is displayed by surrounding the character string to which the tag is set with a dashed frame as in area 325 in Fig. 3, and displaying the tag name above area 325 as in area 326. In S1011, CPU 102 of information processing device 100 determines whether the processes from S1002 to S1010 have been executed for all tag information included in the reference document image. If the processes have not been executed for all tag information, the process returns to S1001. If the processes have been executed for all tag information, the process of the flowchart shown in Fig. 10 ends.

[0030] 12 is a detailed flowchart of the most similar character block extraction process (S1004). In S1201, the CPU 102 of the information processing device 100 calculates a feature vector of the tag character string of the reference document image, and the process proceeds to S1202. The feature vectorization of the character string is performed, for example, using the Word2Vec method of natural language processing. In S1202, the CPU 102 of the information processing device 100 calculates a feature vector of the character string of all the character blocks found in S1002, and the process proceeds to S1203. In S1203, the CPU 102 of the information processing device 100 calculates the feature vector calculated in S1201 and its similarity, and the process proceeds to S1204. In S1204, the CPU 102 of the information processing device 100 extracts the character block having the highest similarity from all the character blocks found in S1002.

[0031] FIG. 13 is a detailed flowchart of the tag frame drag start process (S406) executed by the CPU 102 of the information processing device 100. In S1301, the CPU 102 of the information processing device 100 searches for a tag frame that exists at the drag start coordinate, and the process proceeds to S1302. In S1302, the CPU 102 of the information processing device 100 judges whether a tag frame exists. If the tag frame does not exist, the process of the flowchart shown in FIG. 13 ends. If the tag frame exists, the process proceeds to S1303. In S1303, the CPU 102 of the information processing device 100 generates moving tag frame information and stores the moving tag frame information in the RAM 104). FIG. 14 is an example of moving tag frame information. The moving tag frame information is generated by copying the block ID 901 of the tag frame, the block area 902 of the tag frame, the source block ID 1101, and the source tag ID 1102.

[0032] Fig. 15 is a detailed flowchart of the tag frame drop process (S407) executed by the CPU 102 of the information processing device 100. In S1501, the CPU 102 of the information processing device 100 determines whether moving tag frame information exists in the RAM 104. If moving tag frame information does not exist, the process of the flowchart shown in Fig. 15 ends. If moving tag frame information exists, the process proceeds to S1502. In S1502, the CPU 102 of the information processing device 100 calculates a feature vector of the tag string of the reference document image from the source block ID 1101 and source tag ID 1102 of the moving tag frame information, and the process proceeds to S1503.

[0033] In S1503, CPU 102 of information processing device 100 searches for a character block in the top layer that exists at the drop coordinate, and the process proceeds to S1504. In S1504, CPU 102 of information processing device 100 determines whether a block exists. If the block does not exist, the process proceeds to S1512. If the block exists, the process proceeds to S1505. In S1505, CPU 102 of information processing device 100 calculates a feature vector of a character string of the found character block, and the process proceeds to S1506. In S1506, CPU 102 of information processing device 100 calculates a similarity between the feature vector calculated in S1502 and the feature vector calculated in S1505, and the process proceeds to S1507.

[0034] In S1507, the CPU 102 of the information processing device 100 determines whether the calculated similarity is equal to or greater than a predetermined value. If the calculated similarity is equal to or greater than the predetermined value, the process proceeds to S1508. If the calculated similarity is less than the predetermined value, the process proceeds to S1512. In S1508, the CPU 102 of the information processing device 100 assigns a tag of the reference document image to the drop destination text block, and the process proceeds to S1509. In S1509, the CPU 102 of the information processing device 100 displays the assigned tag, and the process proceeds to S1510. In S1509, the display color of the character string to which the tag is assigned and the assigned tag may be a color different from the display color before the tag is displayed.

[0035] In S1510, the CPU 102 of the information processing device 100 deletes the tag frame block, and the process proceeds to S1511. In S1511, the CPU 102 of the information processing device 100 sets the changed flag to ON, and the process proceeds to S1512. In S1512, the CPU 102 of the information processing device 100 deletes the moving tag frame information (S1012), and the process of the flowchart shown in FIG. 15 ends.

[0036] FIG. 16 is a detailed flowchart of the save process (S408) executed by the CPU 102 of the information processing device 100. In S1601, the CPU 102 of the information processing device 100 determines whether the changed flag is ON. If the changed flag is OFF, the process of the flowchart shown in FIG. 16 ends. If the changed flag is ON, the process proceeds to S1602. In S1602, the CPU 102 of the information processing device 100 saves the input document image, and the process proceeds to S1603. When saving the input document image, if a tag frame exists, the input document image is saved without the tag frame. In S1603, the CPU 102 of the information processing device 100 sets the edited flag to OFF, and the process of the flowchart shown in FIG. 16 ends.

[0037] Fig. 17 is a detailed flowchart of the close process (S409) executed by the CPU 102 of the information processing device 100. The CPU 102 of the information processing device 100 performs the above-mentioned save process (S408), closes the screen (S1701), and ends the process of the flowchart shown in Fig. 17. In this embodiment, it is possible to assign the assigned tag to the entire character block. This makes it possible to efficiently assign tags even to semi-standard documents.

[0038] [Second embodiment] A second embodiment according to the present disclosure will be described. In the following description, only differences from the first embodiment will be described. In the first embodiment, an example of a case where a tag is assigned to an entire character block, such as "invoice" and "ABC Co., Ltd." in FIG. 3, was described. In the second embodiment, a case where a tag assigned to a part of a character block, such as a long sentence, is assigned to a part of a character string in an input document image is described. FIG. 18 is a schematic diagram of a character string extraction process when a tag assigned to a part of a character string in a character block of a reference document image is assigned to a similar character string in a character block of an input document image. The character string extraction process searches for a character string that has a similar character string immediately before the tag character string, and extracts a character string that has a similar character string starting from the next word of the immediately before character string.

[0039] 19 is a flowchart of the tagging process when a tag is assigned to a part of a character block. The basic operation is the same as the tagging process of S405, but the process of extracting the most similar character block (S1004) is replaced with a process of extracting a part of a character string within the block (S1901).

[0040] 20 is a detailed flowchart of the process of extracting a character string in a most similar block (S1901). In S2001, the CPU 102 of the information processing device 100 calculates a feature vector of the immediately preceding character string in the tag character string of the reference document image, and the process proceeds to S2002. In S2002, the CPU 102 of the information processing device 100 calculates a feature vector of the tag character string of the reference document image, and the process proceeds to S2003. The loop process from S2003 to S2007 is performed for all character blocks found in the process of S1002.

[0041] In S2004, the CPU 102 of the information processing device 100 extracts a "character string immediately preceding the tag character string" and a "tag candidate character string" similar to the reference document image from the character block found in the process of S1002, and the process proceeds to S2005. In S2005, the CPU 102 of the information processing device 100 determines whether the similarity of the feature vector of the immediately preceding character string and the similarity of the feature vector of the tag candidate character string are greater than the maximum value so far. If the similarity of the feature vector of the immediately preceding character string and the similarity of the feature vector of the tag candidate character string are greater than the maximum value so far, the process proceeds to S2006. If the similarity of the feature vector of the immediately preceding character string and the similarity of the feature vector of the tag candidate character string are less than the maximum value so far, the process proceeds to S2007. In S2006, the CPU 102 of the information processing device 100 updates the most similar character string information, and the process proceeds to S2007. FIG. 21 shows an example of the most similar character string information. The most similar character string information includes a block ID 2101, a character string range 2102, a similarity 2103 of the immediately preceding character string, and a character string similarity 2104. The determination in S2005 is performed based on whether or not the product of the similarity of the immediately preceding character string of the extraction result in S2004 and the similarity of the tag candidate character string is greater than the product of the similarity 2203 of the immediately preceding character string of the most similar character string information and the character string similarity 2204. The determination may be performed by other methods such as a weighted product. In S2007, the CPU 102 of the information processing device 100 determines whether the processes from S2004 to S2006 have been performed for all the character blocks found in the process of S1002. If the processes have been performed for all the character blocks found in the process of S1002, the character block and character string of the most similar character string information are output as the extraction result, and the process of the flowchart shown in FIG. 20 ends. If the processes have not been performed for all the character blocks found in the process of S1002, the process returns to S2003.

[0042] FIG. 22 is a detailed flowchart of the process of extracting similar character strings in a block (S2004). In S2201, the CPU 102 of the information processing device 100 divides the character block into words, and the process proceeds to S2202. The loop process from S2202 to S2212 is repeated the number of times that i is the number of words divided in the process of S2201. In S2203, the CPU 102 of the information processing device 100 calculates a feature vector of a character string immediately before the i-th word (a character string from the beginning to the i-1th word), and the process proceeds to S2204. In S2204, the CPU 102 of the information processing device 100 calculates a similarity between the feature vector calculated in S2203 and the feature vector calculated in S2001, and the process proceeds to S2205. In S2205, the CPU 102 of the information processing device 100 determines whether the similarity calculated in S2204 (similarity of the immediately preceding character string) is equal to or greater than a predetermined value. If the similarity is less than the predetermined value, the process proceeds to S2212. If the similarity is equal to or greater than the predetermined value, the process proceeds to S2206.

[0043] The loop process from S2206 to S2211 is repeated until j becomes the number of words divided in the process from i to S2201. In S2207, the CPU 102 of the information processing device 100 calculates a feature vector using the i-th to j-th words as tag candidate character strings, and the process proceeds to S2208. In S2208, the CPU 102 of the information processing device 100 calculates the similarity between the feature vector calculated in S2207 and the feature vector calculated in S2002, and the process proceeds to S2209. In S2209, the CPU 102 of the information processing device 100 determines whether the similarity calculated in S2208 (similarity of tag candidate character strings) is equal to or greater than a predetermined value. If the similarity is less than the predetermined value, the process proceeds to S2211. If the similarity is equal to or greater than the predetermined value, the process proceeds to S2210.

[0044] In S2210, the CPU 102 of the information processing device 100 outputs the j-th character string from i to j, the similarity of the immediately preceding character string calculated in S2204, and the similarity of the tag candidate character string calculated in S2208 as extraction results. When the extraction results are output, the process of the flowchart shown in FIG. 22 ends. In S2211, if j is the number of words divided in the process of S2201, the process proceeds to S2212. If j is not the number of words divided in the process of S2201, the process returns to S2206. In S2212, if i is the number of words divided in the process of S2201, the process proceeds to S2213. If i is not the number of words divided in the process of S2201, the process returns to S2202. In S2213, the CPU 102 of the information processing device 100 outputs a null extraction result. When the extraction results are output, the process of the flowchart shown in FIG. 22 ends. In this embodiment, since tags are assigned to parts of character blocks such as long sentences, it is possible to efficiently assign tags even to semi-standard documents.

[0045] [Third embodiment] A third embodiment according to the present disclosure will be described. In the following description, only the points different from the first embodiment will be described. In the first embodiment, it was shown that tags can be semi-automatically added from a reference document image. In the third embodiment, tags used in a reference document image can be identified. This makes it possible to improve the efficiency of tagging work even when tags are added manually from a reference document image.

[0046] FIG. 23 is a flowchart relating to the selection process of the second reference document in this embodiment. The process of S2301 is performed following the process of S804. In S2301, the CPU 102 of the information processing device 100 extracts tags used from the learning data read in S801, and the process proceeds to S2302. In S2302, the CPU 102 of the information processing device 100 changes the display of the tag tool button 310 shown in FIG. 3 to reflect the extraction result. When the display of the tag tool button 310 is changed, the process of the flowchart shown in FIG. 23 ends. The display change of the tag tool button 310 may be, for example, to display the button of the used tag in a different color. Alternatively, the tag tool button 310 may be displayed in order of frequency of use. This allows the user to visually recognize the tags used, making it possible to efficiently assign tags even to semi-standard documents.

[0047] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0048] The disclosure of the above-described embodiment includes the following configurations.

[0049] (Configuration 1) An information processing device that performs a process of assigning named entity tags to document images, comprising: a receiving means for receiving an input document image; a selection means for selecting a reference document image to be compared with the input document image from a group of document images to which named entity tags have already been assigned; an extraction means for extracting from the input document image a character block corresponding to a character string to which the named entity tag has been assigned in the reference document image selected by the selection means; and an assignment means for assigning a named entity tag to a character string represented by the extracted character block based on the similarity between a feature vector of the character string represented by the extracted character block and a feature vector of a character string to which the named entity tag has been assigned in the reference document image.

[0050] (Configuration 2) The information processing apparatus according to configuration 1, wherein the reference document image has a similarity to the input document image that is equal to or greater than a predetermined value.

[0051] (Configuration 3) The information processing device according to configuration 1 or 2, further comprising: a display control means for displaying tag information of a character string represented by the extracted character block on a UI screen.

[0052] (Configuration 4) The information processing device according to configuration 3, wherein the display control means displays only a frame for the named entity tag from which the character string is not extracted.

[0053] (Configuration 5) An information processing device as described in configuration 4, characterized in that when the frame is dragged and dropped onto the corresponding character block on the UI screen, the assignment means assigns the named entity tag to the character string represented by the character block.

[0054] (Configuration 6) The information processing device described in Configuration 5, characterized in that when the assignment means assigns the named entity tag to the character string represented by the character block by drag and drop, the display control means displays tag information of the character string represented by the character block on the UI screen.

[0055] (Configuration 7) The information processing device according to configuration 6, wherein the display control means displays the character block and its tag information in a color different from the display color before the named entity tag is assigned.

[0056] (Configuration 8) The information processing device described in Configuration 1, wherein the assignment means assigns the named entity tag to the first string based on a similarity between a feature vector of a first string representing a part of the character block extracted by the extraction means and a feature vector of a second string to which the named entity tag has been assigned in the reference document image, and a similarity between the feature vector of a string immediately preceding the first string and the feature vector of a string immediately preceding the second string.

[0057] (Configuration 9) The information processing device according to any one of configurations 3 to 7, wherein the display control means displays a list of the named entity tags to be used on the UI screen.

[0058] (Configuration 10) An information processing device as described in Configuration 9, further comprising a memory means for storing the frequency of use of the named entity tags, wherein the display control means displays the named entity tags in the list of named entity tags in order of frequency of use.

[0059] (Configuration 11) The information processing device described in Configuration 10, characterized in that the display control means displays the display colors of the named entity tags that are being used and the display colors of the named entity tags that are not being used in the list of named entity tags in different colors.

[0060] (Configuration 12) A control method for an information processing device that performs a process of assigning named entity tags to document images, comprising the steps of: accepting an input document image; selecting a reference document image to be compared with the input document image from a group of document images to which named entity tags have already been assigned; extracting from the input document image a character block corresponding to a character string to which the named entity tag has been assigned in the reference document image selected in the selection step; and assigning a named entity tag to the character string represented by the extracted character block based on the similarity between a feature vector of the character string represented by the extracted character block and a feature vector of a character string to which the named entity tag has been assigned in the reference document image.

Claims

1. An information processing device that performs a process of adding a named entity tag to a document image, A receiving means for receiving an input document image; a selection means for selecting a reference document image to be compared with the input document image from among a group of document images to which named entity tags have already been added; an extraction means for extracting from the input document image a character block corresponding to a character string to which the named entity tag is added in the reference document image selected by the selection means; an assignment means for assigning a named entity tag to a character string represented by the extracted character block based on a similarity between a feature vector of the character string represented by the extracted character block and a feature vector of a character string to which a named entity tag has been assigned in the reference document image; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the reference document image has a similarity to the input document image that is equal to or greater than a predetermined value.

3. 2. The information processing apparatus according to claim 1, further comprising: a display control means for displaying tag information of a character string represented by the extracted character block on a UI screen.

4. 4. The information processing apparatus according to claim 3, wherein the display control means displays only a frame for the named entity tag from which the character string is not extracted.

5. 5. The information processing device according to claim 4, wherein when the frame is dragged and dropped onto the corresponding character block on the UI screen, the assignment means assigns the named entity tag to the character string represented by the character block.

6. The information processing device according to claim 5, characterized in that, when the assignment means assigns the named entity tag to the character string represented by the character block by drag and drop, the display control means displays tag information of the character string represented by the character block on the UI screen.

7. 7. The information processing apparatus according to claim 6, wherein the display control means displays the character block and its tag information in a color different from the display color before the named entity tag is added.

8. The information processing device according to claim 1, characterized in that the assignment means assigns the named entity tag to the first string based on a similarity between a feature vector of a first string representing a part of the character block extracted by the extraction means and a feature vector of a second string to which the named entity tag has been assigned in the reference document image, and a similarity between a feature vector of a string immediately preceding the first string and a feature vector of a string immediately preceding the second string.

9. The information processing apparatus according to claim 3 , wherein the display control means displays a list of the named entity tags to be used on the UI screen.

10. A storage means for storing a frequency of use of the named entity tag, The information processing apparatus according to claim 9 , wherein the display control means displays the named entity tags in the list of named entity tags in order of frequency of use.

11. The information processing device according to claim 10, characterized in that the display control means displays the named entity tags that are being used and the named entity tags that are not being used in the list of named entity tags in different colors.

12. A method for controlling an information processing device that performs a process of adding a named entity tag to a document image, comprising: accepting an input document image; selecting a reference document image from among document images to which named entity tags have already been assigned, for comparison with the input document image; extracting, from the input document image, a character block corresponding to a character string to which the named entity tag is assigned in the reference document image selected in the selecting step; a step of assigning a named entity tag to the character string represented by the extracted character block based on a similarity between a feature vector of the character string represented by the extracted character block and a feature vector of a character string to which a named entity tag has been assigned in the reference document image; 13. A method for controlling an information processing device, comprising:

13. A program for causing a computer to function as the information processing device according to any one of claims 1 to 11.

Images