Document identification device, document identification system, program, and document identification method

Abstract

In a document recognition technology that combines image features and linguistic features, the calculation of linguistic features is made simpler and more resource-efficient, without generating rotated images from multiple angles or storing positional information of keyword strings. [Solution] The system comprises an image feature calculation unit that calculates image features of a target form image, a language feature calculation unit that calculates language features of the form image, an image similarity calculation unit that calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with image features of pre-registered forms, a language similarity calculation unit that calculates a similarity score of the language features calculated by the language feature calculation unit by comparing them with language features of pre-registered forms, an integration unit that integrates the similarity score of the image features and the similarity score of the language features, and an identification unit that identifies the type of form based on the integrated similarity, wherein the language feature calculation unit utilizes expressions specific to the form in relation to the calculation of language features.

Description

Technical Field

[0001] The present invention relates to a document identification device, a document identification system, a program, and a document identification method.

Background Art

[0002] Conventionally, in document identification technology for identifying the types of documents such as invoices, it has already been known that hybrid processing using image features of the document and language features obtained from the OCR (Optical Character Recognition / Reader) processing result of the document image is utilized.

[0003] Patent Document 1 discloses a technique for calculating the language features of a document by first generating a rotated image and calculating the language feature amount from the rotated image. Further, Patent Document 1 also discloses using the position information of a character string that becomes a specific keyword in the document when calculating the language features of the document.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the conventional document identification technology, in order to calculate the language features, it is necessary to store the position information of the character string that becomes the keyword, and it is also necessary to generate images with a plurality of different angles. Therefore, there is a problem that the processing is complicated and a large amount of resources are required.

[0005] The present invention has been made in view of the above, and in a document identification technology that combines image features and language features, regarding the calculation of language features, it is an object to perform the calculation of language features more simply and with less resources without generating rotated images with a plurality of different angles or storing the position information of the character string that becomes the keyword.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the present invention comprises: an image feature calculation unit that calculates image features of a target form image; a language feature calculation unit that calculates language features of the form image; an image similarity calculation unit that calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with image features of pre-registered forms; a language similarity calculation unit that calculates a similarity score of the language features calculated by the language feature calculation unit by comparing them with language features of pre-registered forms; an integration unit that integrates the similarity score of the image features and the similarity score of the language features; and an identification unit that identifies the type of form based on the integrated similarity, wherein the language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. [Effects of the Invention]

[0007] According to the present invention, in the calculation of language features in a document recognition technology that combines image features and language features, the calculation of language features can be performed more easily and with fewer resources, without generating rotated images at multiple angles or storing positional information of keyword strings. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of a document identification system including a document identification device according to the first embodiment. [Figure 2] Figure 2 is a diagram illustrating the overall operation flow of a document identification system, including a document identification device. [Figure 3] Figure 3 shows an example of the hardware configuration of a document identification device. [Figure 4] Figure 4 shows an example of a form. [Figure 5] Figure 5 shows an example of the configuration of a functional block in a document identification system, including a document identification device. [Figure 6] Figure 6 shows an example of the operation of a functional block in the document identification device. [Figure 7]Figure 7 shows an example of how the identification results are displayed. [Figure 8] Figure 8 shows another example of how the identification results are displayed. [Figure 9] Figure 9 is a flowchart showing an example of the flow of the document identification process of a document identification device. [Figure 10] Figure 10 is a flowchart showing an example of the dictionary registration process for a document identification device. [Figure 11] Figure 11 shows an example of document identification in a document identification device. [Figure 12] Figure 12 shows an example of document identification in a document identification device, specifically example 2. [Figure 13] Figure 13 shows an example of document identification in a document identification device, specifically example 3. [Figure 14] Figure 14 shows an example of the configuration of a functional block in a document identification system including a document identification device according to the second embodiment. [Figure 15] Figure 15 shows an example of the operation of the functional block of the document identification unit of the document identification device. [Modes for carrying out the invention]

[0009] Embodiments of the document identification device, document identification system, program, and document identification method will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the following embodiments, and the components in the following embodiments include those easily conceivable by those skilled in the art, those substantially identical, and those within the scope of so-called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of components can be made without departing from the spirit of the following embodiments.

[0010] Also, in the following, as an example of an information processing apparatus that scans documents such as claim forms, a multifunction peripheral (MFP) will be taken as an example for explanation, but it is not limited thereto. For example, as an information processing apparatus that scans documents, a normal scanner device instead of a multifunction peripheral may be used. Note that a multifunction peripheral (MFP) is a device having a plurality of different functions such as a copying function, a scanner function, a printer function, and a fax function.

[0011] (First Embodiment) FIG. 1 is a diagram showing an example of the configuration of a document identification system 1 including a document identification apparatus 10 according to the first embodiment. FIG. 2 is a diagram for explaining the overall operation flow of the document identification system 1 including the document identification apparatus 10. The system configuration and the outline of the operation of the document identification system 1 including the document identification apparatus 10 will be described while referring to FIGS. 1 and 2.

[0012] As shown in FIG. 1, the document identification apparatus 10 is connected via a network N so as to enable data communication with the multifunction peripheral 20.

[0013] The multifunction peripheral 20 is an example of an information processing apparatus that scans documents. In the present embodiment, the multifunction peripheral 20 functions as a device that reads (scans) a document 121 to be identified and transmits the document image of the read document 121 to the document identification apparatus 10.

[0014] The document identification apparatus 10 is a device that performs OCR (Optical Character Recognition / Reader) processing on the document image of the document 121 read by the multifunction peripheral 20 and identifies the document 121 based on the obtained character information and the like.

[0015] Next, while referring to FIG. 2, an overview of the overall operation flow of the form identification system 1 including the form identification device 10 will be described. First, as described above, in the multifunction device 20, a form 121 to be identified is set, and a reading operation (scanning operation) for the form 121 is performed ((1) Form scanning). The form image of the form 121 obtained by the reading operation is transmitted to the form identification device 10 via the network N.

[0016] Next, in the form identification device 10, OCR processing for reading various characters included in the form image received from the multifunction device 20 is performed ((2) OCR processing). Then, based on the result of the OCR processing, the form identification device 10 obtains language features from the character information, which is the information of the read characters ((3) Language feature calculation processing).

[0017] Next, in the form identification device 10, using the form image received from the multifunction device 20, image features of the form 121 are obtained ((4) Image feature calculation processing).

[0018] Next, in the form identification device 10, form identification processing for determining which form the form is identified as is executed using the image features and language features of the form 121 ((5) Form identification processing).

[0019] Then, in the form identification device 10, based on the character information of the form and the identification result by the form identification processing, only the filled (input) part of the form 121 is extracted to generate data in CSV (Comma-Separated Values) format ((6) Conversion to CSV). The CSV format data obtained by this processing is used for accounting processing in the subsequent accounting system or the like.

[0020] Subsequently, the hardware configuration of the form identification device 10 will be described.

[0021] Figure 3 shows an example of the hardware configuration of the document identification device 10. As shown in Figure 3, the document identification device 10 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an auxiliary storage device 205, a media drive 207, a display 208, a network interface 209, a keyboard 211, a mouse 212, a DVD (Digital Versatile Disc) drive 214, and an external interface 215.

[0022] The CPU 201 is an arithmetic unit that controls the overall operation of the document identification device 10. The ROM 202 is a non-volatile memory device that stores the program for document identification in the document identification device 10. The RAM 203 is a volatile memory device used as the work area of ​​the CPU 201.

[0023] The auxiliary storage device 205 is a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) that stores, for example, image data of the document 121 identified by the document identification process. The media drive 207 is a device that controls the reading and writing of data to the recording medium 206, such as flash memory, according to the control of the CPU 201.

[0024] The display 208 is a display device composed of liquid crystal or organic EL (Electro-Luminescence) that displays various information such as cursors, menus, windows, characters, or images. For example, the display 208 displays the identification result from the identification unit 107 (see Figure 5), which will be described later.

[0025] Network I / F209 is an interface for communicating data with external devices such as the multifunction printer 20 via network N. Network I / F209 is, for example, a NIC (Network Interface Card) that supports Ethernet (registered trademark) and is capable of communication compliant with TCP (Transmission Control Protocol) / IP (Internet Protocol), etc.

[0026] The keyboard 211 is an input device for selecting characters, numbers, various instructions, and moving the cursor. The mouse 212 is an input device for selecting and executing various instructions, selecting the object to be processed, and moving the cursor.

[0027] The DVD drive 214 is a device that controls the reading and writing of data to DVD 213, such as DVD-ROM or DVD-R (Digital Versatile Disk Recordable), which are examples of removable storage media.

[0028] The external I / F215 is an interface compatible with USB (Universal Serial Bus) and other protocols, which allows connection of touch panels or touchpads that enable touch input using a stylus such as a touch pen or a human finger.

[0029] The aforementioned CPU 201, ROM 202, RAM 203, auxiliary storage device 205, media drive 207, display 208, network interface 209, keyboard 211, mouse 212, DVD drive 214, and external interface 215 are connected to each other via a bus 210, including an address bus and a data bus.

[0030] Note that the hardware configuration of the document identification device 10 shown in Figure 3 is an example, and it is not necessary to include all the components shown in Figure 3, or other components may be included. Furthermore, the document identification device 10 is not limited to being composed of a single information processing device as shown in Figure 3, but may be composed of multiple network devices such as multiple information processing devices.

[0031] Next, we will describe the document 121 used in the document identification process of the document identification device 10. Document 121 is an invoice.

[0032] Figure 4 shows an example of form 121. Form 121 shown in Figure 4 includes, for example, the recipient information, such as the company name and address 401. Form 121 also includes, for example, the sender information, such as the company name and department 402, the address 403, and the telephone number 404. Form 121 also includes, for example, a table 405 listing the billing items and payment bank account information 406.

[0033] Note that the form 121 shown in Figure 4 is just one example, and the format of the forms handled by the form identification process of the form identification device 10 is not limited to these. In other words, the format of the forms to be identified can vary widely.

[0034] Generally, when calculating linguistic features for document identification, strings such as the sender's company name and address, which are effective in identifying the document, are extracted. Typically, when extracting such strings, it is common to use regular expressions, which are methods for representing several strings in a single format. However, when extracting strings using regular expressions, unless rotated images from multiple angles are generated or the positional information of the keyword strings is stored, it is not possible to limit the extraction to strings such as the sender's company name and address, and strings such as the recipient's company name and address are also extracted, including data that is unnecessary for identifying the document.

[0035] As will be described in detail later, the document identification device 10 of this embodiment is designed to extract a target string from a single document image by, for example, using a Large Language Model (LLM). A Large Language Model (LLM) is a language model trained using a vast amount of text data to perform Natural Language Processing (NLP) tasks. Specifically, the Large Language Model (LLM) is used to extract various keywords described in the document 121.

[0036] Next, the configuration and operation of the functional blocks of the document identification device 10 will be described.

[0037] Figure 5 shows an example of the configuration of a functional block in the document identification system 1, including the document identification device 10. Figure 6 shows an example of the operation of the functional block in the document identification device 10.

[0038] As shown in Figure 5, the multifunction printer 20 has a scanning unit 21 that functions as an image generation unit. The scanning unit 21 is a functional unit that takes a form 121 as input and obtains a form image by performing a reading operation on the form 121. The scanning unit 21 is implemented by a scanner device built into the multifunction printer 20.

[0039] As shown in Figure 5, the document identification device 10 comprises a calculation unit 100 and a storage unit 110. The calculation unit 100 and the storage unit 110 are realized, for example, by the execution of a program by the CPU 201 shown in Figure 3. The calculation unit 100 and the storage unit 110 are formed in RAM 203 or auxiliary storage device 205.

[0040] The memory unit 110 includes an image feature memory unit 111, a language feature memory unit 112, and a setting information memory unit 113.

[0041] In this embodiment, it is necessary to register dictionary information in advance in order to determine whether or not the form 121 has been pre-registered. Therefore, the image feature storage unit 111 registers the image features of the form 121 that have been extracted in advance as dictionary information. In addition, the language feature storage unit 112 registers the language features of the form 121 that have been extracted in advance as dictionary information.

[0042] The configuration information storage unit 113 stores the configuration information necessary for identification.

[0043] The calculation unit 100 includes an image feature calculation unit 101, a language feature calculation unit 102, an OCR processing unit 103, an image similarity calculation unit 104, a language similarity calculation unit 105, an integration unit 106, and an identification unit 107.

[0044] The image feature calculation unit 101 calculates image features from the form image of the input form 121. One example of calculating features specific to image layout information is a method that calculates image features using CLIP (Contrastive Language-Image Pre-Training).

[0045] The OCR processing unit 103 is a character recognition unit that performs OCR processing on the input form image 121 to recognize the characters written on it and extract text information (strings).

[0046] The language feature calculation unit 102 calculates language features from the text information (strings) recognized by the OCR processing unit 103. More specifically, the language feature calculation unit 102 uses large language models (LLMs) such as GPT4 or Gemini to calculate language features. By using LLMs, it becomes possible to accurately extract specific characters such as the sender's company name and address, resulting in improved performance. Here, an LLM is a computer language model composed of an artificial neural network with a large number of parameters. In an LLM, training is performed using a vast amount of unlabeled text through self-supervised learning or semi-supervised learning.

[0047] The language feature calculation unit 102, in calculating language features, does not generate rotated images at multiple angles or store positional information of keyword strings, but instead utilizes the document-specific representations in the document 121. These document-specific representations include the sender's address, sender's company name, sender's telephone number, payment bank account number, product delivery address, and business location. However, the document-specific representations are not limited to these.

[0048] The image similarity calculation unit 104 compares the image features of the form 121 calculated by the image feature calculation unit 101 with the image features of the form 121 registered in the image feature storage unit 111 to calculate the similarity of the image features. For example, the image feature storage unit 111 stores the image features of the form 121 up to the previous month and compares them with the image features of the form 121 for the current month calculated by the image feature calculation unit 101.

[0049] The language similarity calculation unit 105 calculates the similarity of language features by comparing the language features of the report 121 calculated by the language feature calculation unit 102 with the language features of the report 121 registered in the language feature storage unit 112. For example, the language feature storage unit 112 stores the language features of the report 121 up to the previous month and compares them with the language features of the report 121 for the current month calculated by the language feature calculation unit 102.

[0050] The integration unit 106 integrates the image feature similarity score, which is the result of the image similarity calculation unit 104, and the language feature similarity score, which is the result of the language similarity calculation unit 105. Here, integrating the image feature similarity score and the language feature similarity score may, for example, be done by averaging the respective scores, or by weighting one score and adding it to the other score.

[0051] The identification unit 107 identifies the type of form 121 based on the similarity score integrated by the integration unit 106.

[0052] For example, if the sender's address registered as a language feature in the language feature storage unit 112 does not exist in the language features of the document 121 calculated by the language feature calculation unit 102, the identification unit 107 will prioritize identification based on image features. By adding these controls, it becomes possible to identify documents even when the document 121 does not contain an address.

[0053] Furthermore, the identification unit 107 performs control such as not identifying a document as the same document if, for example, the similarity score of the image features is extremely low, even if the similarity score of the language features of the document 121 calculated by the language feature calculation unit 102 is high. By adding these controls, it is possible to identify documents 121 that have similar sender company and address information as different documents 121.

[0054] Furthermore, the identification unit 107 first sorts the documents in ranking order based on the similarity score of the image features calculated from the image features, extracts the top X documents, and then re-ranks them based on the similarity score of the language features calculated from the language features of the documents 121. By adding these controls, even if the documents 121 have a common layout, the subsequent ranking based on language features makes it possible to accurately distinguish between the company's documents 121 and other documents 121.

[0055] Furthermore, when setting a threshold, the identification unit 107 may display on the display 208 a single document with the highest integrated similarity value that is above the threshold, or multiple documents in the order of their integrated similarity rankings, allowing the user to determine which document a particular document belongs to.

[0056] Figure 7 shows an example of how the identification results are displayed. In the example shown in Figure 7, the document identification system 1 searches for documents similar to document 800, which is the document 121 to be identified, in response to the operation of the search button 821.

[0057] In the example shown in Figure 7, the search results display Form 810 as a form similar to Form 800, which is the form being identified. The sender information of Form 800 (company name and department 802, address 803, telephone number 804) and the sender information of Form 810 (company name and department 812, address 813, telephone number 814) are identical. Also, as shown in Figure 7, the layout of Table 805, which lists the claim items of Form 800, and Table 815, which lists the claim items of Form 810, are different.

[0058] The score 822 displays the similarity score of the image features before integration (image similarity), the similarity score of the language features (language similarity), and the integrated similarity score after integration (integrated similarity).

[0059] Furthermore, in the display example shown in Figure 7, a threshold is explicitly set as shown in threshold setting 823. When a threshold is set, the identification unit 107 displays the document with the highest integrated similarity value that is equal to or greater than the threshold on the display 208. If there are no documents that exceed the threshold, the identification unit 107 does not display any documents on the display 208. Also, when no threshold is set, the identification unit 107 selects the document with the highest integrated similarity to be displayed as a similar document and displays it on the display 208.

[0060] The identification unit 107 identifies a document 810 that resembles document 800, which is the document 121 to be identified, as a type of document 121, based on the operation of the document selection button 820.

[0061] Figure 8 shows another example of how the identification results are displayed. In the example shown in Figure 8, the document identification system 1 searches for documents similar to document 800, which is the document 121 to be identified, in response to the operation of the search button 821.

[0062] The difference between the display example shown in Figure 8 and the display example shown in Figure 7 lies in whether multiple reports are displayed or a single report is displayed. When multiple reports 902, 903, 904, and 905 are displayed as in the display example shown in Figure 8, the integrated similarity is sorted in ranking order, which has the effect of allowing the user to see the desired report even if it is not ranked first. In the display example shown in Figure 8, the search results show that the integrated similarity was highest for reports 902, followed by reports 903, 904, and then reports 905.

[0063] The identification unit 107 identifies one of the documents 902, 903, 904, or 905, which are similar to document 800 (the document 121 to be identified) selected in response to the operation of the document selection button 820, as the type of document 121. Figure 8 shows an example where document 903 is selected.

[0064] Furthermore, the entirety or part of the arithmetic unit 100 may be implemented by hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0065] Furthermore, the functional units of the document identification device 10 shown in Figure 5 are conceptual representations of their functions and are not limited to this configuration. For example, the multiple functional units shown as independent functional units in the document identification device 10 in Figure 5 may be configured as a single functional unit. Alternatively, the functions of one functional unit of the document identification device 10 shown in Figure 5 may be divided into multiple functions and configured as multiple functional units.

[0066] Furthermore, as mentioned above, the document identification device 10 is not limited to being composed of a single information processing device. For example, the OCR processing unit 103 may be implemented by an information processing device separate from the document identification device 10.

[0067] Next, we will explain the flow of the document identification process of the document identification device 10.

[0068] Figure 9 is a flowchart illustrating an example of the document identification process flow of the document identification device 10. First, the document identification device 10 receives input of a document image of the document to be identified 121 from the multifunction printer 20 via the network I / F 209 (step S1).

[0069] Next, the OCR processing unit 103 performs OCR processing on the input form image 121 to recognize the characters (step S2).

[0070] Next, the language feature calculation unit 102 calculates language features using the text information (string) recognized by the OCR processing unit 103 and the LLM, and identifies information such as the name and address of the billing company (step S3).

[0071] Next, the language similarity calculation unit 105 compares the language features calculated by the language feature calculation unit 102 with the language features of the forms registered in the language feature storage unit 112 to calculate the similarity of the language features (step S4).

[0072] Next, the image feature calculation unit 101 extracts image features from the document image of the input document 121 (step S5).

[0073] Furthermore, the image feature calculation unit 101 may perform a preprocessing step in which it fills in and removes the positions of the recognized characters obtained by the OCR processing unit 103 with white in the document image of the document 121 to be identified.

[0074] When identifying form 121, pre-filled characters such as claim items can be variable. However, by removing the recognized characters obtained through OCR processing, it is possible to calculate features specific to the image layout information (ruled line information) of form 121 and calculate the similarity, thereby improving performance.

[0075] Furthermore, in addition to filling the positions of recognized characters obtained by the OCR processing unit 103 with white in the document image of the document 121 to be identified, the positions of the recognized characters may also be filled with a different color or with a color surrounding the position of the recognized characters. By adding these controls, it is possible to delete characters in a more natural way, thus improving performance.

[0076] In this embodiment, the positions of the recognized characters obtained by the OCR processing unit 103 are filled with white, but this is not the only option. Instead of recognizing characters, the positions of the strings extracted by the string extraction process may be filled with white.

[0077] Next, the image similarity calculation unit 104 compares the image features calculated by the image feature calculation unit 101 with the image features of the form 121 registered in the image feature storage unit 111 to calculate the similarity of the image features (step S6).

[0078] Next, the integration unit 106 integrates the image feature similarity score, which is the result of the image similarity calculation unit 104, with the language feature similarity score, which is the result of the language similarity calculation unit 105 (step S7).

[0079] Next, the identification unit 107 makes a final determination by identifying the form 121 based on the similarity scores integrated by the integration unit 106 (step S8).

[0080] Next, we will explain the dictionary registration process for the document identification device 10.

[0081] Figure 10 is a flowchart showing an example of the dictionary registration process of the document identification device 10. First, the image feature calculation unit 101 extracts the image features of the document image of the document 121 to be registered in the dictionary (step S11).

[0082] Next, the image feature storage unit 111 registers the image features calculated by the image feature calculation unit 101 as dictionary information (step S12).

[0083] Next, the OCR processing unit 103 performs OCR processing on the document image of the document 121 to be registered in the dictionary and recognizes the characters (step S13).

[0084] Next, the language feature calculation unit 102 calculates language features using the text information (string) recognized by the OCR processing unit 103 and the LLM, and identifies information such as the name and address of the billing company (step S14).

[0085] Next, the language feature memory unit 112 registers the language features calculated by the language feature calculation unit 102 as dictionary information (step S15).

[0086] [Example 1] Here, Figure 11 shows an example of document identification in the document identification device 10. In the example shown in Figure 11, documents 500 and 510 are shown as examples of document 121.

[0087] As shown in Figure 11, the sender information for document 500 (company name and department 502, address 503, telephone number 504) and the sender information for document 510 (company name and department 512, address 513, telephone number 514) are different. Also, as shown in Figure 11, the layout of Table 505, which lists the claim items for document 500, and the layout of Table 515, which lists the claim items for document 510, are the same.

[0088] As mentioned above, in the example shown in Figure 11, since the sender information differs between document 500 and document 510, it is necessary to identify them as invoices from different companies. However, because the layout of the table listing the billing items is the same in both document 500 and document 510, document identification using only image features would result in them being identified as invoices from the same company.

[0089] Therefore, the document identification device 10 of this embodiment performs document identification not only using image features calculated by the image feature calculation unit 101, but also using language features calculated by the language feature calculation unit 102. The language feature calculation unit 102, in calculating language features, does not generate rotated images at multiple angles or store positional information of keyword strings, but instead utilizes document-specific expressions within the document. Document-specific expressions include sender address, sender company name, sender telephone number, payment bank account number, product delivery address, business location, etc. However, document-specific expressions are not limited to these.

[0090] In other words, the document identification device 10 of this embodiment uses a large-scale language model (LLM) in addition to the layout information of the table in which the claim items are described to extract a target string (for example, sender information (company name and department, address, telephone number, etc.)) from a single image and add it to the comparison target, thereby making the calculation of language features easier and less resource-intensive when identifying documents.

[0091] [Example 2] Here, Figure 12 shows an example of document identification in the document identification device 10, specifically example 2. In the example shown in Figure 12, documents 600 and 610 are shown as examples of document 121.

[0092] As shown in Figure 12, the sender information for form 600 (company name and department 602, address 603, telephone number 604) and the sender information for form 610 (company name and department 612, address 613, telephone number 614) are identical. Also, as shown in Figure 12, the layout of Table 605, which lists the claim items for form 600, and Table 615, which lists the claim items for form 610, are different.

[0093] As described above, in the example shown in Figure 12, since the sender information is the same in both document 600 and document 610, it is required that they be identified as invoices from the same company. However, because the layout of the table listing the billing items is different in document 600 and document 610, document identification using only image features would result in them being identified as invoices from different companies.

[0094] Therefore, the document identification device 10 of this embodiment performs document identification not only using image features calculated by the image feature calculation unit 101, but also using language features calculated by the language feature calculation unit 102. The image feature calculation unit 101, in calculating language features, does not generate rotated images at multiple angles or store positional information of keyword strings, but instead utilizes document-specific expressions within that document. Document-specific expressions include sender address, sender company name, sender telephone number, payment bank account number, product delivery address, business location, etc. However, document-specific expressions are not limited to these.

[0095] In other words, the document identification device 10 of this embodiment uses a large-scale language model (LLM) in addition to the layout information of the table in which the claim items are described to extract a target string (for example, sender information (company name and department, address, telephone number, etc.)) from a single image and add it to the comparison target, thereby making the calculation of language features easier and less resource-intensive when identifying documents.

[0096] [Example 3] Here, Figure 13 shows an example of document identification in the document identification device 10, specifically example 3. In the example shown in Figure 13, documents 700 and 710 are shown as examples of document 121.

[0097] As shown in Figure 13, the sender information for form 700 (company name and department 702, address 703, telephone number 704) and the sender information for form 710 (company name and department 712, address 713, telephone number 714) are different. Also, as shown in Figure 13, the layout of Table 705, which lists the claim items for form 700, and Table 715, which lists the claim items for form 710, are different.

[0098] As mentioned above, in the example shown in Figure 13, since the sender information differs between document 700 and document 710, it is necessary to identify them as invoices from different companies. However, document identification using only linguistic features would identify them as invoices from the same company because the sender information (company name, department, address, telephone number, etc.) is very similar.

[0099] Therefore, the document identification device 10 of this embodiment performs document identification using linguistic features calculated by the linguistic feature calculation unit 102 in addition to image features calculated by the image feature calculation unit 101. The image feature calculation unit 101, in calculating linguistic features, does not generate rotated images at multiple angles or store positional information of keyword strings, but instead utilizes document-specific expressions within that document. Document-specific expressions include sender address, sender company name, sender telephone number, payment bank account number, product delivery address, business location, etc. However, document-specific expressions are not limited to these.

[0100] Thus, according to this embodiment, in a document recognition technology that combines image features and language features, the calculation of language features can be performed more easily and with fewer resources, without generating rotated images at multiple angles or storing positional information of keyword strings.

[0101] In this embodiment, the language feature calculation unit 102 calculates language features using LLM, but this is not the only option. For example, when calculating language features, the language feature calculation unit 102 may extract sender information (company name and department, address, telephone number, etc.) using methods such as regular expressions or word matching, without using LLM, and then extract the unique expression for that form by excluding pre-registered recipient information (company name and address). This makes it possible to calculate language features more easily and with fewer resources without generating rotated images at multiple angles or storing positional information of keyword strings, and has the effect of requiring fewer resources and a simpler configuration compared to the method of calculating language features using LLM.

[0102] (Second Embodiment) Next, a second embodiment will be described.

[0103] The second embodiment differs from the first embodiment, which uses OCR processing for calculating language features, in that it does not use OCR processing for calculating language features. In the following description of the second embodiment, the parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0104] Figure 14 shows an example of the configuration of a functional block of a document identification system 1 including a document identification device according to the second embodiment. Figure 15 shows an example of the operation of the functional block of the document identification unit of the document identification device.

[0105] As shown in Figures 14 and 15, the calculation unit 100 of the document identification device 10 according to this embodiment differs from the first embodiment in that it does not include an OCR processing unit 103.

[0106] Furthermore, as shown in Figures 14 and 15, the language feature calculation unit 102 of the calculation unit 100 of the document identification device 10 according to this embodiment differs from the first embodiment in that it calculates language features without using OCR processing.

[0107] More specifically, the language feature calculation unit 102 uses a technology such as VLM (Vision and Language Model) to input an image and a language instruction (text information) related to that image, and extracts the unique expressions of the document as language features.

[0108] Thus, according to this embodiment, in a document recognition technology that combines image features and language features, the calculation of language features can be performed more easily and with fewer resources, without generating rotated images at multiple angles or storing positional information of keyword strings.

[0109] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.

[0110] Furthermore, the information processing device 10 is not limited to an image forming apparatus, as long as it is a device equipped with communication functions. The information processing device 10 may be, for example, an output device such as a PJ (Projector), IWB (Interactive White Board: an electronic whiteboard with the ability to communicate with each other), or digital signage, a HUD (Head Up Display) device, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, automobiles (Connected Car), notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, or desktop PCs.

[0111] The image features and linguistic features of the pre-extracted forms 121 in the specification may be generated by the learning effect of machine learning. Furthermore, by classifying the image features and linguistic features of the pre-extracted forms 121 using machine learning, it is not necessary to use a correspondence table. Here, machine learning is a technique for enabling computers to acquire human-like learning abilities, in which a computer autonomously generates algorithms necessary for judgments such as data identification from pre-incorporated training data, and applies these to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods may be used; the learning method for machine learning is not restricted.

[0112] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the document identification device 10 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0113] Examples of the present invention are as follows: <1> An image feature calculation unit calculates the image features of the document image of the target document, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, Equipped with, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A document identification device characterized by the following features. <2> The system further includes a character recognition unit that recognizes characters written in the aforementioned document image and extracts text information. The language feature calculation unit calculates the language features of the report image from the text information. Characterized by <1> The document identification device described above. <3> The language feature calculation unit uses a large-scale model for calculating the language features. Characterized by <1> or <2> The document identification device described above. <4> The language feature calculation unit extracts company names and addresses using methods such as regular expressions and word matching, and then extracts expressions unique to that form by excluding the company names and addresses of the company and department that have been registered in advance. Characterized by <1> or <2> The document identification device described above. <5> The aforementioned language feature calculation unit uses technologies such as large-scale visual language models to input images and related language instructions, and then selectively extracts the unique representations of the document to be used as language features. Characterized by <1> The document identification device described above. <6> The image feature calculation unit performs a preprocessing step, which involves removing the positions of the recognized characters obtained by the character recognition unit from the document image. Characterized by <2> The document identification device described above. <7> The image feature calculation unit performs a preprocessing step, which involves removing characters from the document image. Characterized by <1> The document identification device described above. <8> The image feature calculation unit performs a process to remove characters by filling the positions of the characters in the document image with white. Characterized by <7> The document identification device described above. <9> The image feature calculation unit performs a process to remove characters by filling the positions of characters in the document image with a color other than white. Characterized by <7> The document identification device described above. <10> The image feature calculation unit performs a process to remove characters in the document image by filling the character positions with the surrounding color of the character positions. Characterized by <7> The document identification device described above. <11> The identification unit performs control to prioritize identification from the image features if the description of the language features of the form that has been registered in advance is not present in the language features calculated by the language feature calculation unit. Characterized by <1> or <10> A document identification device as described in any one of the following. <12> The identification unit, if the similarity score of the image features calculated by the image feature calculation unit is significantly low, performs control to not identify the documents as the same, even if the similarity score of the language features calculated by the language feature calculation unit is high. Characterized by <1> or <10> A document identification device as described in any one of the following. <13> The identification unit sorts the image features calculated by the image feature calculation unit in ranking order and extracts the top X items, and then re-ranks them based on the similarity scores of the language features calculated by the language feature calculation unit. Characterized by <1> or <10> A document identification device as described in any one of the following. <14> When a threshold is set, the identification unit displays a single document on the display unit that is equal to or greater than the threshold and has the highest integrated similarity score, allowing the user to determine the type of document. Characterized by <1> or <10> A document identification device as described in any one of the following. <15> The identification unit displays multiple forms on the display unit in the order of the integrated similarity ranking, allowing the user to determine the type of form. Characterized by <1> or <10> A document identification device as described in any one of the following. <16> In a document identification system comprising an information processing device and a document identification device, The aforementioned information processing device is It includes an image generation unit that generates a document image of the target document, The aforementioned document identification device is An image feature calculation unit that calculates the image features of the aforementioned report image, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, Equipped with, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A document identification system characterized by the following features. <17> Computers, An image feature calculation unit calculates the image features of the document image of the target document, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, To make it function as, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A program characterized by the following features. <18> The process involves calculating the image features of the document images of the target document, and A language feature calculation step for calculating the language features of the aforementioned report image, An image similarity calculation step calculates a score of similarity to the image features calculated in the image feature calculation step by comparing them with image features of pre-registered forms, A language similarity calculation step calculates a score of similarity to the language features calculated in the language feature calculation step by comparing them with the language features of pre-registered forms, An integration step of combining the similarity scores of the image features and the similarity scores of the language features, An identification step of identifying the type of document based on the integrated similarity, Includes, The language feature calculation process utilizes expressions specific to the form in relation to the calculation of the language features. A method for identifying documents characterized by the following: [Explanation of symbols]

[0114] 1. Document Identification System 10. Document identification device 20 Information Processing Devices 21 Image generation unit 101 Image Feature Calculation Unit 102 Language Feature Calculation Unit 103 Character recognition section 104 Image Similarity Calculation Unit 105 Linguistic similarity calculation unit 106 Integration Department 107 Identification Unit [Prior art documents] [Patent Documents]

[0115] [Patent Document 1] Japanese Patent Publication No. 2023-46514

Claims

1. An image feature calculation unit calculates the image features of the document image of the target document, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, Equipped with, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A document identification device characterized by the following features.

2. The system further includes a character recognition unit that recognizes characters written in the aforementioned document image and extracts text information. The language feature calculation unit calculates the language features of the report image from the text information. The document identification device according to feature 1.

3. The language feature calculation unit uses a large-scale model for calculating the language features. The document identification device according to claim 1 or 2.

4. The language feature calculation unit extracts company names and addresses using methods such as regular expressions and word matching, and then extracts expressions unique to that form by excluding the company names and addresses of the company and department that have been registered in advance. The document identification device according to claim 1 or 2.

5. The aforementioned language feature calculation unit uses technologies such as large-scale visual language models to input images and related language instructions, and then selectively extracts the unique representations of the document to be used as language features. The document identification device according to feature 1.

6. The image feature calculation unit performs a preprocessing step, which involves removing the positions of the recognized characters obtained by the character recognition unit from the document image. The document identification device according to feature 2.

7. The image feature calculation unit performs a preprocessing step, which involves removing characters from the document image. The document identification device according to feature 1.

8. The image feature calculation unit performs a process to remove characters by filling the positions of the characters in the document image with white. The document identification device according to feature 7.

9. The image feature calculation unit performs a process to remove characters by filling the positions of characters in the document image with a color other than white. The document identification device according to feature 7.

10. The image feature calculation unit performs a process to remove characters in the document image by filling the character positions with the surrounding color of the character positions. The document identification device according to feature 7.

11. The identification unit performs control to prioritize identification from the image features if the description of the language features of the form that has been registered in advance is not present in the language features calculated by the language feature calculation unit. The document identification device according to feature 1.

12. The identification unit, if the similarity score of the image features calculated by the image feature calculation unit is significantly low, performs control to not identify the documents as the same, even if the similarity score of the language features calculated by the language feature calculation unit is high. The document identification device according to feature 1.

13. The identification unit sorts the image features calculated by the image feature calculation unit in ranking order and extracts the top X items, and then re-ranks them based on the similarity scores of the language features calculated by the language feature calculation unit. The document identification device according to feature 1.

14. When a threshold is set, the identification unit displays a single document on the display unit that is equal to or greater than the threshold and has the highest integrated similarity score, allowing the user to determine the type of document. The document identification device according to feature 1.

15. The identification unit displays multiple forms on the display unit in the order of the integrated similarity ranking, allowing the user to determine the type of form. The document identification device according to feature 1.

16. In a document identification system comprising an information processing device and a document identification device, The aforementioned information processing device is It includes an image generation unit that generates a document image of the target document, The aforementioned document identification device is An image feature calculation unit that calculates the image features of the aforementioned report image, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, Equipped with, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A document identification system characterized by the following features.

17. Computers, An image feature calculation unit calculates the image features of the document image of the target document, A language feature calculation unit that calculates the language features of the aforementioned report image, An image similarity calculation unit calculates a similarity score of the image features calculated by the image feature calculation unit by comparing them with the image features of a form that has been registered in advance, A language similarity calculation unit calculates a score of similarity to the language features calculated by the language feature calculation unit by comparing them with the language features of pre-registered forms, An integration unit that integrates the similarity scores of the image features and the similarity scores of the language features, An identification unit that identifies the type of the document based on the integrated similarity, To make it function as, The language feature calculation unit utilizes expressions specific to the form in relation to the calculation of the language features. A program characterized by the following features.

18. The process involves calculating the image features of the document image of the target document, and A language feature calculation step for calculating the language features of the aforementioned report image, An image similarity calculation step calculates a score of similarity to the image features calculated in the image feature calculation step by comparing them with image features of pre-registered forms, A language similarity calculation step calculates a score of similarity to the language features calculated in the language feature calculation step by comparing them with the language features of pre-registered forms, An integration step of integrating the similarity scores of the image features and the similarity scores of the language features, An identification step of identifying the type of document based on the integrated similarity, Includes, The language feature calculation process utilizes expressions specific to the form in relation to the calculation of the language features. A method for identifying documents characterized by the following:

Images