Image processing apparatus, image processing method, and computer-readable storage medium

Abstract

The present application relates to an image processing apparatus, an image processing method, and a computer-readable storage medium. The present application obtains a character recognition result by performing character recognition processing on a document image, and recognizes a classification type of the document image based on a predefined condition and a character string included in the character recognition result. A condition for recognizing a classification type of a prompt as a charge item is defined in advance.

Description

Technical Field

[0001] This invention relates to an image processing apparatus for identifying the classification type of document images, and to an image processing method and a non-transitory computer-readable storage medium. Background Technology

[0002] In recent years, it has become common to scan documents and digitize them using image scanners included in multifunction peripherals (MFPs) (multifunction devices with printing, copying, faxing, etc.). Furthermore, it is also common to digitize documents by capturing images of them using the camera functions of mobile devices such as digital cameras and smartphones. Therefore, obtaining document images (scanned document images) by optically scanning documents containing handwritten and printed characters or by capturing images of documents has become easier. Moreover, by performing optical character recognition (OCR) processing on document images, character images in document images can be converted into computer-usable character codes. Due to the use of character recognition processing, automated tasks such as converting paper-based business forms (receipts, invoices, etc.) into digital data and inputting that digital data into systems (e.g., expense reimbursement) have become widespread. Therefore, it is expected to improve the productivity of data input tasks.

[0003] In Japanese Patent Application Publication No. 2018-097813, a journaling AI was created by extracting journaling elements from images used for learning and performing machine learning on the accounts of the journaling elements. The journaling elements include at least the date, seller, amount, remarks, and appearance such as size and color. Then, to process voucher images, journaling elements are extracted from the voucher images, and the journaling AI is used to select the accounts.

[0004] The method described in Japanese Patent Application Publication No. 2018-097813 requires prior learning via machine learning and necessitates numerous image data items for learning. Furthermore, when the expense item journal rules differ for each user, it is necessary to prepare learning data for each user. Summary of the Invention

[0005] The image processing apparatus disclosed herein includes: a character recognition unit that obtains character recognition results by performing character recognition processing on a document image; and a first recognition unit that identifies a classification type based on predefined conditions and strings included in the character recognition results.

[0006] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0007] Figure 1A , Figure 1Band Figure 1C It is a diagram showing an overview of the bookkeeping and / or accounting process, examples of document images, and examples of input content for the process.

[0008] Figure 2 This is a diagram showing the system architecture of an image processing system.

[0009] Figure 3 This is a diagram showing the hardware structure of an image forming apparatus.

[0010] Figure 4A and Figure 4B These are diagrams illustrating the hardware structure of the image processing server and the hardware structure of the user terminal, respectively.

[0011] Figure 5A and Figure 5B This is a diagram illustrating examples of receipts according to one or more aspects of this disclosure.

[0012] Figure 6 It is a diagram illustrating the overall processing according to one or more aspects of this disclosure.

[0013] Figure 7A , Figure 7B and Figure 7C This is a diagram illustrating an example of processing rules according to one or more aspects of this disclosure.

[0014] Figure 8 This is a diagram illustrating the output processing of item values ​​according to one or more aspects of this disclosure.

[0015] Figure 9 This is a diagram illustrating examples of receipts according to one or more aspects of this disclosure.

[0016] Figure 10A , Figure 10B and Figure 10C This is a diagram illustrating examples of input information according to one or more aspects of this disclosure.

[0017] Figure 11 This is a diagram illustrating the processing of item value outputs according to one or more aspects of this disclosure.

[0018] Figure 12 This is a diagram illustrating examples of receipts according to one or more aspects of this disclosure.

[0019] Figure 13A , Figure 13B and Figure 13C This is a diagram illustrating examples of rules according to one or more aspects of this disclosure. Detailed Implementation

[0020] Figure 1AIt illustrates the information flow between the accounting system and various systems that coordinate with it.

[0021] Accounting system 101 supports accounting and / or bookkeeping tasks. Personnel responsible for accounting and / or bookkeeping record transactions involving the company's cash, deposits, assets, products, etc., in accounting system 101 while classifying transactions for administrative or tax purposes. Various forms, typically used as business forms, are computerized in accounting system 101. The recorded content is stored as accounting processing results 102 and output as various business forms 107 as needed. Accounting system 101 needs to classify the expense items of transactions according to their content and purpose, and uses expense item codes to identify each expense item.

[0022] In addition, the aforementioned systems include a deposit / withdrawal management system 103 for managing cash, deposits, etc.; a budget management system 104 for managing departmental budgets; an inventory management system 105 for managing product inventory; and an asset management system 106 for managing various assets. When transactions involving cash, deposits, assets, products, etc. occur, the increase or decrease of cash, etc., involved in the transaction is recorded in each management system (103, 105, and 106).

[0023] Typically, the person responsible for accounting and / or bookkeeping refers to the information of each transaction recorded in the various management systems (103, 105, and 106) and records the information in the accounting system 101. In addition, the budget management system 104 uses budget codes to identify each budget in order to manage the budget.

[0024] Figure 1B A business form document 110 is shown as an example of a receipt (business form document) issued when purchasing office supplies. This receipt serves as proof (written evidence) of the purchase of office supplies. Various information items are written in the sections included in business form document 110. For example, the information indicating that this business form document is a "receipt" is written in the document title 111, which is the title section of business form document 110. The issuance date 112 indicates the date the business form document was issued. When the receipt of the business form document is stored as a voucher in accounting and / or bookkeeping work, the information "November 12, 2020" as the issuance date 112 can be used to identify the business form document.

[0025] The company name, address, and telephone number are written on the issuer's page (Issuer 113) as information of the company that created and issued the business form document. When a business form document is received and processed for accounting and / or bookkeeping, it is necessary to clarify the purchased items and their purpose in order to classify expense items, and this can be done using the company name "AAA Business Machine Corporation" of issuer 113. Furthermore, when a receipt for a business form document is stored as a voucher, the company name information can be used to identify the business form document.

[0026] The name of the company that purchased and paid for office supplies is written on Name 114. The total amount of the purchase and payment is written on Total Amount 115. This amount (and, if necessary, the purchase tax amount) is used as the transaction amount when processing business form documents for accounting and / or bookkeeping. Detailed billing information is written on Detail 116. For each product name, the unit price, quantity, price, etc., of each product are written. In addition, Total 117 indicates that price subtotals and tax information, etc., are summed to form the total amount written on Total Amount 115.

[0027] Figure 1C An example of business form information typically recorded in accounting system 101 by those responsible for accounting and / or bookkeeping is shown. Transaction record 120 on the business form indicates transactions used for accounting and / or bookkeeping in the business form and is presented in tabular form. The table indicates a transaction in one row, with the transaction date written in the initial position and the various information items written in the table.

[0028] The following describes an example of recording transaction information for a department (section). For example, in the budget management system 104, the purchase of office supplies for work is recorded, and the amount to be paid for the office supplies is recorded in advance as the department budget. Suppose that, according to the purchase content, the department purchased office supplies from a company named "AAA Commercial Machines Company" on "November 12, 2020" and paid cash for the office supplies from the deposit / withdrawal management system 108.

[0029] In this case, it is necessary to record information indicating the withdrawal of the purchase amount. Furthermore, in this case, assuming that to prove the purchase and payment have been completed, a company named "AAA Commercial Machines Co., Ltd." issues Business Form Document 110 as proof, and BBB Company receives Business Form Document 110. In this case, in the transaction record 120 on the business form, the following is written in the three columns of "Credit" and subsequent items (columns six to eight from the left). That is, the following is written: "Cash" and the code "100" indicating the cash type included in the withdrawal record of the deposit / withdrawal management system 103; the department name "Kamata Branch" and the budget code "221" as budget information in the budget management system 104; and the amount paid, "7700" yen. Furthermore, in the transaction record 120 on the business form, the following is written in the three columns of "Debit" and subsequent items (columns two to four from the left). That is, information instructing "Kamata Branch" to use "7700" yen as an expense for office supplies is written. Therefore, the person responsible for accounting and / or bookkeeping writes the expense item "Office Supplies Expenses" and expense item code "300" into the "Debit" side, the department name "Kamata Branch" and budget code "221" into the "Department" side, and "7700" yen into the "Amount" side. Furthermore, as a basis for expense item classification, the person in charge writes "AAA Commercial Machines Co., Ltd." into the "Remarks" side for easy association with business form document 110. By writing such information as described above, facts such as cash withdrawals from assets and the purchase of office supplies based on the departmental budget can be recorded in a related manner as transactions used for office supplies expenses. The person responsible for accounting and / or bookkeeping confirms the various facts, including vouchers, to record information, classifies the information into expense items, and records the information in accounting system 101. The records stored in accounting system 101 constitute accounting processing results 102.

[0030] When a function is provided to automatically transmit the content written on business form document 110 (date, amount, company name, etc.) to accounting system 101 for this purpose, the workload of those responsible for accounting and / or bookkeeping can be reduced. Therefore, in recent years, the use of image scanners to read documents such as vouchers as electronic images and perform character recognition processing to extract and transmit the written information of the documents is being considered.

[0031] However, company names may be written in logos or special fonts, or characters stamped on documents such as receipts may be unclear, which may lead to recognition errors in character recognition processing.

[0032] First Embodiment

[0033] The following embodiments describe a data input support device that extracts item names and item values ​​from a document image and displays the extracted item names and extracted item values.

[0034] Figure 2 This is a diagram illustrating an example construction of an image processing system 200 according to one or more aspects of this disclosure. The image processing system 200 includes an image forming apparatus 201, an image processing server 202, and a user terminal 203, which are communicatively interconnected via a network 204.

[0035] Image forming apparatus 201 can receive print requests (print data) for printing image data from user terminal 203 and can print the received image data. Image forming apparatus 201 can read image data using a scanner included in image forming apparatus 2011 and print the image data read by the scanner. Furthermore, image forming apparatus 201 can store the print data received from user terminal 203 and send the image data read by the scanner of image forming apparatus 201 to user terminal 203 and image processing server 202. Image forming apparatus 201 can implement the functions of known image forming apparatuses such as multifunction peripheral devices (MFPs). User terminal 203 can use an application with a user interface to display the image processing results received from image processing server 202 and interactively process the image processing results according to instructions from the user. Image processing server 202 can be deployed in the cloud, i.e., on the Internet.

[0036] In this embodiment, it is assumed that the user terminal 203 is a general-purpose personal computer (PC) with a display, keyboard and mouse, but the user terminal 203 may be a mobile terminal, for example, with a touch panel.

[0037] In a series of data input support processes, image forming apparatus 201 scans documents such as receipts, image processing server 202 extracts information from the images of the documents, and user uses user terminal 203 to confirm and modify the results of the extracted information. The following describes a series of data input support processes.

[0038] Figure 3 This is a diagram illustrating an example of the construction of an image forming apparatus 201. The image forming apparatus 201 includes a controller 301, a printer 302, a scanner 303, and an operation unit 304. The controller 301 includes a CPU 311, RAM 312, HDD 313, a network interface (I / F) 314, a printer I / F 315, a scanner I / F 316, an operation unit I / F 317, and an expansion I / F 318.

[0039] The CPU 311 controls the overall operation of the image forming apparatus 201. The CPU 311 can control the sending and receiving of data to and from the RAM 312, HDD 313, network I / F 314, printer I / F 315, scanner I / F 316, operation unit I / F 317, and expansion I / F 318. Furthermore, the CPU 311 loads the control program (commands) read from the HDD 313 into the RAM 312 and executes the commands loaded into the RAM 312.

[0040] HDD 313 stores control programs executable by CPU 311, setting values ​​to be used by image forming apparatus 201, data related to user-requested processing, etc. RAM 312 has an area for temporarily storing commands read by CPU 311 from HDD 313. RAM 312 can store various data required to execute the command. For example, in image processing, CPU 311 can load input data into RAM 312 and process it.

[0041] Network I / F 314 is an interface for communicating with the devices included in the image processing system 200. Network I / F 316 can send information to the CPU 311 indicating that data has been received, and can send data from RAM 312 to network 204.

[0042] Printer I / F 315 can send print data sent from CPU 311 to printer 302, and send the status of printer 302 received from printer 302 to CPU 311. Scanner I / F 316 can send image read instructions sent from CPU 312 to scanner 303, send image data received from scanner 303 to CPU 331, and then send the status of scanner 303 received from scanner 303 to CPU 311.

[0043] The operation unit I / F 317 can send instructions input by the user from the operation unit 304 to the CPU 311, and send screen information to be operated by the user to the operation unit 304. The expansion I / F 318 is an interface that enables the image forming apparatus 201 to connect to external devices. For example, the expansion I / F 318 has a Universal Serial Bus (USB) interface. When an external storage device, such as a USB memory, is connected to the expansion I / F 318, the image forming apparatus 201 can read data stored in the external storage device and write data to the external storage device.

[0044] Printer 302 can print image data received from printer I / F 315 onto paper and send the status of printer 302 to printer I / F 315. Scanner 303 can read and computerize the information indicated on the paper placed on scanner 303 according to the image reading instruction received from scanner I / F 316, and send the computerized information to scanner I / F 316. In addition, scanner 303 can send the status of scanner 303 to scanner I / F 316.

[0045] The operation unit 304 is an interface that allows the user to issue various commands to the image forming apparatus 201. For example, the operation unit 304 includes a liquid crystal screen with a touch panel, provides an operation screen to the user, and receives operations from the user.

[0046] Figure 4A This diagram illustrates an example of the construction of an image processing server 202. The image processing server 202 includes a CPU 401, RAM 402, HDD 403, and network I / F 404. The CPU 401 controls the entire image processing server 302. The CPU 401 can control the sending and receiving of data to and from the RAM 402, HDD 402, and network I / F 404. Furthermore, the CPU 401 loads programs (commands) read from the HDD 403 into the RAM 402 and executes the commands loaded in the RAM 402, thus serving as a processing unit for executing the processes described in the flowchart below.

[0047] Figure 4BThis diagram illustrates an example of the construction of a user terminal 203. The user terminal 203 includes a CPU 411, RAM 412, HDD 413, network I / F 414, and input / output I / F 415. The CPU 411 controls the entire user terminal 203. The CPU 411 can control the sending and receiving of data to and from the RAM 412, HDD 413, network I / F 414, and input / output I / F 415. The display 420 is composed of a liquid crystal display device or the like, and displays the display information received from the input / output I / F 415. The input device 430 is composed of a keyboard and a pointing device such as a mouse or touch panel. The input device 430 receives operations from the user and sends operation information to the input / output I / F 415. The HDD 413 can store image processing results received from the image processing server 202 via the network I / F 414. In this embodiment, the CPU 411 loads the application program read from the HDD 413 into the RAM 412 and executes the application program loaded in the RAM 412 so that the input / output I / F 415 displays display information and receives user operations.

[0048] Figure 5A and Figure 5B An example of a document image 500 generated by scanning a document using an image forming apparatus is shown. Figure 5A The example of document image 500 shown is an image obtained by reading a taxi receipt by the image forming apparatus 201.

[0049] This embodiment describes the process of extracting company name information from a document image. Figure 5A and Figure 5BThe company name information on the receipt shown is the issuer information used in accounting systems and expense reimbursement processing to estimate expense items such as transportation costs. Company name information typically indicates the company name specific to the company. However, in this embodiment, when a specific company name cannot be identified, information about the company classification type (company name type information), such as railway company, airline, taxi, or toll road, is output. When the classification type can be determined as a railway company receipt, airline receipt, taxi receipt, etc., the receipt can be estimated to indicate payment for transportation costs, and the operation of inputting the receipt information into the accounting system can be easily performed. That is, when a specific company name can be identified in the memo field, etc., of the accounting system, the company name information is displayed. Even if a specific company name cannot be identified, user input can be supported by displaying the company classification type information. Since information about a specific company name is initially more detailed, it is desirable to display that specific company name information. However, when it is difficult to extract a specific company name for various reasons, information other than the company name is used to identify and display the company classification type. As methods for extracting company names, there are, for example, methods that search for company names from document recognition results using a company name dictionary; and methods that perform pattern extraction based on string rules to extract phone numbers included in document recognition results and search for company names from dictionaries associated with company names and phone numbers. In both of these methods, a specific company name cannot be identified when characters at positions corresponding to the company name and phone number cannot be recognized or are misrecognized.

[0050] Figure 6 This is a flowchart illustrating the process of extracting company name types (process for determining company classification types) according to one or more aspects of this disclosure. For example, the input is described below. Figure 5A The document image 500 shown is processed to identify "taxi" as the company name type (company classification type).

[0051] In S601, the image processing server 202 acquires the document image 500 read by the scanner 303 from the image forming apparatus 201.

[0052] In S602, the CPU 401 of the image processing server 202 analyzes the document image 500, detects character regions from the document image 500, and performs character recognition processing on the character regions. As a result of the character recognition processing, the CPU 401 identifies the coordinates of the character regions, the coordinates of each character in the character regions, and the character codes included in the character recognition results. The array of character codes for each character region obtained in this case is called an OCR string.

[0053] In S603, the CPU 401 of the image processing server 202 loads the information extraction rules stored in the HDD 403 into the RAM 402. The information extraction rules include a dictionary for extracting item values, pattern information, and conditions for outputting item values.

[0054] Figure 7A , Figure 7B and Figure 7C An example of information extraction rules is shown. Figure 7A This is a diagram of Table 700, which shows a list of dictionaries that indicate rules for information extraction. In Table 700, each row defines a dictionary. Table 700 includes a column for numbers, a column for dictionary names, and a column for a list of search strings indicating search strings. Dictionary 701 has the name "total key" and is associated with a list of search strings such as "total amount," "payment amount," and "received amount," which can be hints for total amount items. Dictionary 702 has the name "phone-key" and is associated with a list of search strings such as "TEL" and "telephone," which can be hints for telephone number items. Dictionary 703 has the name "taxi terminology A" and is associated with a list of search strings such as "taxi" and "limousine," which can be hints for taxi receipts. Dictionary 704 has the name "taxi terminology B" and is associated with a list of search strings such as "license plate number," "vehicle number," and "radio number," which can be hints for taxi receipts. Dictionary 705 has the name "Taxi Terminology C" and is associated with a list of search strings such as "fare" and "meter rate," which may be prompts for taxi receipts. Dictionaries 701 to 705 in this embodiment are examples for description purposes, and this embodiment is not limited thereto.

[0055] Figure 7BThis is a diagram illustrating Table 710, which is a list of patterns defined as rules for information extraction. In Table 710, each row defines a pattern. Table 710 includes a column for numbers, a column for pattern names, and a column for regular expressions indicating the search patterns. Pattern 711 has the name "Amount" and is used to extract string patterns that match the regular expression "\?[\d,]+JPY". The regular expression for pattern 711 is a string pattern where the symbol "\" is absent, or a symbol "\" is present at the beginning, one or more consecutive digits and one or more commas are present, and the character "JPY" is present at the end. For example, the string "\1000JPY" matches this pattern. The regular expression for amount is an example described. Other patterns can be used to extract strings indicating amount values, as well as patterns based on misidentification or variations in character recognition results. Multiple search patterns can be used. Furthermore, the method of representing search patterns is not limited to regular expressions. Pattern 712 has the name "Phone Number" and a string pattern that matches the regular expression "0\d{1,3}[-(]\d{2,4}[-)]\d{4}". The regular expression for pattern 712 is a string pattern where the first character is "0", followed by one to three consecutive characters, then a "-" or "(", followed by two to four digits, then another "-" or ")", and finally four consecutive digits. For example, a string indicating a phone number such as "03(1234)5678" matches this pattern. Similar to the pattern used to search for amounts, the regular expression for phone numbers is an example described, and this embodiment is not limited thereto.

[0056] In S604, the CPU 401 of the image processing server 202 performs a text search to search for a string that matches the conditions of the search string in the dictionary list 700 loaded in S603 from the OCR string of the character recognition results obtained in S602. Figure 5B The results of text searches performed from OCR strings are shown. Search result 511 is a search for "license plate number" from taxi terminology B in dictionary 704. Search result 512 is a search for "fare" from taxi terminology C in dictionary 705. Search result 513 is a search for "total" from the total keyword in dictionary 701. Search result 514 is a search for "TEL" from the telephone keyword in dictionary 7002.

[0057] In S605, the CPU 401 of the image processing server 202 performs a text search to search for a string that matches the search pattern of the pattern list 710 loaded in S603 from the OCR string of the character recognition results obtained in S602. Figure 5BSearch results 515 and 516 shown are results for amounts that match the search pattern 711. Furthermore, Figure 5B The search result 517 shown is the result of searching for phone numbers that match the search pattern 712.

[0058] Figure 7C Table 720 is shown as an example of output conditions for item values ​​of text search results based on S604 and S605. Table 720 includes a number column for output condition number, an item name column for the item name to be output, a type column, a condition column for output conditions, a column indicating the text search result to be determined, and an output value column. When the search result matches the output condition, the output value indicates what is output as the item value. The output value indicates "search result value determined as value (value of text search result determined as value)" as the output target, or indicates a string such as "taxi" as the item value as the output target. The "position of search result" written in the type column in Table 720 indicates the position of the text search result that will be used for condition determination. In addition, the "logical operation of search result" written in the type column indicates that calculations are performed using logical operations on the text search result.

[0059] When the type column indicates "Position of search results" in Table 720, the condition column indicates which condition must be met for the positional relationship of the text search results to output a value. For example, the expression "Value to the right of keyword" in the condition column indicates that the positions of the two text search results for the keyword and the value must meet the condition when the value is to the right of the keyword. Figure 7C Output condition 721, shown, has the item name "Total Amount" and is such that the item value is output when the text search result for "Total Keyword" is used as the keyword, the text search result for "Amount" is used as the value, and the value is to the right of the keyword. Output condition 721 is described as a rule and instructs that the value of the amount in the "Amount Pattern" 711 to the right of the strings "Total Amount" and "Payment Amount" written in the "Total Keyword" dictionary 701 may be the total amount on the receipt. Similarly, output condition 722, with the item name "Phone Number," is such that the value is output when the text search result for "Phone Number" is used as the keyword, the text search result for "Phone Number" is used as the value, and the value is to the right of the keyword.

[0060] Furthermore, when the Type column in Table 720 indicates "Logical operation of search results," the Condition column indicates which logical operation expression the search result satisfies to output a value. An expression indicating "Logical operation result is true" means that the value will be output when the logical operation of the specified logical expression is true. For example, Figure 7CThe output condition 723 shown is the output condition for outputting a taxi when the logical operation of the search result is used as the type and the logical expression "taxi term A | (taxi term B & taxi term C)" is true. Output condition 723 is described as a rule and indicates that the output of the item value of type "taxi" is set to taxi when "taxi term A" exists in dictionary 703, or when the string belonging to "taxi term B" in dictionary 704 and "taxi term C" in dictionary 705 exists. That is, "taxi term A" in dictionary 703 is a strong term that can be identified as taxi on its own. The individual terms in "taxi term B" in dictionary 704 and "taxi term C" in dictionary 705 are weak bases for individually indicating taxi. However, when both "taxi term B" and "taxi term C" exist simultaneously, "taxi term B" and "taxi term C" are likely to indicate taxi.

[0061] In S606, the CPU 401 of the image processing server 202 determines the item value to be output based on the output conditions of the rules loaded in S603, that is, the output value corresponding to the item name of the types total amount, telephone number, and company name, and outputs the determined value. (Reference) Figure 8 The flowchart describes in detail the item value output processing in S606.

[0062] Figure 8 This is a flowchart illustrating the details of the processing of item value outputs according to one or more aspects of this disclosure.

[0063] In S801, the CPU 401 of the image processing server 202 sequentially selects the output conditions of the rules loaded in S603 and processes them before entering S802. For example, suppose the first selection... Figure 7C The output condition 721 is shown in the figure.

[0064] In S802, the CPU 401 of the image processing server 202 obtains the text search results from the text search results in S604 and S605 as the targets of the output conditions. Since the text search results with "total keyword" as the keyword and the text search results with "amount" as the value are the targets of the output condition 721, the CPU 401 obtains the corresponding text search results 513 for "total keyword" and the corresponding text search results 615 and 516 for "amount".

[0065] In S803, the CPU 401 of the image processing server 202 determines the type of output condition. When the type is "position of search results", processing proceeds to S804. When the type is "logical operation of search results", processing proceeds to S809. When output condition 721 is selected, the type is "position of search results", and processing proceeds to S804.

[0066] In S804, the CPU 401 of the image processing server 202 selects one text search result that can be a keyword and one text search result that can be a value, and creates a combination of the selected text search results. Since the keyword of the output condition 721 is the text search result 513, and the values of the output condition 721 are the text search results 515 and 516, there are two combinations, that is, the combination of the text search result 513 and the text search result 515, and the combination of the text search result 513 and the text search result 516. First, the CPU 401 selects the text search result 513 and the text search result 515, and the process proceeds to S805.

[0067] In S805, the CPU 401 of the image processing server 202 determines whether the combination of the keyword and the value selected in S805 matches the position condition. When the combination matches the position condition, the process proceeds to S806. When the combination does not match the position condition, the process proceeds to S807. Since the position condition of the output condition 721 is "the value is on the right side of the keyword", the CPU 401 uses the coordinate values of the text search results to determine whether the text search result 515 is on the right side of the text search result 513. In this embodiment, the determination of "whether the value is on the right side of the keyword" is described below using a coordinate system with the upper left origin. When the rectangular coordinates of the text search result of the keyword are the upper left coordinates (KX1, KY1) and the lower right coordinates (KX2, KY2), and the rectangular coordinates of the text search result of the value are the upper left coordinates (VX1, VY1) and the lower right coordinates (VX2, VY2), it is sufficient to satisfy "KX2 < VX1", "KY2 > VY1", and "KY1 < VY2".

[0068] That is, the condition is that the X coordinate value at the left end of the text search result of the value is greater than the X-axis coordinate value at the right end of the text search result of the keyword, and the range of the Y-axis coordinate values of the rectangle of the text search result of the keyword overlaps with the range of the Y-axis coordinate values of the rectangle of the text search result of the value. The method for determining whether the value is on the right side of the keyword according to this embodiment is an example, and other methods can be used. Since the text search result 513 and the text search result 515 do not match the condition, the process proceeds to S807.

[0069] In S807, the CPU 401 of the image processing server 202 determines whether there are remaining combinations of keywords and values. If there are remaining combinations, the process proceeds to S804. If there are no remaining combinations, the process proceeds to S808. After determining the combination of text search results 513 and 515, a remaining combination of keywords and values ​​exists, and the process proceeds to S804. The CPU 401 selects the combination of text search results 513 and 516 as the next combination, and the process proceeds to S805. In S805, the text search result 516 for the value is to the right of the text search result 513 for the keyword, so the process proceeds to S806.

[0070] In S806, since the text search results 513 for the keyword and the text search results 516 for the value satisfy output condition 721, the CPU 401 of the image processing server 202 determines the output value of output condition 721 and proceeds to S807. That is, since the text search results 513 for the keyword and the text search results 516 for the value satisfy output condition 721, the CPU 401 determines the text search result 516 as "total amount" as output. Subsequently, in S807, since all combinations of text search results for "total amount" have been processed, the process proceeds to S808.

[0071] In S808, the CPU 401 of the image processing server 202 determines whether all output conditions have been processed. If there are still unprocessed output conditions, processing proceeds to S801. If there are no remaining unprocessed output conditions, processing ends.

[0072] When the determination of output condition 721 is completed, output conditions 722 and 723 have not yet been processed. Processing proceeds to S801, the next output condition 722 is set as the processing target, and processing proceeds to S802.

[0073] Since output condition 722 indicates the "location of search results" in the same way as output condition 721, it is processed in a similar manner in S802 to S808. CPU 401 evaluates the combination of text search results 514 for keywords and text search results 517 for values, determines that the combination matches output condition 722, and determines text search results 517 as the output value for "phone number".

[0074] After determining output condition 722, in S801, output condition 723 is set as the processing target, and processing proceeds to S802. In S802, CPU 401 retrieves the text search results for "taxi terminology A" from dictionary 703, "taxi terminology B" from dictionary 704, and "taxi terminology C" from dictionary 705, as the targets for the text search results in S604 and S605. As a result, the text search result for "taxi terminology A" from dictionary 703 does not exist. CPU 401 can retrieve the text search result 511 for "taxi terminology B" from dictionary 704 and the text search result 512 for "taxi terminology C" from dictionary 705, and processing proceeds to S803. In S803, since the type of output condition 723 indicates the logical operation of the search results, processing proceeds to S809.

[0075] In S809, the CPU 401 of the image processing server 202 determines whether the text search result matches the logical expression of output condition 723. When the text search result matches the logical expression of output condition 723, processing proceeds to S810. When the text search result does not match the logical expression of output condition 723, processing proceeds to S808. Output condition 723 is whether the logical expression "Taxi term A|(Taxi term B&Taxi term C)" is true. When the number of text search results obtained in S802 is applied to the logical expression of output condition 723, the logical expression is "0|(1&1)" and is true, the text search result matches output condition 723, and processing proceeds to S810.

[0076] In S810, since output condition 723 is met, the CPU 401 of the image processing server 202 determines the output value of output condition 723, and processing proceeds to S808. For output condition 723, the CPU 401 determines "taxi" as the output value of "company name type (company classification type)". Afterwards, in S808, since all output conditions have been processed, processing ends.

[0077] In this embodiment, an output condition (output condition 723) is used to describe the project name of the company name type (company classification type), but it is usually necessary to add output conditions for the number of types of company name types to be determined.

[0078] return Figure 6The flowchart describes the process. In S607, the CPU 401 of the image processing server 202 checks whether the phone number output in S605 exists in the company name dictionary. When the phone number output in S605 exists in the company name dictionary, the CPU 401 identifies the company name from the search results and outputs the search results as the item name "Company Name". The company name dictionary is a database in which phone numbers are associated with company names, and company names can be identified based on phone numbers. However, it is difficult to cover all phone numbers for all company names. When there is a recognition error in the OCR, the company name cannot be searched. For example, when "03-1234-5678" is obtained as a phone number from receipt 500, but the company name cannot be obtained from the company name dictionary, the CPU 401 determines that the company name does not exist, and the process proceeds to S608. Although a method of searching for company names using phone numbers is used, the method is not limited to this. As mentioned above, the company name can be searched not only based on the phone number but also based on the character recognition result.

[0079] In S608, the CPU 401 of the image processing server 202 determines the final company name information based on the "company name" and "company name type (company classification type)" determined in S601 to S607. When an output value for "company name" exists, the CPU 401 determines the output value for "company name". When an output value for "company name" does not exist but an output value for "company name type (company classification type)" exists, the CPU 401 determines the output value for "company name category (company classification category)". When neither an output value for "company name" nor an output value for "company name type (company classification type)" exists, the CPU 401 determines that no output value exists. In this example, since an output value for "company name" does not exist but an output value for "company name type" exists, "taxi" as the "company name category (company classification type)" is determined as the final output of the "company name information".

[0080] As an application Figure 6 and Figure 8 The result of the processing in the flowchart, in the example of receipt 500, outputs three items and the values ​​of the three items: text search results 516 as "total amount", text search results 517 as "phone number", and "taxi" as "company name information".

[0081] Figure 9 A sample receipt 900, serving as a gas station receipt, is shown below. The following describes the use of... Figures 7A to 7CThe information extraction rules shown are used to process receipt 900 in steps S601 to S608. As a result of text searching the document recognition results in S603 and S604, the following results are obtained: No results are obtained for "Taxi Terminology A" in dictionary 703; text search result 901 "License Plate Number" is obtained for "Taxi Terminology B" in dictionary 704; and no results are obtained for "Taxi Terminology C" in dictionary 705. When the number of text search results is applied to the logical expression of output condition 723, the logical expression is "0|(1&0)". The text search results do not match output condition 723 in S809, therefore "Company Name Type" is not output.

[0082] As described above, when applying this embodiment, a dictionary and pattern can be used to search based on the string of the document recognition result, and logical operations can be performed on the search results to output "taxi" as a company name type (company category type). Furthermore, for receipts containing similar terms that are not taxi receipts, the company name type is not output unless an error is confirmed.

[0083] In this embodiment, rule-based conditions are predefined for identifying category types, which are prompts for expense items. The definition of category types can vary from user to user without needing to identify expense items. For processing document images, the company category type identified using rule-based conditions is used.

[0084] Second Embodiment

[0085] In the first embodiment, the occurrence of terms is determined based on document recognition results, and the company name type (company classification type) is determined based on the number of logical operation results. However, other conditions can be used, such as input image information like the size of the receipt.

[0086] Figure 10A An example of an input document image is shown. Assume the input document image is a railway company receipt 1000. Only the logo 1001 exists on receipt 1000, serving as a clue to the railway company's name or the receipt itself. Typically, it's difficult to recognize images with decorations such as logos using OCR, making it hard to directly identify the company name from the OCR string. However, it's possible to output a receipt issued by the railway company and with the same size as a ticket. The receipt's size can have specific characteristics. The following describes a method using image size as an output condition type.

[0087] Figure 10B Table 1010 shows rules for information extraction according to one or more aspects of this disclosure. The output conditions 721 to 723 indicated in Table 1010 are related to… Figure 7CThe output conditions 721 to 723 indicated in Table 720 are the same. Output condition 1011 is the condition for identifying the company name type. When the image size is 85mm wide and 58mm high, "Railway" is output based on output condition 1011. "Image Size" is newly included as a type in output condition 1011. "Image Size" indicates that "Railway" is output when the size information of the input image is the defined size.

[0088] Figure 10C Table 1020 is shown, indicating image information obtained according to receipt 1000 in S601. The image information indicated in Table 1020 indicates a resolution of 300 dpi in the width direction and a resolution of 300 dpi in the height direction, and indicates a width direction of 1000 pixels and a height direction of 680 pixels.

[0089] Figure 11 This is a flowchart illustrating the output processing of item values ​​using image dimensions according to one or more aspects of this disclosure. (By...) Figure 8 The flowchart uses the same symbols to represent the steps (S801, S802, and S804 to S810) of the processing. Figure 8 The corresponding steps shown are the same or similar, and their detailed descriptions are omitted.

[0090] In S1101, the CPU 401 of the image processing server 202 determines the type of output condition. When the type of output condition indicates "location of search results", the process proceeds to S804. When the type of output condition indicates "logical operation of search results", the process proceeds to S809. When the type of output condition indicates "image size", the process proceeds to S1102.

[0091] In S1102, the CPU 401 of the image processing server 202 determines whether the size of the detected original document image matches the conditions. If the size matches the conditions, the process proceeds to S1103. If the size does not match the conditions, the process proceeds to S808. In this embodiment, it is assumed that the image to be processed is an image cropped according to the outline of the original document. However, when the image to be processed is not cropped, the CPU 401 can detect the outline of the original document, calculate the size of the original document, and determine whether the size of the original document matches the conditions.

[0092] In S1103, the CPU 401 of the image processing server 202 determines the output value. When based on... Figure 11When processing receipt 1000 in the flowchart shown, in S1102, CPU 401 determines whether receipt 1000 matches the size of output condition 1011. Based on a resolution of 300 dpi, 1000 pixels in the width direction corresponds to 85 mm, and based on a resolution of 300 dpi, 680 pixels in the height direction corresponds to 58 mm. Since the size of receipt 1000 matches the image size of output condition 1011, "Railway" is output as a company name type (company classification type).

[0093] In the example of output condition 1011, only image size is used as a condition. However, this embodiment is not limited to this. Not only image size can be used, but also composite conditions obtained by combining the results of logical operations obtained from text search results from OCR strings can be used.

[0094] In addition, not only image dimensions can be used, but also features of the original document (such as paper color and paper background).

[0095] As described above, when this embodiment is applied, even if the characteristics of the company name type are not obtained from the OCR string, the output value can be output as the company name type by using image information such as image size.

[0096] Third Embodiment

[0097] In the first embodiment, when both a company name type (company classification type) and a specific company name are identified, the specific company name is output as the final information about the company name in S608. Typically, when both a company name type and a specific company name exist, the specific company name provides more detailed information than the company name type. Therefore, in cases where it is difficult to extract the company name, the company name type is used as supplementary information for expense reimbursement. However, in some situations, it may be desirable to output the company name type (classification type).

[0098] Figure 12 A receipt 1200 for an airline ticket is shown according to one or more aspects of this disclosure. Receipt 1200 is a receipt obtained when paying for the ticket at a convenience store, and the convenience store's company name and telephone number are written on the receipt 1200. When using a telephone number and company name dictionary to output the company name, the company name is "CamonMart," which is the name of the convenience store. However, in expense reimbursement work, it is more appropriate to output the type of supporting information (air ticket) used for input purposes than to output the convenience store-specific company name.

[0099] Figure 13A , Figure 13B and Figure 13C Examples of rules based on one or more aspects of this disclosure are shown. Figure 13ATable 1300 shows dictionaries 1301, 1302, 1303, and 1304. Dictionary 1301 is a dictionary of airline ticket terms, including the search strings "airline" and "aviation," which are hints that the company name type (purchase type) is an airline ticket. Dictionary 1302 is a dictionary of convenience store terms, including the search string "convenience store," which is a hint issued by a convenience store. Dictionary 1303 is a dictionary of website terms, including the term "display date," which is a hint for receipts issued via a website. Dictionary 1304 is a dictionary of agency terms, including search strings such as "travel," "agent," and "agency," which are hints for receipts issued by travel agencies.

[0100] Figure 13B Table 1310 shows a list of output conditions using the aforementioned terms. Output condition 1311 has the item name "Company Name Type (Purchase Type)", the condition type "Logical Operation of Search Results", and the condition "Airline Ticket Terms and Convenience Store Terms", and outputs "Airline Ticket (Convenience Store)" as the output value. Output condition 1312 has the item name "Company Name Type (Purchase Type)", the condition type "Logical Operation of Search Results", and the condition "Airline Ticket Terms and Website Terms", and outputs "Airline Ticket (Online)" as the output value. Output condition 1313 has the item name "Company Name Type (Purchase Type)", the condition type "Logical Operation of Search Results", and the condition "Airline Ticket Terms and Agency Terms", and outputs "Airline Ticket (Agency)" as the output value. The fact that the item names of output conditions 1311, 1312, and 1313 are defined as Company Name Type (Purchase Type) indicates that, in addition to the information on the company name type, information on the seller who purchased the air ticket, i.e., the seller who paid for the air ticket, is also output. Output condition 1311 should output "Airline Ticket (Convenience Store)", meaning that if both terms indicating an airline ticket and a convenience store term exist, it should indicate information about an airline ticket paid for at a convenience store. Output condition 1312 should output "Airline Ticket (Online)", meaning that if both terms indicating an airline ticket and a website-issued term exist, it should indicate information about an airline ticket paid for online. Output condition 1313 should output "Airline Ticket (Agency)", meaning that if both terms indicating an airline ticket and an agency-issued term exist, it should indicate information about an airline ticket paid for at an agency.

[0101] The following describes how to process the image of receipt 1200 according to the dictionary indicated in Table 1300 and the output conditions indicated in Table 1310. Figure 6 and Figure 8The flowchart illustrates the processing. On receipt 1200, there are text search results 1201 "Airline" (searched using dictionary 1301) and text search results 1202 "Convenience Store" (searched using dictionary 1302). Therefore, the text search results match the conditions of output condition 1311, and the output value "Airline (Convenience Store)" is obtained as the company name type (purchase type).

[0102] Figure 13C Table 1320 is shown as a list of conditions used to determine the company name information to be output in S608. The company name information determination condition list 1320 includes: a condition number column; a condition column indicating whether a company name has been obtained; a condition column indicating whether a company name type (purchase type) has been obtained; a condition column indicating the purchase type; and a column indicating that the output value is used as the company name information output when the result matches the condition. Determination is performed sequentially starting from the top row of the table, and the output of the company name information corresponding to the row indicating that the result matches the condition is used. Company name information determination condition 1321 indicates that when a "company name" exists, a "company name type" exists, and the "purchase type" is convenience store, "company name category (purchase type)" is output as company name information. Company name information determination condition 1322 indicates that when a "company name" exists, a "company name type" exists, and the "purchase type" is agency, "company name category (purchase type)" is output as company name information. Condition 1323 for determining company name information indicates that when a "Company Name" exists but the "Company Name Type (Purchase Type)" is not obtained, the "Company Name" will be used as the company name information. Condition 1324 for determining company name information indicates that when a "Company Name Type" exists but a "Company Name Type" does not exist, the "Company Name Type (Purchase Type)" will be output. Condition 1325 for determining company name information indicates that when neither a "Company Name" nor a "Company Name Type" is obtained, company name information will not be output.

[0103] When company name information is determined based on table 1320, the result obtained from receipt 1200 matches the company name information determination condition 1321, and the purchase type "airline ticket" is output as company name information instead of company name.

[0104] As described above, when applying this embodiment, except that a specific company name must be used when simply obtaining the company name, the values ​​required for expense reimbursement can be output by prioritizing the company name type under certain conditions.

[0105] Other embodiments

[0106] Embodiments of the invention can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (also more fully referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above embodiments, and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions in the above embodiments. Furthermore, embodiments of the invention can be implemented using a method by which the computer of the system or device, for example, reads and executes the computer-executable instructions from the storage medium to perform one or more functions in the above embodiments, and / or controls the one or more circuits to perform one or more functions in the above embodiments. The computer may include one or more processors (e.g., central processing unit (CPU), microprocessor unit (MPU)) and may include separate computers or a network of separate processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), the memory of a distributed computing system, or an optical disc (such as a compact disc (CD), a digital versatile optical disc (DVD), or a Blu-ray disc (BD)). TM One or more of the following: flash memory devices and memory cards.

[0107] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0108] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The appended claims should be given the broadest possible scope to cover all such variations and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: The obtaining unit obtains one or more strings identified from the document image through character recognition processing of the document image; The first identification unit identifies information about the company that issued the document based on one or more strings and predefined conditions. The second identification unit identifies the company name of the company that issued the document, the company name being included in the obtained one or more strings; as well as The output unit outputs information about the company name. Specifically, if the company name is identified, the identified company name will be output as information about the company name, and In cases where the company name is not identified but information about the company is identified, the identified information about the company will be output as information about the company name instead of the company name.

2. The image processing apparatus according to claim 1, wherein The predefined condition is: whether a predetermined string exists among the obtained one or more strings.

3. The image processing apparatus according to claim 1, wherein The predefined conditions include conditions based on logical operations.

4. The image processing apparatus according to claim 1, wherein, The predefined conditions include those based on image size.

5. The image processing apparatus according to claim 1, wherein The first identification unit identifies information about the company that issued the document and identifies the information about the seller based on the one or more strings obtained and another predefined condition.

6. An image processing method, comprising: The step involves obtaining one or more strings identified from the document image through character recognition processing of the document image; The first identification step involves identifying information about the company that issued the document based on one or more strings obtained and predefined conditions. The second identification step is to identify the company name of the company that issued the document, the company name being included in the obtained one or more strings; as well as Output steps: Output information about the company name. Specifically, if the company name is identified, the identified company name will be output as information about the company name, and In cases where the company name is not identified but information about the company is identified, the identified information about the company will be output as information about the company name instead of the company name.

7. The image processing method of claim 6, wherein, The predefined condition is: whether a predetermined string exists among the obtained one or more strings.

8. The image processing method of claim 6, wherein, The predefined conditions include conditions based on logical operations.

9. The image processing method of claim 6, wherein, The predefined conditions include those based on image size.

10. The image processing method according to claim 6, wherein, In the first identification step, information about the company that issued the document and the distributor is identified based on the one or more strings obtained and another predefined condition.

11. A non-transitory computer-readable storage medium storing a program, wherein The program causes the processor to perform: One or more strings are obtained from the document image by character recognition processing; Based on one or more strings obtained and predefined conditions, identify information about the company that issued the document; Identify the company name of the company that issued the document, the company name being included in the obtained one or more strings; and Output information about the company name. Specifically, if the company name is identified, the identified company name will be output as information about the company name, and In cases where the company name is not identified but information about the company is identified, the identified information about the company will be output as information about the company name instead of the company name.

12. The non-transitory computer-readable storage medium according to claim 11, wherein, The predefined condition is: whether a predetermined string exists among the obtained one or more strings.

13. The non-transitory computer-readable storage medium according to claim 11, wherein, The predefined conditions include conditions based on logical operations.

14. The non-transitory computer-readable storage medium according to claim 11, wherein, The predefined conditions include those based on image size.

15. The non-transitory computer-readable storage medium according to claim 11, wherein, In identifying information about the company that issued the document, information about the company that issued the document and information about the distributor are identified based on one or more strings obtained and another predefined condition.

Images