Information processing system, form recognition method, program, and form recognition system

Abstract

To provide an information processing system, a form recognition system, a form recognition method, and a program for more appropriately extracting item values written in a plurality of lines.SOLUTION: In a form recognition system 100, a server apparatus 200 provides a terminal apparatus 400 with recognition results obtained by recognizing character strings from image data in a tabular format having rows and columns. The server apparatus 200 comprises a form recognition processing unit 220 for concatenating item values of a first row that contains values in the preset column heading column and the item values of a second row that does not contain preset exclusion terms and is not blank, and a display screen generation unit for providing the terminal apparatus 400 with the item values of the first row and the second row that are concatenated by the form recognition processing unit 220.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a form recognition method, a program, and a form recognition system. [Background technology]

[0002] There is a known technology that scans forms in which field names and their values ​​are arranged in a table format and converts them into text data. Each company creates forms to suit their business, so the form formats vary from company to company. For this reason, if the form is not recognized correctly, the user must directly specify the area from which to extract the field values.

[0003] A technique for setting an area corresponding to a predetermined item value with a simple operation is known (see, for example, Patent Document 1). Patent Document 1 discloses an information processing system that displays an extraction guide on a tabular image and accepts settings for the position of the extraction guide to specify an extraction area. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have the problem that they cannot properly extract item values ​​written on multiple lines. For example, if the content of a description spans multiple lines, the content will be extracted in pieces if the line separators are not properly recognized.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an information processing system that can more appropriately extract item values ​​written in multiple lines. [Means for solving the problem]

[0006] In view of the above-mentioned problems, the present invention provides an information processing system that provides a terminal device with a recognition result obtained by recognizing a character string from image data in a table format having rows and columns, and that recognizes a character string in a first row having a value in a column of a preset column header from the recognition result. In a given columnThe field value and the second line that does not contain the predefined exclusion terms and is not a blank line of the predetermined column Field Value and Concatenate And, Tied Section The target price is sent to the terminal device. Display this It is characterized by the following. [Effects of the Invention]

[0007] It is possible to provide an information processing system that can more appropriately extract item values ​​written on multiple lines. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a document recognition system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a server device. [Figure 3] FIG. 2 illustrates an example of a hardware configuration of a terminal device. [Figure 4] FIG. 10 is a diagram illustrating an example of an industry determination database. [Figure 5] FIG. 10 is a diagram showing types of detail definition information stored in a detail definition database. [Figure 6] FIG. 10 is a diagram illustrating an example of general-purpose detail definition information. [Figure 7] FIG. 10 is a diagram illustrating an example of business type specification definition information. [Figure 8] FIG. 10 is a diagram illustrating an example of individual company specification definition information. [Figure 9] FIG. 2 is a diagram illustrating functions of a server device. [Figure 10] FIG. 2 is a diagram illustrating functions of a terminal device. [Figure 11A] 1 is an example of a sequence diagram illustrating the operation of a document recognition system (part 1); [Figure 11B] 10 is an example of a sequence diagram illustrating the operation of the document recognition system (part 2). [Figure 12] FIG. 10 is a diagram illustrating an example of a job list. [Figure 13]FIG. 10 is a first flowchart illustrating an example of processing by a form recognition unit. [Figure 14] FIG. 10 is a second flowchart illustrating an example of processing by the form recognition unit. [Figure 15] FIG. 10 is a third example flowchart illustrating the processing of the document recognition unit. [Figure 16] FIG. 2 is a first diagram illustrating form image data. [Figure 17] FIG. 10 is a second diagram illustrating extraction of detailed information. [Figure 18] FIG. 10 is a third diagram illustrating detailed information of the detailed section generated based on the read result data. [Figure 19] FIG. 10 is a diagram illustrating an example of a recognition result confirmation screen. [Figure 20] 10A and 10B are diagrams illustrating an example of adjusting line breaks in a recognition result. [Figure 21] 10 is a flowchart illustrating an example of a procedure in which a form recognition unit connects a plurality of rows. [Figure 22] FIG. 10 is a diagram illustrating the concatenation of rows labeled A and D. [Figure 23] FIG. 10 is a diagram showing an example of concatenation of rows labeled A and D. [Figure 24] 22 is a diagram showing labels A to D set to each line of the form image data by the linking process of FIG. 21. FIG. [Figure 25] FIG. 22 is a diagram illustrating the product names linked by the linking process of FIG. 21. [Figure 26] FIG. 10 is a diagram illustrating a method for recognizing line breaks to be compared. [Figure 27] FIG. 10 is a diagram illustrating an outline of a method for recognizing line breaks. [Figure 28] 10 is a sequence diagram illustrating an example of an operation of the document recognition system (embodiment 2); [Figure 29] FIG. 29 is a diagram for explaining the product names linked by the linking process of FIG. 28 (Example 2). [Figure 30] FIG. 10 is a diagram showing a part of a multi-column form in which consumption tax is also displayed in the amount column (Example 2). [Figure 31]FIG. 10 is a diagram showing an example of drawing logical ruled lines of a multi-column form when the user specifies that the row separators of the logical ruled lines are to be in the "amount" column, skipping every other row (Example 2). DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing system and a form recognition method performed by the information processing system will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings. [Example]

[0010] <Terminology> In this embodiment, the user refers to, for example, an establishment, company, organization, etc. that has concluded a contract to use the services provided by the document recognition system, as well as their employees and staff.

[0011] A user obtains a document from a business partner, which is the issuer of the document, and uses a document recognition system to recognize the document for accounting purposes.

[0012] A row is a horizontal arrangement of characters, etc., and a column is a vertical arrangement of characters, etc. A table is an arrangement of things that has at least some rows and columns and is used to make things easier to read. A display method in which things are arranged in this way is called tabular format. When rows are separated by lines, a line is the space between the lines, and when there are no lines, a line is the entire length of the table, including the area where the characters are arranged horizontally. A blank line is a line that is approximately one character high. A blank line may contain a simple string of symbols (for example, --------, ******, etc.).

[0013] If columns are separated by vertical lines, a column is defined as the space between the lines. If columns are not separated by vertical lines, a column is defined as the width of the first character string (column header) of the table separated by spaces on the left and right, and the height of the column is defined as the length of the lowest character string vertically below this character string.

[0014] Correct line separation in a form refers to the line separation intended by the issuer of the form. In other words, correct line separation in a form refers to determining which lines should be joined when the content of the form spans multiple lines. Naturally, the line separation intended by the issuer is also correct for the user. In this embodiment, a character string refers to alphanumeric characters, kana characters, kanji characters, various symbols (!$%, etc.), etc., and the description will be given assuming that the item value for the item name is a character string. Depending on the form, the character string may also include marks and codes that are only used within the company.

[0015] <System configuration example> 1 is a diagram showing an example of the system configuration of a form recognition system. The form recognition system 100 of this embodiment includes a server device 200, an image forming device 300, and a terminal device 400. In the form recognition system 100 of this embodiment, the server device 200, the image forming device 300, and the terminal device 400 are connected via a network. The server device 200 of this embodiment is an example of an information processing system.

[0016] Furthermore, the server device 200 is connected to a core system 500, for example, via a network. The core system 500 is a system that performs specific processing using the results of recognizing a form in the form recognition system 100 of this embodiment. Specifically, the core system 500 may be, for example, an accounting system, an estimate creation system, or the like.

[0017] In the document recognition system 100 of this embodiment, the server device 200 extracts the items and item values ​​contained in the document from the image data of the document read by the scanner function of the image forming device 300, and converts the extracted items and item values ​​into text data in which the items and item values ​​are associated with each other.

[0018] The server device 200 may be located on the Internet or on-premise. If it is located on the Internet, it should be compatible with cloud computing. Cloud is a term used when no specific hardware resources are intended.

[0019] In the following description, the image or image data of the form read by the scanner function of the image forming apparatus 300 is referred to as a form image or form image data.

[0020] In the following description, extracting items and their item values ​​included in a form image, converting the items and item values ​​into text data, and associating them is referred to as form recognition.

[0021] The server device 200 of this embodiment has a storage unit 210 and a form recognition processing unit 220. The storage unit 210 is provided with a form image database 230, an industry determination database 240, a description definition database 250, and a recognition result database 260.

[0022] The form image database 230 stores form image data. The business type determination database 240 stores information that is referenced when determining the business type of the issuer of a form. The detail definition database 250 stores detail definition information that is referenced when performing form recognition by the form recognition processing unit 220. The recognition result database 260 stores recognition result data, which is information that indicates the results of form recognition by the form recognition processing unit 220.

[0023] The form recognition processing unit 220 of this embodiment acquires form image data stored in the form image database 230, and refers to the industry determination database 240 to determine the industry of the issuer of the form and identify the detail definition information to be referenced. The form recognition processing unit 220 then performs form recognition using the identified detail definition information in the detail definition database 250, and stores the results in the recognition result database 260. The detail definition information includes information indicating the item names of the detail items included in the form and the positions where the item names are written.

[0024] Here, when performing form recognition, the document recognition processing unit 220 of this embodiment refers to the individual company detail definition information, industry detail definition information, and general-purpose detail definition information stored in the detail definition database 250.

[0025] The general-purpose detail definition information in this embodiment is information that defines words corresponding to detail item names in general forms. The industry detail definition information is information that defines words corresponding to detail item names in forms for each industry. The individual company detail definition information is information that defines words corresponding to detail item names in forms specific to the user who uses the form recognition system 100 and information to be extracted from the form image. Details of the industry determination database 240 and the detail definition database 250 will be described later.

[0026] In this embodiment, by using three types of detail definition information when recognizing a form, it is possible to recognize the detail details of a form in addition to general-purpose forms, thereby improving the accuracy of form recognition.

[0027] A brief explanation of the details section of a form is provided below. The details section of a form is the section that shows the results of data aggregation, and is the section where data in the same format is printed repeatedly vertically for each detail item. A detail item is a data item that is aggregated in the details section. Typical detail items include product name, quantity, unit price, amount, etc. Specific examples of the details section will be provided later.

[0028] In the document recognition system 100 of this embodiment, the image forming device 300 is a multifunction peripheral with a scanner function. The image forming device 300 is equipped with applications for realizing functions such as copying, faxing, and scanning, and these functions are realized by selecting the application corresponding to each function. The image forming device 300 may also be an information processing device equipped with a camera, such as a smartphone, document camera, or tablet terminal. An OCR application runs on these information processing devices, and the image forming device 300 can perform processing similar to that of this embodiment, either independently or in cooperation with the server device 200. Alternatively, a standalone reading function such as a scanner device may be connected to an information processing device equipped with an OCR application via a network, or the OCR function may be provided in a server device.

[0029] The terminal device 400 of this embodiment is used by a user who uses the form recognition system 100. Furthermore, the terminal device 400 may display the results of form recognition by the server device 200.

[0030] 1, the server device 200 has four databases in the storage unit 210, but this is not limiting. A part of each database may be provided in an external device of the server device 200, or all of each database may be provided in an external device.

[0031] 1, the server device 200 implements the form recognition processing unit 220, but the present invention is not limited to this. The form recognition processing unit 220 may be implemented by a plurality of information processing devices.

[0032] In addition, in the example of Figure 1, the document recognition system 100 includes one image forming device 300 and one terminal device 400, but the number of image forming devices 300 and one terminal device 400 included in the document recognition system 100 may be any number.

[0033] Next, the hardware configuration of the server device 200 of this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the hardware configuration of the server device 200.

[0034] The server device 200 of this embodiment is constructed by a computer, and as shown in FIG. 2, includes a CPU 201, a ROM 202, a RAM 203, an HD 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 208, a network I / F 209, a data bus B, a keyboard 211, a pointing device 212, an optical drive 214, and a media I / F 216.

[0035] Of these, the CPU 201 controls the overall operation of the server device 200. The ROM 202 stores programs used to drive the CPU 201, such as an IPL. The RAM 203 is used as a work area for the CPU 201. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the CPU 201. The display 206 is a display unit that displays various information such as a cursor, menus, windows, characters, or images.

[0036] The external device connection I / F 208 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 209 is an interface for data communication using the communication network N. The data bus B is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG. 2.

[0037] The keyboard 211 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 214 controls reading and writing of various data from an optical storage medium 213, which is an example of a removable storage medium. The optical storage medium 213 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 216 controls reading and writing (storing) of data from a storage medium 215, such as a flash memory.

[0038] The server device 200 of this embodiment is, for example, a smartphone, a tablet terminal, a PDA, (Personal Digital Assistant), wearable PC, etc.

[0039] The terminal device 400 of this embodiment is a computer having an arithmetic processing unit and a storage device, and may be, for example, a PC (Personal Computer), a tablet terminal, a smartphone, etc. Figure 3 is a diagram showing an example of the hardware configuration of the terminal device 400.

[0040] The terminal device 400 of this embodiment includes a CPU 401 , a ROM 402 , a RAM 403 , an EEPROM 404 , a CMOS sensor 405 , an image sensor I / F 406 , an acceleration / direction sensor 407 , a media I / F 409 , and a GPS receiving unit 411 .

[0041] Of these, the CPU 401 is an arithmetic processing unit that controls the overall operation of the terminal device 400. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as the IPL. The RAM 403 is used as a work area for the CPU 401. The EEPROM 404 reads or writes various data, such as smartphone programs, under the control of the CPU 401. The ROM 402, RAM 403, and EEPROM 404 are examples of storage devices of the terminal device 400.

[0042] The CMOS (Complementary Metal Oxide Semiconductor) sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 401. Note that the CMOS sensor may be an imaging means such as a CCD (Charge Coupled Device) sensor.

[0043] The imaging element I / F 406 is a circuit that controls the driving of the CMOS sensor 305. The acceleration / azimuth sensor 407 is a variety of sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 409 controls the reading and writing (storage) of data from and to a recording medium 408, such as a flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.

[0044] The terminal device 400 also includes a long-distance communication circuit 412, an antenna 412a of the long-distance communication circuit 412, a microphone 415, a speaker 416, an audio input / output I / F 417, a display 418, an external device connection I / F (Interface) 419, a short-distance communication circuit 420, an antenna 420a of the short-distance communication circuit 420, and a touch panel 421.

[0045] Of these, the long-distance communication circuit 412 is a circuit that communicates with other devices via a communication network. The microphone 415 is a built-in circuit that converts sound into an electrical signal. The speaker 416 is a built-in circuit that converts the electrical signal into physical vibrations to produce sounds such as music or voice. The sound input / output I / F 417 is a circuit that processes the input and output of sound signals between the microphone 415 and the speaker 416 under the control of the CPU 401.

[0046] The display 418 is a type of display means such as a liquid crystal display or organic electroluminescence (EL) display that displays an image of a subject, various icons, etc. The external device connection I / F 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows a user to operate the terminal device 400 by pressing the display 418. The display 418 is an example of a display unit that the terminal device 400 has.

[0047] <Database configuration example> Next, the business type determination database 240 and the description definition database 250 included in the server device 200 of this embodiment will be described.

[0048] 4 is a diagram showing an example of the business type determination database 240. The business type determination database 240 of this embodiment is provided for each of various business types.

[0049] In the business type determination database 240, business types are associated with the issuers of documents. In the example of Fig. 3, the business type "retail" is associated with companies A, B, and C. Therefore, it can be seen that companies A, B, and C are organizations engaged in the retail business.

[0050] Next, the detail definition database 250 of this embodiment will be described with reference to Figures 5 to 8. Figure 5 is a diagram showing the types of detail definition information stored in the detail definition database.

[0051] The detail definition database 250 of this embodiment stores individual company detail definition information 251, industry detail definition information 252, and general-purpose detail definition information 253. The individual company detail definition information 251 is detail definition information that defines the detail item names and writing positions included in forms that are uniquely designed by the issuer (or used in a management system or the like that is uniquely managed by the issuer), and is detail definition information exclusive to the issuer. Of the three detail definition information, the individual company detail definition information 251 is the definition that should be most respected and is the detail definition information with the highest priority.

[0052] The industry detail definition information 252 of this embodiment is detail definition information that defines the item names and positions of items included in forms that are generally used in each industry. The priority of the industry detail definition information 252 is intermediate among the three detail definition information.

[0053] The general-purpose detail definition information 253 in this embodiment is detail definition information that defines the detail item names and writing positions included in a highly versatile form that can be used by multiple users. The general-purpose detail definition information 253 is the detail definition information with the lowest priority among the three detail definition information.

[0054] The form recognition processing unit 220 of this embodiment refers to these three pieces of detail definition information to recognize form image data. Therefore, in this embodiment, even if the form image data is for a form that includes a detail item name that is not defined in the individual company detail definition information 251, this form image data can be recognized as long as it is defined in the industry detail definition information 252 or the general-purpose detail definition information 253.

[0055] In the following explanation, an invoice will be used as an example of a form. However, the form may be any form that includes a detailed description. Specifically, the form may be, for example, a delivery note, an estimate, a purchase slip, etc.

[0056] If the document is a delivery note, the issuer is the supplier who issued the delivery note, and if the document is a purchase slip, the issuer is the supplier who issued the purchase slip.

[0057] 6 is a diagram showing an example of general-purpose detail definition information. General-purpose detail definition information 253 defines the item names used in the details of a general invoice, and has item names and keywords as information items, with the two being associated with each other.

[0058] The value of the "item name" field indicates the name of the item used in the invoice details. The value of the "keyword" field indicates a keyword (word) that is considered synonymous with the item name in the details.

[0059] That is, the general-purpose detail definition information 253 is information that defines detail items used in general forms and keywords to be extracted as detail items from forms.

[0060] In the example of Figure 6, the detailed item name "product name" is treated as synonymous with keywords such as "product," "product," "item name," etc. Also, in the example of Figure 6, the detailed item name "amount" is treated as synonymous with keywords such as "amount," "total amount," and "total."

[0061] In addition, the example in Figure 6 indicates that the detailed item name "order number" is treated as synonymous with keywords such as "order no.", "slip number," etc. In addition, the example in Figure 6 indicates that the detailed item name "product code" is treated as synonymous with keywords such as "product no.", "product code," etc.

[0062] 7 is a diagram showing an example of business type detail definition information. Business type detail definition information 252 is information provided for each business type, and defines the item names used in the details of a general invoice for the industry indicated by the business type.

[0063] The business type detailed definition information 252 has information specifying the business type, an item "detailed item name," and an item "keyword," which are associated with each other. In the example of Fig. 7, the business type is shown as "retail business."

[0064] The value of the "Item Name" field indicates the name of the item used in the details of a typical invoice. The value of the "Keyword" field indicates the keyword (word) that corresponds to the general item name in an invoice from a retailer.

[0065] That is, the business type detail definition information 252 is information that defines the business type, the detail items used in the form, and the keywords to be extracted as the detail items from the form.

[0066] The example in Figure 7 shows that the item name "Product Name" on a typical form may be replaced with keywords such as "Product Name" or "Product Name" in the details of an invoice issued by a retailer.

[0067] 8 is a diagram showing an example of individual company detailed definition information. In this embodiment, the individual company detailed definition information 251 is registered by the user for each issuer. Even if the issuer is the same, the individual company detailed definition information 251 may differ depending on the user.

[0068] In the individual company detail definition information 251, information specifying the billing source is associated with the item "detail item name" and the item "extracted information."

[0069] The information identifying the billing source includes the name, telephone number, account number, etc. In the example of Fig. 8, the information identifying the billing source includes the name, telephone number, and account number of the billing source, but the information identifying the billing source may be any one of these pieces of information.

[0070] The value of the item "detailed item name" indicates the name of the detailed item used in the details of a general invoice. In other words, the detailed item name in the individual company detailed definition information 251 indicates the detailed item used in the form.

[0071] The value of the item "Extracted Information" includes keywords (words) to be extracted as keywords corresponding to general item details in the invoice issued by the billing source, and information indicating the position where this keyword is written in the document image showing the invoice.

[0072] That is, the extraction information in the individual company detail definition information 251 includes keywords extracted as detail items from the form and information indicating the positions of the keywords in the form. Therefore, the individual company detail definition information 251 is information that defines the detail items used in the form, the keywords extracted as detail items, and information indicating the positions of the keywords in the form.

[0073] The example in Figure 8 shows that the item name "Product Name" in a typical form may be replaced with a keyword such as "Product Name" in the details of an invoice from Company A.

[0074] In addition, the example in Figure 8 shows that the information extracted to indicate the position of the keyword "item name" in the form image showing an invoice from Company A includes keyword coordinates indicating the coordinates of the keyword, cell coordinates indicating the coordinates of the cell in which the keyword "item name" is entered, and column coordinates indicating the coordinates of the column in which the keyword "item name" is entered.

[0075] In addition, the information indicating the position where a keyword is written in a form image showing an invoice can be keyword coordinates. Keyword coordinates are the coordinates of two points on a diagonal line in a rectangle circumscribing the keyword. From these coordinates, the width and height of the rectangle circumscribing the keyword can be determined. Similarly, cell coordinates are the coordinates of two points on a diagonal line in a rectangle circumscribing the character string in the cell immediately below the keyword. Note that the width and height of cells in the same column are assumed to be the same. Column coordinates are the coordinates of two points on a diagonal line in a rectangle circumscribing the character string in the cell immediately below the keyword to the character string in the bottommost cell.

[0076] <Server device functions> Next, the functions of the server device 200 of this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining the functions of the server device 200.

[0077] In the server device 200 of this embodiment, the CPU 201 reads and executes an information processing program stored in the HD 204, RAM 203, or the like, thereby realizing the functions of each unit described below.

[0078] The server device 200 of this embodiment includes a form recognition processing unit 220, a definition registration unit 290, and a communication unit 298. The form recognition processing unit 220 includes an input receiving unit 221, an authentication processing unit 222, and a form recognition unit 270.

[0079] The input accepting unit 221 accepts various inputs to the server device 200. Specifically, the input accepting unit 221 accepts input of authentication information for logging in to the form recognition system 100. The authentication information is, for example, a user ID and a password. The input accepting unit 221 also accepts input of form image data transmitted from the image forming device 300, for example. The input accepting unit 221 may accept input to the server device 200 via the communication unit 298, or may accept input made directly to the server device 200.

[0080] The authentication processing unit 222 performs authentication based on the authentication information input and accepted by the input accepting unit 221. Note that the authentication processing unit 222 may, for example, transmit the input authentication information to an authentication server provided outside the server device 200 and obtain the result of authentication by the authentication server. When the user is authenticated, the authentication processing unit 222 may pass the form image data accepted by the input accepting unit 221 to the form recognition unit 270.

[0081] The document recognition unit 270 of this embodiment has an image data storage unit 271, a job list generation unit 272, an image data acquisition unit 273, a preprocessing unit 274, a character recognition unit 275, a definition identification unit 276, an item extraction unit 277, a display screen generation unit 278, and an output unit 279.

[0082] When the input receiving unit 221 receives form image data from the image forming apparatus 300 via the communication unit 298 , the image data storage unit 271 stores the data in the form image database 230 .

[0083] When the input receiving unit 221 receives input of form image data from the image forming apparatus 300 via the communication unit 298, the job list generating unit 272 registers the job in the job list and manages the job list. Details of the job list will be described later.

[0084] The image data acquisition unit 273 acquires form image data stored in the form image database 230. The preprocessing unit 274 performs preprocessing on the form image data acquired by the image data acquisition unit 273 in order to perform form recognition.

[0085] The character recognition unit 275 extracts character strings and information indicating the positions where the character strings are arranged from the form image data, and stores the extracted data as read result data. The read result data will be described in detail later.

[0086] The definition specifying unit 276 is an acquisition unit that specifies the detail definition information to be referenced during document recognition based on the character string included in the read result data, and acquires the specified detail definition information from the detail definition database 250.

[0087] The item extraction unit 277 extracts and associates items and item values ​​included in the form image data based on the read result data to generate recognition result data, and stores the data in the recognition result database 260 .

[0088] At this time, the item extraction unit 277 refers to the detail definition information identified by the definition identification unit 276, and extracts the detail item names written in the detail section of the form and the item values ​​of the detail items in association with each other.

[0089] In the following description, information that associates the item values ​​of the detail items with the item names written in the detail sections of the form may be referred to as detail information. The detail information in this embodiment is information included in the recognition result data and is a part of the recognition result data.

[0090] Furthermore, the recognition result data in this embodiment includes information (issuer information) that indicates the issuer of the form, which is written on the form, etc. In other words, the recognition result data may include all character strings written on the form.

[0091] The display screen generation unit 278 generates a recognition result confirmation screen using the recognition result data stored in the recognition result database 260. The display screen generation unit 278 instructs the terminal device 400 to display a screen via the communication unit 298. Specifically, the display screen generation unit 278 causes the terminal device 400 to display a screen for generating individual company description definition information. In addition, the display screen generation unit 278 generates an XML format file for displaying the screen, and causes the terminal device 400 to display an input screen on the web browser of the terminal device 400 via the communication unit 298.

[0092] In the present embodiment, the display screen generation unit 278 displays a screen for generating the individual company description definition information 251 on the terminal device 400, but is not limited to this. The device on which the display screen generation unit 278 displays the screen may be the server device 200 or a device other than the terminal device 400, and may be any display device having a display function.

[0093] The output unit 279 outputs the recognition result confirmation screen generated by the display screen generation unit 278 via the communication unit 298. In other words, the output unit 279 causes the recognition result confirmation screen generated by the display screen generation unit 278 to be displayed on the web browser of the terminal device 400 via the communication unit 298.

[0094] The definition registration unit 290 of this embodiment includes an operation content acquisition unit 291 , a definition generation unit 293 , and a definition update unit 294 .

[0095] The operation content acquisition unit 291 receives an operation input to a screen displayed on the web browser of the terminal device 400 via the communication unit 298. More specifically, in this embodiment, the communication unit 298 receives information indicating an operation from the terminal device 400, and the operation content acquisition unit 291 receives the input of the information received by the communication unit 298.

[0096] The definition generating unit 293 generates individual company specification definition information based on information input from the terminal device 400 to the input screen, and stores the information in the specification definition database 250 .

[0097] The definition update unit 294 updates the individual company detailed definition information by reflecting the corrections to the individual company detailed definition information already registered in the detailed definition database 250 .

[0098] The communication unit 298 of the present embodiment is a function realized by, for example, the network I / F 209 or the like, and transmits and receives data between the server device 200 and an external device. Specifically, for example, the communication unit 298 transmits a notification to the terminal device 400 instructing it to display a screen in response to control from the display screen generation unit 278.

[0099] <Terminal device functions> Next, the functions of the terminal device 400 of this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining the functions of the terminal device 400.

[0100] The functions of each unit described below are realized by the CPU 401 of the terminal device 400 reading and executing programs stored in the EEPROM 404, the ROM 402, etc.

[0101] The terminal device 400 of this embodiment has an operation reception unit 431, a display control unit 432, and a communication unit 433. The operation reception unit 431 receives input to a screen displayed on the web browser of the terminal device 400 by the display control unit 432.

[0102] The display control unit 432 controls the display on the display 418 of the terminal device 400. Specifically, the display control unit 432 causes the display 418 to display a screen in accordance with an instruction from the server device 200.

[0103] The communication unit 433 is a function realized by the long-distance communication circuit 412, the antenna 412a of the long-distance communication circuit 412, etc., and transmits and receives data between the terminal device 400 and other devices.

[0104] <Action or Processing> Next, the operation of the form recognition system 100 of this embodiment will be described with reference to Figures 11A and 11B. Figures 11A and 11B are sequence diagrams illustrating the operation of the form recognition system.

[0105] In the form recognition system 100, when the image forming apparatus 300 receives a request from a user to start an application for performing form recognition (step S1101), the image forming apparatus 300 starts the application (step S1102).

[0106] Next, when the image forming apparatus 300 receives authentication information input from the user (step S1103), it transmits the authentication information to the server apparatus 200 and makes an authentication request (step S1104).

[0107] The communication unit 298 of the server device 200 receives the authentication request, performs authentication using the authentication processing unit 222, and notifies the image forming device 300 of the result (step S1105). Note that the operation shown here is for the case where the user authentication is successful.

[0108] Next, the image forming apparatus 300 receives an instruction to scan the form (step S1106), scans the form, and acquires form image data (step S1107). Next, the image forming apparatus 300 transmits the form image data to the server apparatus 200 (step S1108).

[0109] When the input receiving unit 221 of the server device 200 receives input of form image data via the communication unit 298, the authentication processing unit 222 passes this form image data to the form recognition unit 270 (step S1109). Note that, in the example of Fig. 11A, the form image data is passed to the form recognition unit 270 via the authentication processing unit 222, but this is not limiting. The form image data may be passed to the form recognition unit 270 without passing through the authentication processing unit 222.

[0110] When the form recognition unit 270 of the server device 200 receives the form image data, the job list generation unit 272 registers the job in the job list (step S1110). Next, the form recognition unit 270 stores the form image data in the form image database 230 using the image data storage unit 271 (step S1111).

[0111] Next, the form recognition unit 270 uses the image data acquisition unit 273 to acquire form image data to be recognized from the form image database 230 (step S1112), and executes processing to recognize the form image (step S1113). Details of the processing in step S1113 will be described later.

[0112] Next, the form recognition unit 270 stores the recognition result data generated by the item extraction unit 277 in the recognition result database 260 (step S1114).

[0113] In the document recognition system 100, when the terminal device 400 receives a browser display instruction input from the user via the operation reception unit 431 (step S1115), it sends a browser display request to the server device 200 via the communication unit 433 to display the recognition result data (step S1116).

[0114] Furthermore, the terminal device 400 receives authentication information input from the user through the operation receiving unit 431 (step S1117), and transmits this authentication information to the server device 200 through the communication unit 433 (step S1118).

[0115] The communication unit 298 of the server device 200 receives the authentication request, and the authentication processing unit 222 authenticates the user (step S1119). The communication unit 298 sends the authentication result and a job list to the terminal device 400 (step S1120). The communication unit 433 of the terminal device 400 receives the recognition result and the job list, and the display control unit 432 displays them on the browser. Note that the process shown here is for when user authentication is successful.

[0116] Next, when the terminal device 400 receives a job selection from the job list via the operation reception unit 431 (step S1121), the terminal device 400 sends a viewing request to the server device 200 via the communication unit 433, regarding this selection as an instruction to view the recognition results of the form image (step S1122).

[0117] The communication unit 298 of the server device 200 receives the viewing request, and the display screen generation unit 278 acquires the recognition result data stored in the recognition result database 260 (step S1123) and generates a recognition result confirmation screen. The communication unit 298 transmits screen information of the recognition result confirmation screen to the terminal device 400. The communication unit 433 of the terminal device 400 receives the screen information of the recognition result confirmation screen, and the display control unit 432 displays it on the browser (step S1124). The user can check the recognition result on the displayed recognition result confirmation screen.

[0118] Next, a description will be given with reference to FIG. 11B. Generally, the recognition results using the industry-specific definition information 252 or the general-purpose definition information 253 do not correctly reflect the row and column boundaries of the form. This is also true when the individual company-specific definition information 251 is used. Depending on the content of the form (e.g., a long product name), the row height may change, and the recognition results may not correctly reflect the row boundaries of the form. For this reason, the user operates the terminal device 400 to adjust the row and column positions of the recognition results (S1125). The operation reception unit 431 of the terminal device 400 receives the user operation, and the display control unit 432 displays the row and column adjustment results in response to the user operation (S1126). The screen displayed when adjusting the rows will be described with reference to FIG. 19. Note that in this embodiment, the adjustment results are new row boundaries, but the user can also adjust the column positions in a similar manner. Therefore, the adjustment results include row and column position information of the form.

[0119] In this embodiment, lines are concatenated by the line concatenation process described later, so the user only needs to make adjustments to the extent that the form image data is divided into lines at equal intervals. Also, if the form recognition unit 270 divides the form image data into lines at equal intervals, the user does not need to make any adjustments.

[0120] When the user has finished making the adjustments, he instructs the terminal device 400 to perform form recognition, including row concatenation (S1127). Since form recognition has already been completed in step S1113, form recognition is performed again. The operation reception unit 431 receives the instruction, and the communication unit 433 sends a form recognition request, including the results of the user's adjustments (row and column position information), to the server device 200 (S1128).

[0121] The communication unit 298 of the server device 200 receives the form recognition request, and the form recognition unit 270 performs form recognition including line concatenation (S1129). Details of the process in step S1129 will be described with reference to FIG.

[0122] The communication unit 298 of the server device 200 transmits the recognition result data including the result of concatenating the rows to the terminal device 400 via the form recognition unit (S1130).

[0123] The communication unit 433 of the terminal device 400 receives the recognition result data, and the display control unit 432 displays the recognition result including the line concatenation result (S1131).

[0124] As a result, lines that were recognized as different lines depending on the line height are displayed as connected lines.

[0125] The user instructs the terminal device 400 to input an instruction to register the individual company detailed definition information. When the terminal device 400 receives the input of the instruction to register the individual company detailed definition information through the operation receiving unit 431 (step S1132), the terminal device 400 transmits a request to register the individual company detailed definition information to the server device 200 through the communication unit 433 (step S1133).

[0126] When the communication unit 298 of the server device 200 receives this registration request, the display screen generation unit 278 generates a registration screen for the individual company detailed definition information. The communication unit 298 transmits screen information for the registration screen for the individual company detailed definition information to the terminal device 400. The communication unit 433 of the terminal device 400 receives the screen information for the registration screen for the individual company detailed definition information, and the display control unit 432 displays it on the browser (step S1134).

[0127] Next, when the operation reception unit 431 of the terminal device 400 receives input of individual company detail definition information (step S1135), the communication unit 433 transmits the input information to the server device 200 (step S1136). The communication unit 298 of the server device 200 receives the input information, and the definition generation unit 293 generates individual company detail definition information based on the information transmitted from the terminal device 400 (step S1137), stores the information in the detail definition database 250 (step S1138), and terminates the process.

[0128] <Job List> Next, a job list generated by the job list generating unit 272 of this embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of a job list.

[0129] When the server device 200 of this embodiment receives form image data from the image forming device 300, the job list generating unit 272 stores the form image data and the job ID as a job list in association with each other.

[0130] The job list 101 of this embodiment has information items such as a job ID, a billing source, a form image file path, a recognition result file path, a status, and a user ID.

[0131] The value of the item "Job ID" is identification information for identifying a job. In other words, the value of the item "Job ID" is identification information for identifying the form image data received from the image forming apparatus 300.

[0132] The item "biller" is the name of the company that issued the form, etc. Since the biller is unknown when the job is registered, it may be set after the form is recognized.

[0133] The value of the item "Image file path" is information indicating the location where the form image data is stored. The value of the item "Recognition result file path" is information indicating the location where the recognition result data resulting from recognizing the form image is stored.

[0134] The value of the "status" item indicates the progress of recognition of the form image. In other words, the value of the "status" item indicates the state of the job. In this embodiment, the value of the "status" item is exemplified as four values: unprocessed state, draft saved state, confirmed state, and externally output state, but is not limited to these. Each state is as follows:

[0135] The unprocessed state (state 1) is the state immediately after the recognition result data has been acquired by form recognition, and the recognition result has not yet been confirmed.

[0136] The draft saved state (state 2) is a state in which the recognition result data is still being checked, and the recognition result data has not yet been finalized.

[0137] The confirmed state (state 3) is a state in which the recognition result data has been confirmed and stored in the recognition result database 260 as confirmed information.

[0138] The external output completed state (state 4) is a state in which the output of the recognition result data to an accounting system or the like linked with the document recognition system 100 is completed.

[0139] The value of the item "user ID" is identification information for identifying the user who has logged in to the document recognition system 100.

[0140] When the server device 200 of this embodiment receives form image data from the image forming device 300, the job list generation unit 272 assigns a job ID to the form image data, associates it with the user ID acquired as authentication information, and adds a record to the job list 101. Then, when the form image data is stored in the form image database 230, the job list generation unit 272 adds information indicating the storage location as the value of the item "image file path."

[0141] Furthermore, when the recognition result data is stored in the recognition result database 260, the job list generation unit 272 adds the issuer to the item "billing source", adds information indicating the storage destination as the value of the item "recognition result file path", and sets the value of the item "status" to "confirmed".

[0142] In this embodiment, the terminal device 400 may display, for example, a job list generated by the job list generation unit 272. In this case, the user can check the progress of recognition of the form image via the job list screen displayed based on the job list.

[0143] <Recognition of forms> Next, the processing of the form recognition unit 270 of this embodiment will be described with reference to Fig. 13. Fig. 13 is a first flowchart illustrating the processing of the form recognition unit 270. Fig. 13 shows details of the processing of step S1113 in Fig. 11A. Fig. 13 also shows the processing when multiple pieces of form image data are received from the image forming device 300.

[0144] The image data acquisition unit 273 acquires form image data for which recognition result data of the form has not been generated from the form image database 230 (S1300).

[0145] The preprocessing unit 274 performs processing to recognize rows and columns (S1301) for the form image data acquired by the image data acquisition unit 273. The recognition here may be rough, based on horizontal and vertical lines, areas of different colors, etc.

[0146] The preprocessing unit 274 performs preprocessing for form recognition (step S1302) on the form image data acquired by the image data acquisition unit 273. Specifically, the preprocessing includes correction processes such as binarization, noise removal, and top-to-bottom alignment of the form image orientation.

[0147] Next, the form recognition unit 270 cuts out characters from the form image in an area where the characters are formed (hereinafter, character field) and performs character recognition (step S1303) using the character recognition unit 275. At this time, the character recognition unit 275 also acquires coordinates indicating the position of the cut-out characters.

[0148] Next, the form recognition unit 270 generates and stores reading result data based on the positional relationship of each character string recognized by the character recognition unit 275 (step S1304). Specifically, the form recognition unit 270 recognizes characters whose mutual distance is within a preset threshold as one character string based on the coordinates of each recognized character, and generates reading result data by associating the character string with information such as coordinates indicating the position of the area where the character string is formed.

[0149] Next, the document recognition unit 270 uses the definition identification unit 276 to refer to information identifying the biller included in the read result data (hereinafter, biller information) and the business type determination database 240 to identify the business type of the biller (step S1305).

[0150] Next, the document recognition unit 270 determines, via the definition specification unit 276, whether or not the individual company detail definition information 251 corresponding to the billing source information exists in the detail definition database 250 (step S1306).

[0151] In step S1306, if the individual company detailed definition information 251 exists, the document recognition unit 270 causes the item extraction unit 277 to extract detailed information based on the individual company detailed definition information 251 (step S1307), and proceeds to step S1311, which will be described later. The processing of step S1307 will be described later in detail.

[0152] If the individual company detailed definition information 251 does not exist in step S1306, the definition identification unit 276 determines whether the detailed definition database 250 contains industry detailed definition information 252 corresponding to the industry identified in step S1305 (step S1308).

[0153] In step S1307, if the business type detailed definition information 252 exists, the document recognition unit 270 causes the item extraction unit 277 to extract detailed information based on the business type detailed definition information 252 (step S1309), and proceeds to step S1311, which will be described later. The processing of step S1309 will be described later in detail.

[0154] If the business type detail definition information 252 does not exist in step S1308, the document recognition unit 270 causes the item extraction unit 277 to extract detail information based on the general-purpose detail definition information 253 (step S1310), and proceeds to step S1311, which will be described later in detail. The processing of step S1310 will be described later in detail.

[0155] The form recognition unit 270 determines whether or not the extraction of the detailed information has been successful (step S1311). Specifically, the form recognition unit 270 determines whether or not the detailed information extracted in any of the processes in steps S1307, S1309, and S1310 exists.

[0156] If the extraction of the statement information fails in step S1311, the document recognition unit 270 notifies the terminal device 400 or the image forming device 300 of an error (step S1312), and the process ends.

[0157] If the extraction of the detail information is successful in step S1311, the form recognition unit 270 causes the item extraction unit 277 to generate recognition result data by merging the extracted detail information, and stores the data in the recognition result database 260 (step S1313).

[0158] Next, the form recognition unit 270 causes the job list generation unit 272 to register the storage destination of the recognition result data in the job list 101 (step S1314).

[0159] Next, the form recognition unit 270 creates a recognition result confirmation screen using the recognition result data by the display screen generation unit 278. The communication unit 433 of the terminal device 400 receives screen information for the recognition result confirmation screen, and the display control unit 432 displays the recognition result confirmation screen (step S1315), thereby terminating the process.

[0160] 13, the process up to the display of the recognition result data has been described as a series of operations, but the present invention is not limited to this. The form recognition unit 270 may display the recognition result data after receiving a request to view the recognition result data from the terminal device 400.

[0161] <Extraction of detailed information> Next, extraction of detail information by the item extraction unit 277 of this embodiment will be described with reference to Figures 14 and 15. Figure 14 is a second flowchart illustrating the processing of the form recognition unit 270. Figure 15 shows details of the processing of step S1307 in Figure 13.

[0162] In the form recognition unit 270 of this embodiment, when the read result data is stored, the item extraction unit 277 acquires the coordinate values ​​of each character (step S1401). Next, the item extraction unit 277 acquires the extracted character string (extracted in step S1304) and its coordinate values ​​(step S1402).

[0163] Next, the item extraction unit 277 searches for the detailed item name included in the individual company detailed definition information 251 using each acquired character string (step S1403), and determines whether or not a search result exists (step S1404). Specifically, the item extraction unit 277 determines whether or not the individual company detailed definition information 251 contains a detailed item name or keyword that matches the character string.

[0164] If there is no matching detailed item name or keyword in step S1404, the form recognition unit 270 terminates the process. If there is a matching detailed item name or keyword in step S1404, the form recognition unit 270 extracts item values ​​from the vertical column below the detailed item name or keyword that matches the search (step S1405), and terminates the process.

[0165] The process of step S1309 in FIG. 13 is the same as the process of FIG. 14 except that the detailed definition information to be referenced in step S1403 in FIG. 14 is the industry detailed definition information 252, and therefore a description thereof will be omitted.

[0166] Figure 15 is a third flowchart illustrating the processing of the document recognition unit 270. Figure 15 shows details of the processing of step S1310 in Figure 13. In other words, Figure 15 shows the processing of extracting detail information by referencing general detail definition information 253 when neither individual company detail definition information 251 nor industry detail definition information 252 corresponding to the billing source exists.

[0167] The processing in steps S1501 and S1502 in FIG. 15 is similar to the processing in steps S1401 and S1402 in FIG. 14, and therefore a description thereof will be omitted.

[0168] When the item extraction unit 277 acquires the character string and the coordinate value, it determines whether or not vertical rules (vertical lines) exist in the form image data (step S1503). If no vertical rules exist in step S1503, the item extraction unit 277 proceeds to step S1510, which will be described later. Note that well-known methods such as the Hough transform can be used to detect straight lines (vertical lines).

[0169] In step S1503, if vertical lines are present, the item extraction unit 277 extracts the coordinate values ​​of each cell of the vertical lines (step S1504). At this time, the item extraction unit 277 may assign identification information (ID) to each cell to identify it. The identification information may also be assigned, for example, as a column ID that identifies a column of a table and a row ID that identifies a row of the table. By assigning a column ID and a row ID in this way, a cell can be identified by the combination of the column ID and the row ID.

[0170] Next, the item extraction unit 277 identifies the character strings contained in each cell of the table, and stores them in association with the coordinate values ​​of the cells (step S1505).

[0171] Next, the item extraction unit 277 identifies the character strings (column headings) contained in each cell in the first row of the table (step S1506).

[0172] Next, the item extraction unit 277 searches the general-purpose detail definition information 253 for the character string (column heading) included in each cell of the first row (step S1507).

[0173] Next, the item extraction unit 277 determines whether or not there is a detail item name or keyword in the general-purpose detail definition information 253 that matches the character string (column header) included in each cell in the first row (step S1508).

[0174] If the determination in step S1508 is No, the item extraction section 277 ends the process.

[0175] If the determination in step S1508 is Yes, the item extraction unit 277 extracts the item values ​​of the detail items from the column containing the character string included in each cell of the first row (step S1509), and ends the process.

[0176] If no vertical lines are present in step S1503, the form recognition unit 270 identifies the area of ​​the detail portion in the form image (step S1510). Specifically, the form recognition unit 270 may identify, for example, an area where character strings are arranged vertically as the area of ​​the detail portion.

[0177] Next, the form recognition unit 270 uses the item extraction unit 277 to identify the character string on the first line in the area identified in the details section, and stores the identified character string in association with the coordinate value (step S1511). The character string on the first line in the identified area is the character string located at the top of the identified area.

[0178] Next, the item extraction unit 277 searches the general-purpose detail definition information 253 for the character string identified in step S1511 (step S1512). Next, the item extraction unit 277 determines whether or not the general-purpose detail definition information 253 contains a detail item name or keyword that matches the identified character string (step S1513).

[0179] If the determination in step S1513 is No, the document recognition unit 270 ends the process.

[0180] If the determination in step S1513 is Yes, the item extraction unit 277 obtains the width of the area circumscribing the character string from the coordinate values ​​of the character string identified in step S1511, extracts the character strings lined up vertically within this width as the item values ​​of the detail items (step S1514), and terminates the processing.

[0181] 15, if vertical lines are present in the form image, the coordinate values ​​of the cells are extracted, but horizontal lines may be used instead of vertical lines. In this case, the form recognition unit 270 determines whether horizontal lines are present in the form image, and if horizontal lines are present, identifies the character strings located between the horizontal lines and the coordinates of the character strings.

[0182] Then, if the general-purpose detail definition information 253 contains a detail item name or keyword that matches the leftmost string of characters between the horizontal lines, the document recognition unit 270 will use the second or subsequent string of characters between the horizontal lines as the item value of the detail item.

[0183] <Extracting detailed information using forms> Next, the processing of the document recognition unit 270 will be further described with reference to Figures 16 to 18. Figure 16 is a first diagram for explaining extraction of detail information.

[0184] Figure 16 is the first diagram illustrating form image data. The invoice 161 shown in Figure 16 includes billing source information 161a that identifies the billing source, and a detail section 161b in which data in the same format is repeatedly printed vertically for each itemized item. In the invoice 161 shown in Figure 16, the billing source information 161a is the name of the billing source, and the detail section 161b is written in a table format.

[0185] In this embodiment, the invoice 161 is read by the image forming device 300 to generate form image data, and the form recognition unit 270 executes processing on the form image data to generate recognition result data.

[0186] 17 is a second diagram for explaining the extraction of detail information. Information 171 shown in FIG. 17 is the result of character recognition performed by the character recognition unit 275 on the form image data showing the invoice 161.

[0187] Information 171 shows an example in which a character field is cut out from the details portion 161b of the invoice 161, and extracted from the results of recognizing the characters in the character field and the coordinates indicating the positions of the characters.

[0188] The character recognition unit 275 of this embodiment recognizes a character string (character string) by connecting adjacent characters from this information 171. Specifically, the character recognition unit 275 may recognize a combination of characters in which the distance between characters is within a predetermined number of pixels as one word.

[0189] 17 is information indicating the character string recognized by the character recognition unit 275 and the area of ​​the character string, and is data indicating the result of reading the form image by the character recognition unit 275.

[0190] Furthermore, the character recognition unit 275 recognizes an area separated by vertical and horizontal lines from the form image data as one cell, and assigns identification information (cell ID) for identifying the cell to each cell.

[0191] Information 173 shown in FIG. 17 is information indicating the association between the area of ​​each cell recognized from the detail portion 161b and the cell ID.

[0192] In FIG. 17, for example, the area in which the column heading "product name" is recognized is specified by the height and width in the information 172, with the position indicated by the X and Y coordinates as the reference.

[0193] Also, it can be seen that the area in which the column heading "product name" is recognized in information 173 is included in cell ID "2." Therefore, the column heading "product name" is associated with cell ID "2" as a column heading included in detail portion 161b.

[0194] In the information 173, a cell ID is associated with an area where the column heading "product name" is recognized, but this is not limiting. A cell ID may be identified by a row ID that identifies a column in which row headings are arranged vertically and a column ID that identifies a column in which column headings are arranged horizontally.

[0195] In this way, the form recognition unit 270 of this embodiment associates character strings with cells from the form image data.

[0196] 18 is a third diagram showing the detailed information of the detailed portion 161b generated based on the reading result data corresponding to the invoice 161. Therefore, the detailed information 174 shown in Fig. 18 is a part of the recognition result data of the entire document image data.

[0197] 18 is generated by the item extraction unit 277, and includes information items such as a column ID, a column header, a column coordinate, a row ID, an item value, and a detailed item name.

[0198] The value of the item "Column ID" is identification information that identifies the column included in the details portion 161b (table), and the value of the item "Column Heading" is the column heading in each cell of the top row of the table. The value of the item "Column Coordinates" indicates the coordinates of the column. In this embodiment, the column coordinates are the coordinate values ​​of the top left point and the bottom right point of the detail column in the form image.

[0199] The value of the item "row ID" is identification information that identifies a row, and the value of the item "item value" indicates the item value corresponding to a detailed item.

[0200] The value of the item "detailed item name" indicates to which item the corresponding detailed item name and the coordinates indicating the position of the item name are associated in the individual company detailed definition information 251. The coordinates indicating the position of the item name are indicated by, for example, a combination of a column coordinate and a row ID.

[0201] In the detail information 174, the column heading "voucher date" is associated with the value "date" of the item "detailed item name." The column heading "voucher date" is also associated with the column coordinates and row ID.

[0202] When the user decides to update the individual company detail definition information 251 and performs that operation, the following are registered in the "Date" of the individual company detail definition information 251 based on the column coordinates of "Invoice Date" and the row ID: keyword: "Invoice Date", keyword coordinates (one record of the column heading of information 172 for the product name), cell coordinates (one record of the cell for the product name in information 173), and column coordinates (cell coordinates x coordinates for the number of rows).

[0203] 18, the detailed information 174 includes column coordinates, column IDs, and row IDs as information items, but is not limited to this. The detailed information 154 only needs to include detailed item names and item values ​​as information items, and other items may not be included in the detailed information 174.

[0204] Furthermore, in this embodiment, when the detailed information 174 is displayed on the recognition result confirmation screen, the display screen generation unit 278 only needs to display the detailed item names and item values.

[0205] <Example of recognition results> Next, a display example of the recognition result of form image data will be described with reference to Fig. 19. Fig. 19 is a diagram showing a recognition result confirmation screen 501. The recognition result confirmation screen 501 is displayed on the terminal device 400 in step S1125 of Fig. 11B, for example.

[0206] The recognition result confirmation screen 501 includes a display field 502, a display field 503, and display components 508 and 509. A form image based on form image data is displayed in the display field 502. In the example of Fig. 19, the form is an invoice, and an area 504 indicating the details section in the form image is included.

[0207] A display field 503 displays the recognition result data obtained by performing form recognition on the form image data displayed in the display field 502 by the form recognition unit 270 .

[0208] Display column 503 includes display columns 505, 506, and 507. Display column 505 displays the value of the item "status" in job list 101. Display column 506 displays billing source information. The billing source information is part of the recognition result data.

[0209] Display field 507 displays the detailed information. Specifically, display field 507 displays a list of the values ​​of the detailed items included in the detailed information. Display field 507 repeatedly displays the detailed items of each extracted row. Although two rows are displayed in FIG. 19, the detailed items of other rows can be displayed by scrolling, etc. Display field 507 includes display fields 511, 513, 514, 515, and 516.

[0210] Display field 511 displays the value of the detail item with the detail item name "item name" displayed in area 504 showing the detail details in the form image. Display field 513 displays the value of the detail item with the detail item name "amount" displayed in area 504.

[0211] Display field 514 displays the value of the detail item with the detail item name "Unit Price" displayed in area 504. Display field 515 displays the value of the detail item with the detail item name "Quantity" displayed in area 504. Display field 516 displays the value of the detail item with the detail item name "Slip No." displayed in area 504.

[0212] In this embodiment, the terminal device 400 extracts item values ​​from items corresponding to the detailed item names in the detailed section of the form image, and displays them in the display fields 511, 513, 514, 515, and 516. In other words, the items corresponding to these detailed item names are the items to be extracted (extraction target items). The extraction target items may be fixed items, or may be changeable by the user.

[0213] The details extraction button 510 is a button that the user uses to request form recognition in step S1127 of FIG. 11B in order to correctly recognize line breaks in a form that includes lines of different heights.

[0214] The display component 508 is a display component for changing the status value of the job list 101 from an unprocessed state to a draft saved state. When the display component 508 is operated, the display field 505 changes from "unprocessed" to "draft saved."

[0215] The display component 509 is a display component for setting the status value of the job list 101 to the confirmed state. In the example of Fig. 19, the recognition result data was successfully extracted from the form image data, so the recognition result data was not corrected. Therefore, in the example of Fig. 19, when the display component 509 is operated, the status value of the job list 101 is set to the confirmed state, and the individual company description definition information 251 is not generated.

[0216] <Adjusting recognition results> As described above, there are cases where line breaks cannot be correctly recognized even when recognizing a form using the individual company statement definition information 251. In such cases, the user adjusts the lines (although user adjustment is not required) and issues instructions for form recognition.

[0217] FIG. 20 shows an example of a recognition result in which a row has been erroneously recognized. For ease of explanation, FIG. 20 only shows the form image data displayed in area 504. For ease of explanation, row numbers are also shown in the right column. In FIG. 20(a), as an example, every two rows are recognized. The recognition result is presented with a semi-transparent component 540 superimposed on the form image data, which distinguishes between rows and columns using colors or other means to make the boundaries between rows and columns clear. In FIG. 20, the boundaries between rows and columns are indicated by dotted lines. The user can align each row to the same height, add rows, or delete rows by dragging the semi-transparent component 540 with a mouse or other device.

[0218] Here, the user wants to separate the first line and lines 4 to 8 into single lines, and wants to connect the second and third lines. Therefore, the user operates the semi-transparent component 540 to divide the form image data into eight lines at equal intervals, as shown in FIG. 20(b). In this state, the server device 200 can connect the second and third lines. This process corresponds to the process in step S1129 in FIG. 11B.

[0219] Although this embodiment does not specifically describe adjusting column boundaries, the user can also adjust columns. A semi-transparent component 540 displays keywords 521-526 registered in the individual company detail definition information 251, industry detail definition information 252, and general detail definition information 253, in association with each column of the form image data. In FIG. 20(a), the keywords 521-526 are displayed above the column headers. The display position is merely an example. If the correspondence between the keywords 521-526 and the columns is incorrect, the user can move the semi-transparent component 540 for each column and adjust the correspondence between the column headers and the keywords 521-526. For example, the keyword 524 "quantity" corresponds to "quantity" and "unit" in the form image data, but the "unit" can be excluded.

[0220] FIG. 21 is an example of a flowchart illustrating the procedure by which the form recognition unit 270 connects a plurality of lines.

[0221] First, the document recognition unit 270 extracts rows that contain item values ​​in the amount field or the quantity and unit price fields, and sets a label A (S1129-1). The label is row type information that is set for convenience, since rows must be distinguished in later processing. Note that a row to which label A is set is an example of a first row.

[0222] The reason why the amount item is focused on is because the amount is the subject of processing in the ledger, and the reason why the quantity and unit price are focused on is because there are cases where the quantity and unit price are listed even if the amount is not listed on the ledger.

[0223] If an item such as amount is not specified, the amount item is not taken into consideration. Also, although amount is an item set by the user, if the user sets another item, the document recognition unit 270 will take note of that item. Also, if none of amount, quantity, or unit price is set, label A is set for all rows.

[0224] The form recognition unit 270 connects a line with a label D (described later) to a line with a label A. For this reason, the form recognition unit 270 sets a label to each line as follows. Label B is a row in which the words "consumption tax", "subtotal", "total", or "voucher total" (corresponding to excluded terms) are included in any of the items. "Consumption tax", "subtotal", "total", or "voucher total" are terms that are not used as product names. Note that a row with label B set is an example of a third row. Furthermore, excluded terms can be set by the user as appropriate, and the excluded terms in this embodiment are merely an example. Label C is a row that does not have a value in any item. A row with label C set is an example of a fourth row. Label D is another row. Note that a row to which label D is set is an example of a second row.

[0225] Therefore, the document recognition unit 270 changes the label of the line containing any of the characters "consumption tax," "subtotal," "total," and "voucher total" from A to B (S1129-2).

[0226] The form recognition unit 270 assigns a label C to the row in which no value exists in any item (S1129-3).

[0227] The form recognition unit 270 sets the label D to the other lines (S1129-4). Note that the following process will be explained in more detail in FIG.

[0228] Next, the document recognition unit 270 determines whether or not label A is set in the rows above and below (both above and below) the row with label D (S1129-5). If there are rows with label D above and below the row with label D, the determination is made for the row with label D at the farthest end.

[0229] If the determination in step S1129-5 is No, the document recognition unit 270 determines whether or not label A is set in the row above or below the row with label D (S1129-6).

[0230] If the determination in step S1129-6 is Yes, the form recognition unit 270 connects the row with label D to the row with label A (S1129-7). If the determination in step S1129-6 is No, the form recognition unit 270 does not connect label D to any row.

[0231] If the determination in step S1129-5 is Yes, the document recognition unit 270 determines whether the line above the line with label A that first appears in the document (details) is None (the line itself does not exist; outside the range of the document image data), Label B, or Label C (S1129-8).

[0232] If the determination in step S1129-8 is Yes, the document recognition unit 270 connects the row with label A above the row with label D (S1129-9).

[0233] If the determination in step S1129-8 is No, the row above the row with label A is labeled D, but the document recognition unit 270 connects the row with label D to the row with label A below the row with label D (S1129-10).

[0234] The document recognition unit 270 determines whether or not "quantity" and "unit price" are present in multiple lines as a result of linking label A to label D (S1129-11).

[0235] The document recognition unit 270 uses the "quantity" and "unit price" that are closer to the line with label A (S1129-12).

[0236] 22 is a diagram illustrating the concatenation of rows with labels A and D. First, FIG. 22(a) shows a case where rows with label A exist above and below (both above and below) a row with label D. In this case, as explained in the flowchart of FIG. 21, it is determined whether the row above the row with label A adjacent to the upper side of label D has no label, label B, or label C (S1129-8).

[0237] 22(b) shows the case where no label is added. In this case, the form recognition unit 270 connects the label A in the adjacent row above the label D to the row with the label D.

[0238] 22(c) shows a case where the row above the row with label D is label D. In this case, the document recognition unit 270 concatenates label D with labels A above and below it, and the row with label A adjacent to the lower side of label D. If a row with label B is below label A, label B is also concatenated.

[0239] As shown in FIG. 23, there may be a case where label D is adjacent to label D above it, and there are rows with label A above and below it. The number of rows with label D is not limited to two. In this case, it is also determined that label A is set in the rows above and below (both above and below) the row with label D. In this case, it is determined whether the row above the outermost label A is none, label B, or label C (S1129-8).

[0240] Fig. 24 shows labels A to D that have been set to each row of the form image data by the concatenation process of Fig. 21. As shown in Fig. 24, label D has been set to the third row. Because the only row above the row to which label D has been set has label A, the form recognition unit 270 concatenates the second and third rows. Therefore, the form recognition unit 270 can extract "xyz data transfer fee" and "[Your company order No. 123456789]", which are spread across two rows, as a single line of product names.

[0241] Note that the server device 200 does not save the result of linking the row with label A to the row with label D. In other words, the server device 200 does not save the coordinates of the cells and columns in the individual company detailed definition information 251 with the row with label A and the row with label D linked together. This is because the location of the row with label D is determined by the content of the description rather than by the form, and there is little benefit to saving it. Therefore, the individual company detailed definition information 251 registered in step S1138 of FIG. 11B is in a state in which the rows and columns have been adjusted in S1126.

[0242] FIG. 25 is a diagram illustrating the product names linked by the linking process of FIG. 21. FIG. 25 shows the display field 507 of the recognition result confirmation screen 501. FIG. 25(a) shows the display field 507 before the linking process, and FIG. 25(b) shows the display field 507 after the linking process. Before the linking process, the product name in the display field 511 was divided into "xyz data transfer fee" and "[Your company order number: 123456789]", but by the linking process, the product name has been linked to "xyz data transfer fee [Your company order number: 123456789]".

[0243] <Major Effects> As described above, the server device 200 of this embodiment can extract by linking item values ​​that exist in rows with different heights or that are written in multiple rows. [Example]

[0244] In this embodiment, a server device 200 is described that uses virtual ruled lines and logical ruled lines to extract lines while correctly reflecting line breaks in a form, even if there are lines of different heights or item values ​​are entered across multiple lines. In this embodiment, the line breaks in a form are correctly recognized, so the same results as when lines are linked in embodiment 1 are obtained.

[0245] Furthermore, the line break recognition process of this embodiment may be performed completely independently of the first embodiment, or may be performed before or after the linking process of the first embodiment.

[0246] In this embodiment, the hardware configuration diagrams of FIGS. 2 and 3 and the functional block diagrams of FIGS. 9 and 10 described in the above embodiments can be used.

[0247] <Recognizing line breaks using virtual and logical lines> First, an outline of the line break recognition method in this embodiment will be described with reference to Figures 26 and 27. Figure 26 shows the line break recognition method to be compared, and Figure 27 shows the outline of the line break recognition method in this embodiment.

[0248] Assume that the server device 200 saves the individual company detail definition information 251 based on the row and column arrangement (which may or may not be adjusted by the user) for form A in Fig. 26(a). When the server device 200 extracts rows from form B in Fig. 26(b), the first product name will be "XXXXXX" and the second product name will be "XXX", meaning that the extracted rows do not follow the correct row divisions in the form.

[0249] Therefore, as shown in Fig. 27, the server device 200 uses virtual ruled lines and logical ruled lines to extract lines at correct line breaks in the form. Note that the form image data in Fig. 27 is displayed on the recognition result confirmation screen 501.

[0250] First, as shown in form A in FIG. 27(a), the user specifies which column to use as a row separator. For example, the user drags the width of the column header with the mouse. This draws an arrow 530, making the column specified by the user clear. In this example, it is assumed that the "Amount" column has been selected as a row separator. The server device 200 extracts the top of the item value in the Amount column as the top of the row.

[0251] Even when form B in Figure 27(b) is scanned, the server device 200 uses the information that the "Price" column was selected to extract the top of the item value in the Price column as the top of the row. This makes it possible to correctly determine the line breaks whether the product name is in one line or multiple lines. The ruled lines extracted in this way are called "logical ruled lines 531."

[0252] However, if line breaks are determined solely by the method shown in FIGS. 27(a) and 27(b), problems arise in the following forms:

[0253] In form C in FIG. 27(c), the product name corresponding to the first amount "1000" is only "△△△", and "○○○" is not included in the product name.

[0254] Therefore, as shown in Figure 27(d), the user specifies "virtual ruled lines 532" that divide the detail area at equal intervals. In some cases, lines are recognized as being equally spaced even without the user specifying any ruled lines. In Figure 27(d), three virtual ruled lines 532 are displayed, but the logical ruled lines 531 overlap the second and third virtual ruled lines 532 from the top. When the logical ruled lines 531 and the virtual ruled lines 532 overlap, the server device 200 determines that the ruled lines are correct line separators.

[0255] In this way, the line recognition method of this embodiment can extract lines by correctly reflecting the line separators in the form, using information on which columns the user intends to use as line separators, logical ruled lines 531 (an example of a second ruled line), and virtual ruled lines 532 (an example of a first ruled line).

[0256] <Operation or processing of the document recognition system> Next, the operation or processing of the document recognition system 100 will be described with reference to Fig. 28. The processing of Fig. 28 is executed instead of steps S1125 to S1131 of Fig. 11B or following S1131.

[0257] FIG. 28 is a sequence diagram illustrating the operation of the document recognition system.

[0258] The user specifies a detail area by moving, deforming, etc. the semi-transparent component 540 in the area 504 of the recognition result confirmation screen 501 displayed on the terminal device 400 (S2101). The user overlays the semi-transparent component 540 on the form image data so that it covers the entire form image data without excess or deficiency. The operation reception unit 431 receives the operation.

[0259] Next, the user specifies the number of lines at which the form image is to be separated (S2102). The user simply inputs a numerical value. This number of lines may include blank lines. The operation reception unit 431 receives the operation. Alternatively, the user does not have to specify the number of lines. Since the rows and columns have been recognized in step S1301 of FIG. 13, the form recognition unit 270 can use this to draw virtual ruled lines 532.

[0260] Next, the communication unit 433 of the terminal device 400 specifies the number of lines and transmits a display request for the virtual ruled lines 532 to the server device 200 (S2103). The terminal device 400 may display the virtual ruled lines 532 at equal intervals by dividing the height of the form image by the number of lines.

[0261] The communication unit 298 of the server device 200 receives the display request for the virtual ruled lines 532, and the form recognition unit 270 draws the virtual ruled lines 532 (S2104). Drawing means determining the positions of the virtual ruled lines 532 in the form image data. The form recognition unit 270 divides the height of the form image data by the number of lines to draw the equally spaced virtual ruled lines 532. If the user does not specify the number of lines, the form recognition unit 270 draws the equally spaced virtual ruled lines 532 at the height of the shortest line among the lines recognized in step S1301 of FIG. 13.

[0262] The server device 200's 298 transmits the position information of the virtual ruled line 532 to the terminal device 400 (S2105). The communication unit 433 of the terminal device 400 receives the position information of the virtual ruled line 532, and the display control unit 432 displays the virtual ruled line 532 by superimposing it on the form image data in the area 504 (S2106). The user corrects any errors in the virtual ruled line 532 as appropriate.

[0263] Next, the user specifies the column to be used to separate rows using a mouse or the like (S2107). For example, the column to be used to separate rows is the price column. This is because price is often used to separate the product names and other items in a row. The operation acceptance unit 431 accepts the operation. Note that by dragging the width of the column header, at least one of the column ID, column name, or column coordinates is specified.

[0264] The communication unit 433 of the terminal device 400 specifies the column ID, column name, or column coordinates, and transmits a display request for the logical ruled lines 531 to the server device 200 (S2108).

[0265] The communication unit 298 of the server device 200 receives the display request for the logical ruled lines 531, and the form recognition unit 270 draws the logical ruled lines 531 (S2109). That is, the form recognition unit 270 draws the logical ruled lines 531 at the top of the item values ​​in the specified column.

[0266] The server device 200's 298 transmits the position information of the logical ruled line 531 to the terminal device 400 (S2110). The communication unit 433 of the terminal device 400 receives the position information of the logical ruled line 531, and the display control unit 432 displays it superimposed on the form image data in the area 504 (S2111). The display control unit 432 displays the virtual ruled line 532 and the logical ruled line 531 in different colors, so that the user can distinguish between the logical ruled line 531 and the virtual ruled line 532. The user can correct any errors in the logical ruled line 531 as appropriate.

[0267] Next, the user instructs the terminal device 400 to recognize the form (extract item values) based on the virtual ruled lines 532 and logical ruled lines 531 (S2112). The user may also instruct which items' item values ​​should be concatenated. If the user does not instruct, the item names will be concatenated. The user presses the extract details button 510. The operation reception unit 431 receives the operation.

[0268] The communication unit 433 of the terminal device 400 specifies the specified item, virtual ruled line 532, and logical ruled line 531 and transmits an item value extraction request to the server device 200 (S2113).

[0269] The communication unit 298 of the server device 200 receives the item value extraction request, and the form recognition unit 270 determines that the character string between the line separators where the virtual ruled line 532 and the logical ruled line 531 overlap is the item value (S2114).

[0270] The communication unit 298 of the server device 200 transmits the recognition result in which the items and item values ​​are associated with each other to the terminal device 400 (S2115). The communication unit 433 of the terminal device 400 receives the recognition result, and the display control unit 432 displays it in the detailed information of the display field 507 (S2116).

[0271] Fig. 29 is a diagram illustrating the product names linked by the linking process of Fig. 28. Note that Fig. 29 mainly explains the differences from Fig. 25. It shows the display field 507 of the recognition result confirmation screen 501. Fig. 29(a) shows the display field 507 based on line breaks determined only by logical ruled lines 531, and Fig. 29(b) shows the display field 507 based on line breaks determined only by logical ruled lines 531 and virtual ruled lines 532. With only logical ruled lines 531, the product name in display field 511 is divided into "XXX" and "△△△", but with logical ruled lines 531 and virtual ruled lines 532, the product name has been linked to "XXX△△△".

[0272] In this way, for example, even if the product name spans two or more lines for one line of price, the character strings in the product name columns of two or more lines are concatenated and displayed in display field 511 (where the product name is displayed) on recognition result confirmation screen 501.

[0273] <Multi-column form with another string in the specified "Amount" column> With reference to Figure 30, we will explain a multi-column form that has another character string in the specified item "Amount" column. Figure 30 shows a part of a form in which consumption tax is also displayed in the amount column. In the case of a multi-column form in which other information (e.g., consumption tax) is also entered in the "Amount" column, as in the form in Figure 30, when logical ruled line 531 is drawn at the top of the characters in the "Amount" column, the amount and consumption tax lines are extracted as separate lines.

[0274] Therefore, for multi-column forms, when the user specifies the column to be used for line breaks using a mouse or other tool, they set how many lines (for each skip number) to skip when drawing logical ruled lines 531. For example, if the user right-clicks the arrow 530 specifying the column, a dialog box for setting how many lines to skip is displayed. In the example of Figure 30, skipping one line is sufficient, so the user enters one line.

[0275] Figure 31 shows an example of how logical ruled lines 531 are drawn on a multi-column form when the user specifies that the row separator for logical ruled lines 531 should be the "amount" column, skipping every other row. In Figure 31, the amount (and consumption tax) and the product name are correctly associated and extracted.

[0276] <Major Effects> According to this embodiment, it is possible to extract lines that correctly reflect the line breaks in the form, using information about which columns the user intends to use as line breaks, logical ruled lines 531, and virtual ruled lines 532.

[0277] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0278] For example, in the present embodiment, the image forming apparatus 300 and the server apparatus 200 communicate with each other and the server apparatus 200 performs form recognition, but the image forming apparatus 300 may perform form recognition independently. Also, in the present embodiment, after the server apparatus 200 performs form recognition, the terminal apparatus 400 displays the recognition result confirmation screen 501, but the image forming apparatus 300 may display the recognition result confirmation screen 501.

[0279] Alternatively, instead of the image forming apparatus 300, a scanner device connected to a PC may perform scanning, and the PC may transmit the form image data to the server apparatus 200. In this case, the PC serves as the terminal apparatus 400. In the present embodiment, the item value of the product name is linked, but the user may be able to set which item's item value is linked. Also, the item values ​​of multiple items may be linked.

[0280] In addition, the configuration examples in Fig. 9 and the like are divided according to main functions to make it easier to understand the processing by the server device 200. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the server device 200 can be divided into even more processing units depending on the processing content. Furthermore, it can also be divided so that one processing unit includes even more processes.

[0281] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, server device 200 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0282] Furthermore, the server device 200 can be configured to share the disclosed processing steps, such as those shown in FIG. 11, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the server device 200. Furthermore, the server device 200 may be integrated into a single server device or may be divided into multiple devices.

[0283] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. [Explanation of symbols]

[0284] 100 Document Recognition System 200 Server device 400 Terminal Equipment [Prior art documents] [Patent documents]

[0285] [Patent Document 1] Patent Publication No. 2021-140788

Claims

1. An information processing system that provides a terminal device with a recognition result obtained by recognizing a character string from image data in a table format having rows and columns, From the recognition result, concatenate an item value of a predetermined column in a first row having a value in a column of a preset column header with an item value of the predetermined column in a second row that does not include a preset excluded term and is not a blank row; Displaying the linked item values ​​on the terminal device. An information processing system comprising:

2. The predetermined column has a column heading of a product name, 2. The information processing system according to claim 1, wherein the excluded words are words that are not used as product names.

3. 3. The information processing system according to claim 2, wherein the excluded words are consumption tax, subtotal, total, or slip total.

4. The information processing system described in Claim 3, characterized in that the pre-set column headings are amounts, or quantities and unit prices.

5. 5. The information processing system according to claim 4, wherein, when only one of the upper row or the lower row adjacent to the second row is the first row, the item value of the specified column of the first row adjacent to the second row is concatenated with the item value of the specified column of the second row.

6. The row containing the excluded term is the third row, and the row with no values ​​in all columns is the fourth row. When both the upper row and the lower row adjacent to the second row are the first row, If the row above the first row that first appears in the table-format image data is the third row, the fourth row, or does not exist, 5. The information processing system according to claim 4, wherein the item value of the predetermined column of the first row adjacent to the second row above is concatenated with the item value of the predetermined column of the second row.

7. The row containing the excluded term is the third row, and the row with no values ​​in all columns is the fourth row. When both the upper row and the lower row adjacent to the second row are the first row, If the row above the first row that first appears in the table-format image data is the second row, 5. The information processing system according to claim 4, wherein the item value of the specified column of the first row adjacent to and below the second row is concatenated with the item value of the specified column of the second row.

8. 1. A form recognition method performed by an information processing system that provides a terminal device with a recognition result obtained by recognizing character strings from image data in a table format having rows and columns, comprising: From the recognition result, concatenate an item value of a predetermined column in a first row having a value in a column of a preset column header with an item value of the predetermined column in a second row that does not include a preset excluded term and is not a blank row; Displaying the linked item values ​​on the terminal device. A document recognition method characterized by:

9. The predetermined column has a column heading of a product name, 9. The document recognition method according to claim 8, wherein the excluded words are words that are not used as product names.

10. 10. The document recognition method according to claim 9, wherein the excluded words are consumption tax, subtotal, total, or slip total.

11. A document recognition method as described in Claim 10, characterized in that the pre-set column headings are amounts, or quantities and unit prices.

12. A form recognition method as described in claim 11, characterized in that if only one of the rows above or below the second row adjacent to the first row is the first row, the item value of the specified column of the first row adjacent to the second row is concatenated with the item value of the specified column of the second row.

13. The row containing the excluded term is the third row, and the row with no values ​​in all columns is the fourth row. When both the upper row and the lower row adjacent to the second row are the first row, If the row above the first row that first appears in the table-format image data is the third row, the fourth row, or does not exist, The document recognition method according to claim 11, characterized in that the item value of the specified column of the first row adjacent to the second row above is concatenated with the item value of the specified column of the second row.

14. The row containing the excluded term is the third row, and the row with no values ​​in all columns is the fourth row. When both the upper row and the lower row adjacent to the second row are the first row, If the row above the first row that first appears in the table-format image data is the second row, The document recognition method according to claim 11, characterized in that the item value of the specified column of the first row adjacent to and below the second row is concatenated with the item value of the specified column of the second row.

15. An information processing system that provides a terminal device with a recognition result obtained by recognizing a character string from image data in a table format having rows and columns, From the recognition result, concatenate an item value of a predetermined column in a first row having a value in a column of a preset column header with an item value of the predetermined column in a second row that does not include a preset excluded term and is not a blank row; Displaying the linked item values ​​on the terminal device. A program for executing a process.

16. A form recognition system having an information processing system that recognizes character strings from image data in a table format having rows and columns and provides the recognition results to a terminal device, and a terminal device that displays the recognition results, The information processing system includes: From the recognition result, concatenate an item value of a predetermined column in a first row having a value in a column of a preset column header with an item value of the predetermined column in a second row that does not include a preset excluded term and is not a blank row; Displaying the linked item values ​​on the terminal device; The terminal device A form recognition system comprising a display control unit that displays the recognition results.

Images