Information processing system, information processing method, and program

Abstract

To set an area for extracting an item value of one or more prescribed extraction items with a simple operation, to an image of a tabular format.SOLUTION: An information processing system includes: a display control part configured to cause a terminal device to display a configuration screen for setting an extraction area for extracting each item value of one or more prescribed extraction object items from an image of a tabular format, the configuration screen in which an extraction guide showing an extraction area according to each of the one or more extraction object items is displayed on the image of the tabular format; and an operation acceptance part configured to accept an operation to set a position of the extraction guide in the configuration screen.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program for recognizing an image in a table format. [Background technology]

[0002] Conventionally, a form recognition technology has been known that, in a form that lists combinations of multiple item names and item values, by specifying a combination of multiple related item values ​​in advance, it is possible to extract item values ​​corresponding to item names even if the item names are not written in the form image.

[0003] Conventionally, it is known that when a form is not correctly recognized, the user directly designates an area from which an item value is to be extracted. Summary of the Invention [Problem to be solved by the invention]

[0004] Some form images include images expressed in a table format, such as the details of a form. However, conventionally, extraction of item values ​​of one or more predetermined items to be extracted from an image expressed in a table format has not been taken into consideration. In addition, when setting an area for extracting item values ​​of one or more predetermined items to be extracted from an image expressed in a table format, it is expected that operations such as specifying which area of ​​an item to be extracted is to be set will be complicated.

[0005] The present invention has been made in view of the above circumstances, and has an object to enable a user to set, with a simple operation, an area for extracting item values ​​of one or more predetermined extraction items in a tabular image. [Means for solving the problem]

[0006] The disclosed technology includes a display control unit that causes a terminal device to display a setting screen for setting an extraction area for extracting each item value of one or more predetermined extraction target items from a tabular image, the setting screen including an extraction guide indicating the extraction area corresponding to each of the one or more extraction target items being displayed on the tabular image, and an operation receiving unit that receives an operation to set a position of the extraction guide on the setting screen. The setting screen includes a list of the extraction target items that can be selected and accepted, and when the display control unit accepts a selection of the extraction target item in the list, the display control unit erases the extraction guide corresponding to the extraction target item selected from the setting screen. It is an information processing device. Effect of the Invention

[0007] With a simple operation, a region for extracting item values ​​of one or more predetermined extraction items can be set in a tabular image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a system configuration of a document recognition system. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a server device. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a terminal device. [Figure 4] FIG. 13 is a diagram illustrating an example of an industry determination database. [Diagram 5] FIG. 13 is a diagram showing types of detail definition information stored in a detail definition database. [Figure 6] FIG. 13 is a diagram showing an example of general-purpose detail definition information. [Figure 7] FIG. 13 is a diagram showing an example of business type specification definition information. [Figure 8] FIG. 13 is a diagram illustrating an example of individual company detail definition information. [Figure 9] FIG. 2 is a diagram illustrating functions of a server device. [Figure 10] FIG. 2 is a diagram illustrating functions of a terminal device. [Figure 11] FIG. 2 is a sequence diagram illustrating the operation of the document recognition system. [Figure 12] FIG. 4 is a diagram illustrating an example of a job list. [Figure 13]11 is a first flowchart illustrating a process of a document recognition unit. [Figure 14] 13 is a second flowchart illustrating the process of the document recognition unit. [Figure 15] 13 is a third flowchart illustrating the process of the document recognition unit. [Figure 16] FIG. 11 is a first diagram illustrating extraction of detailed information. [Figure 17] FIG. 11 is a second diagram illustrating extraction of detailed information. [Figure 18] 13 is a first flowchart illustrating a process of a definition registration unit. [Figure 19] 13 is a second flowchart illustrating the process of the definition registration unit. [Figure 20] FIG. 11 is a first diagram showing a display example. [Figure 21] FIG. 13 is a second diagram showing a display example. [Figure 22] FIG. 13 is a third diagram showing a display example. [Figure 23] FIG. 4 is a fourth diagram showing a display example. [Figure 24] FIG. 5 is a fifth diagram showing a display example. [Diagram 25] FIG. 6 is a sixth diagram showing a display example. [Figure 26] FIG. 7 is a seventh diagram showing a display example. [Figure 27] FIG. 8 is an eighth diagram showing a display example. [Figure 28] FIG. 9 is a ninth diagram showing a display example. [Figure 29] FIG. 10 is a tenth diagram showing a display example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment will be described below with reference to the drawings. Fig. 1 is a diagram showing an example of a system configuration of a document recognition system.

[0010] The form recognition system 100 of the present embodiment includes a server device 200, an image forming device 300, and a terminal device 400. In the form recognition system 100 of the present embodiment, the server device 200, the image forming device 300, and the terminal device 400 are connected via a network. The form recognition system 100 of the present embodiment is an example of an information processing system.

[0011] Furthermore, the terminal device 400 of this embodiment is connected to a core system 500, for example, via a network. The core system 500 is a system that performs a specific process using the results of recognizing a form in the form recognition system 100 of this embodiment. Specifically, the core system 500 may be, for example, an accounting system, an estimate creation system, or the like.

[0012] In the document recognition system 100 of this embodiment, the server device 200 extracts items and item values ​​contained in the document from image data showing an image of the document read by the scanner function of the image forming device 300, and converts the items and item values ​​into text data that corresponds to each other.

[0013] In the following description, an image of a form read by the scanner function of the image forming apparatus 300 is referred to as a form image, and image data representing the form image is referred to as form image data.

[0014] In the following description, extracting items and their values ​​included in a form image, and converting the items and their values ​​into text data and associating them with each other is referred to as form recognition.

[0015] The server device 200 of this embodiment includes a storage unit 210 and a form recognition processing unit 220. The storage unit 210 includes a form image database 230, an industry determination database 240, a description definition database 250, and a recognition result database 260.

[0016] The form image database 230 stores form image data. The business type determination database 240 stores information referenced when determining the business type of the issuer of a form. The detail definition database 250 stores detail definition information referenced when performing form recognition by the form recognition processing unit 220. The recognition result database 260 stores recognition result data, which is information indicating the result of form recognition by the form recognition processing unit 220.

[0017] The form recognition processing unit 220 of this embodiment acquires form image data stored in the form image database 230, and refers to the business type determination database 240 to determine the business type of the issuer of the form and to identify the detailed definition information to be referenced. Then, the form recognition processing unit 220 performs form recognition using the detailed definition information identified in the detailed definition database 250, and stores the result in the recognition result database 260. The detailed definition information includes information indicating the item names of detailed items included in the form and the positions where the item names are written.

[0018] Here, the form recognition processing unit 220 of this embodiment refers to the individual company detail definition information, the industry detail definition information, and the general-purpose detail definition information stored in the detail definition database 250 when performing form recognition.

[0019] The general-purpose detail definition information in this embodiment is information that defines words corresponding to detail item names in general forms. The industry detail definition information is information that defines words corresponding to detail item names in forms for each industry. The individual company detail definition information is information that defines words corresponding to detail item names in forms specific to a user who uses the form recognition system 100 and information to be extracted from the form image. Details of the industry determination database 240 and the detail definition database 250 will be described later.

[0020] In this embodiment, by using three types of detail definition information when recognizing a form, in addition to general-purpose forms, the detail parts of the form can be recognized, thereby improving the accuracy of form recognition.

[0021] The details section of a form is explained below. The details section of a form is the section that shows the results of data compilation, and is the section where data of the same format is repeatedly printed vertically for each detail item. A detail item is an item of data that is compiled in the details section. Common detail items include product name, quantity, unit price, amount, etc. Specific examples of the details section will be given later.

[0022] In the document recognition system 100 of this embodiment, the image forming device 300 is a multifunction device having a scanner function. The image forming device 300 is equipped with applications for realizing a copy function, a FAX function, a scanner function, etc., and these functions are realized by selecting an application corresponding to each function.

[0023] The terminal device 400 of this embodiment is used by a user who uses the form recognition system 100. Furthermore, the terminal device 400 may display the result of form recognition by the server device 200.

[0024] In this embodiment, the user is, for example, a company that issues a form. More specifically, the user is, for example, an establishment, company, organization, or the like that has concluded a contract to use the service provided by the form recognition system 100.

[0025] 1, the server device 200 has four databases in the storage unit 210, but this is not limiting. A part of each database may be provided in an external device of the server device 200, or the entire database may be provided in an external device.

[0026] 1, the server device 200 realizes the form recognition processing unit 220, but the present invention is not limited to this. The form recognition processing unit 220 may be realized by a plurality of information processing devices.

[0027] In addition, in the example of Figure 1, the document recognition system 100 includes one image forming device 300 and one terminal device 400, but the number of image forming devices 300 and terminal devices 400 included in the document recognition system 100 may be any number.

[0028] Next, a hardware configuration of the server device 200 of this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the hardware configuration of the server device.

[0029] The server device 200 of this embodiment is constructed by a computer, and as shown in FIG. 2, includes a CPU 201, a ROM 202, a RAM 203, an HD 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 208, a network I / F 209, a data bus B, a keyboard 211, a pointing device 212, a DVD-RW (Digital Versatile Disk Rewritable) drive 214, and a media I / F 216.

[0030] Of these, the CPU 201 controls the overall operation of the server device 200. The ROM 202 stores programs used to drive the CPU 201, such as an IPL. The RAM 203 is used as a work area for the CPU 201. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the CPU 201. The display 206 is a display unit that displays various information such as a cursor, a menu, a window, characters, or an image.

[0031] The external device connection I / F 208 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 209 is an interface for data communication using the communication network N. The data bus B is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 201 shown in FIG. 2.

[0032] The keyboard 211 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 214 controls reading and writing of various data from a DVD-RW 213 as an example of a removable recording medium. Note that the medium is not limited to DVD-RW, and may be a DVD-R, etc. The media I / F 216 controls reading and writing (storing) of data from a recording medium 215 such as a flash memory.

[0033] The server device 200 of this embodiment may be, for example, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a wearable PC, or the like.

[0034] The terminal device 400 of this embodiment is a computer having an arithmetic processing device and a storage device, and may be, for example, a tablet-type terminal device, a smartphone, etc. Fig. 4 is a diagram showing an example of the hardware configuration of the terminal device.

[0035] The terminal device 400 of this embodiment includes a CPU 401 , a ROM 402 , a RAM 403 , an EEPROM 404 , a CMOS sensor 405 , an image sensor I / F 406 , an acceleration / direction sensor 407 , a media I / F 409 , and a GPS receiving unit 411 .

[0036] Of these, the CPU 401 is an arithmetic processing device that controls the overall operation of the terminal device 400. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as an IPL. The RAM 303 is used as a work area for the CPU 301. The EEPROM 404 reads or writes various data, such as smartphone programs, under the control of the CPU 401. The ROM 402, the RAM 403, and the EEPROM 404 are examples of storage devices of the terminal device 400.

[0037] The CMOS (Complementary Metal Oxide Semiconductor) sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 401. Note that instead of a CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may also be used.

[0038] The imaging element I / F 406 is a circuit that controls the driving of the CMOS sensor 305. The acceleration / direction sensor 407 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I / F 409 controls the reading and writing (storage) of data from and to a recording medium 408 such as a flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.

[0039] The terminal device 400 also includes a long-distance communication circuit 412, an antenna 412a of the long-distance communication circuit 412, a microphone 415, a speaker 416, an audio input / output I / F 417, a display 418, an external device connection I / F (Interface) 419, a short-distance communication circuit 420, an antenna 420a of the short-distance communication circuit 420, and a touch panel 421.

[0040] Of these, the long-distance communication circuit 412 is a circuit that communicates with other devices via a communication network. The microphone 415 is a built-in circuit that converts sound into an electrical signal. The speaker 416 is a built-in circuit that converts the electrical signal into physical vibrations to generate sounds such as music and voice. The audio input / output I / F 417 is a circuit that processes the input and output of audio signals between the microphone 415 and the speaker 416 under the control of the CPU 401.

[0041] The display 418 is a type of display means such as a liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, etc. The external device connection I / F 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows a user to operate the terminal device 400 by pressing the display 418. The display 418 is an example of a display unit that the terminal device 400 has.

[0042] Next, the business type determination database 240 and the item definition database 250 included in the server device 200 of this embodiment will be described.

[0043] 4 is a diagram showing an example of the business type determination database 240. The business type determination database 240 of this embodiment is provided for each of various business types.

[0044] In the business type determination database 240, business types are associated with the issuers of documents. In the example of Fig. 3, the business type "retail" is associated with companies A, B, and C. Therefore, it can be seen that companies A, B, and C are organizations engaged in the retail business.

[0045] Next, the detail definition database 250 of this embodiment will be described with reference to Figures 5 to 8. Figure 5 is a diagram showing the types of detail definition information stored in the detail definition database.

[0046] The detail definition database 250 of this embodiment stores individual company detail definition information 251, industry detail definition information 252, and general-purpose detail definition information 253. In this embodiment, the individual company detail definition information 251 is the first detail definition information, the industry detail definition information 252 is the second detail definition information, and the general-purpose detail definition information 253 is the third detail definition information.

[0047] The individual company detailed definition information 251 in this embodiment is detailed definition information that defines the detailed item names and description positions included in a form that is uniquely designed by the user, and is detailed definition information exclusive to the user. Among the three detailed definition information, the individual company detailed definition information 251 is the definition that should be most respected and is the detailed definition information with the highest priority.

[0048] The industry type detailed definition information 252 in this embodiment is detailed definition information that defines detailed item names and description positions included in forms that are generally used in each industry. The priority of the industry type detailed definition information 252 is intermediate among the three detailed definition information.

[0049] The general-purpose detail definition information 253 in this embodiment is detail definition information that defines the detail item names and writing positions included in a highly versatile form that is used by multiple users among forms. The general-purpose detail definition information 253 is the detail definition information with the lowest priority among the three detail definition information.

[0050] The form recognition processing unit 220 of this embodiment refers to these three pieces of detail definition information to recognize the form image data. Therefore, in this embodiment, even if the form image data is for a form including a detail item name that is not defined in the individual company detail definition information 251, this form image data can be recognized as long as it is defined in the industry detail definition information 252 or the general-purpose detail definition information 253.

[0051] In the following description, an invoice will be used as an example of a form. However, the form may be any form that includes a detailed description. For example, the form may be a delivery note, a quotation, a purchase slip, etc.

[0052] If the document is a delivery note, the issuer is the delivery source that issued the delivery note, and if the document is a purchase slip, the issuer is the supplier that issued the purchase slip.

[0053] 6 is a diagram showing an example of general-purpose detail definition information. General-purpose detail definition information 253 defines the item names used in the details of general invoices, and has item names and keywords as information items, with the two being associated with each other.

[0054] The value of the field "Detailed Item Name" indicates the name of the detailed item used in the detailed section of the invoice. The value of the field "Keyword" indicates a keyword (word) that is considered to be synonymous with the detailed item name in the detailed section.

[0055] That is, the general-purpose detail definition information 253 is information that defines detail items used in general forms and keywords to be extracted as detail items from forms.

[0056] In the example of Fig. 6, the detailed item name "Product name" is treated as synonymous with keywords such as "product", "product", "item name", etc. Also, in the example of Fig. 6, the detailed item name "Amount" is treated as synonymous with keywords such as "Amount", "Total amount", "Total", etc.

[0057] Also, the example in Fig. 6 indicates that the line item name "order number" is treated as synonymous with keywords such as "order No.", "slip number", etc. Also, the example in Fig. 6 indicates that the line item name "product code" is treated as synonymous with keywords such as "product No.", "product number", etc.

[0058] 7 is a diagram showing an example of business type detail definition information. Business type detail definition information 252 is information provided for each business type, and defines the item names used in the details of general invoices in the industry indicated by the business type.

[0059] The business type definition information 252 has information for identifying the business type, an item "detailed item name," and an item "keyword," which are associated with each other. In the example of FIG. 7, the business type is indicated as "retail business."

[0060] The value of the "item name" field indicates the name of the line item used in the details of a typical invoice. The value of the "keyword" field indicates the keyword (word) that corresponds to the general line item name in an invoice from a retailer.

[0061] That is, the business type detail definition information 252 is information that defines the business type, detail items used in the form, and keywords to be extracted as detail items from the form.

[0062] The example in Figure 7 shows that the detailed item name "Product Name" on a typical document may be replaced with keywords such as "Item Name" or "Product Name" in the detailed section of an invoice issued by a retailer.

[0063] 8 is a diagram showing an example of individual company statement definition information. Individual company statement definition information 251 of this embodiment is registered for each user. The user here is the issuer of the invoice (billing source).

[0064] In the individual company detail definition information 251, information for identifying the billing source is associated with the item "detailed item name" and the item "extracted information."

[0065] The information for identifying the biller includes the name, telephone number, account number, etc. In the example of Fig. 8, the information for identifying the biller includes the name, telephone number, and account number of the biller, but the information for identifying the biller may be any one of these pieces of information.

[0066] The value of the item "detailed item name" indicates the name of the detailed item used in the detailed portion of a general invoice. In other words, the detailed item name in the individual company detailed definition information 251 indicates the detailed item used in the form.

[0067] The value of the item "Extracted information" includes a keyword to be extracted as a keyword (word) corresponding to a general detail item in an invoice issued by the billing source, and information indicating the position where this keyword is written in the document image showing the invoice.

[0068] That is, the extraction information in the individual company detailed definition information 251 includes keywords extracted as detailed items from the form and information indicating the positions of the keywords in the form. Therefore, the individual company detailed definition information 251 is information that defines detailed items used in the form, keywords extracted as detailed items, and information indicating the positions of the keywords in the form.

[0069] The example in Figure 8 shows that the detailed item name "Product Name" on a typical form may be replaced with a keyword such as "Product Name" in the detailed section of an invoice from Company A.

[0070] In addition, the example in Figure 8 shows that, as information indicating the position of the keyword "product name" in a document image showing an invoice from Company A, keyword coordinates indicating the coordinates of the keyword, cell coordinates indicating the coordinates of the cell in which the keyword "product name" is written, and column coordinates indicating the coordinates of the column in which the keyword "product name" is written are extracted.

[0071] In addition, in a form image showing an invoice, the information indicating the position where a keyword is written may be at least one of the coordinate values ​​of the keyword, the value coordinates of a cell, and the coordinate values ​​of a column.

[0072] Specifically, the coordinate values ​​of a keyword may be the coordinates of two diagonal points in a rectangle circumscribing the keyword, from which the width and height of the rectangle circumscribing the keyword can be determined.

[0073] Next, functions of the server device 200 of this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining functions of the server device.

[0074] In the server device 200 of this embodiment, for example, the processor 26 reads and executes an information processing program stored in the memory device 25, the auxiliary storage device 24, etc., thereby implementing the functions of each unit described below.

[0075] The server device 200 of this embodiment includes a form recognition processing unit 220, a definition registration unit 290, and a communication unit 298. The form recognition processing unit 220 includes an input reception unit 221, an authentication processing unit 222, and a form recognition unit 270.

[0076] The input reception unit 221 receives various inputs to the server device 200. Specifically, the input reception unit 221 receives an input of authentication information for logging in to the form recognition system 100. The authentication information is, for example, a user ID and a password. The input reception unit 221 also receives an input of form image data transmitted from the image forming device 300, for example. The input reception unit 221 may receive an input to the server device 200 via the communication unit 298, or may receive an input made directly to the server device 200.

[0077] The authentication processing unit 222 performs authentication based on the authentication information accepted by the input accepting unit 221. The authentication processing unit 222 may transmit the input authentication information to an authentication server provided outside the server device 200, and obtain the result of authentication by the authentication server. When the user is authenticated, the authentication processing unit 222 may pass the form image data accepted by the input accepting unit 221 to the form recognition unit 270.

[0078] The document recognition unit 270 of this embodiment has an image data storage unit 271, a job list generation unit 272, an image data acquisition unit 273, a preprocessing unit 274, a character recognition unit 275, a definition identification unit 276, an item extraction unit 277, a display screen generation unit 278, and an output unit 279.

[0079] When the input receiving unit 221 receives form image data from the image forming apparatus 300 via the communication unit 298 , the image data storage unit 271 stores the form image data in the form image database 230 .

[0080] When input receiving unit 221 receives input of form image data from image forming apparatus 300 via communication unit 298, job list generating unit 272 registers the job in the job list and manages the job list. Details of the job list will be described later.

[0081] The image data acquisition unit 273 acquires form image data stored in the form image database 230. The preprocessing unit 274 performs preprocessing on the form image data acquired by the image data acquisition unit 273 in order to perform form recognition.

[0082] The character recognition unit 275 extracts character strings and information indicating the positions where the character strings are arranged from the form image data, and stores the extracted character strings and information as read result data. The read result data will be described in detail later.

[0083] The definition specifying unit 276 is an acquisition unit that specifies the detail definition information to be referenced during document recognition based on the character string included in the read result data, and acquires the specified detail definition information from the detail definition database 250 .

[0084] The item extraction unit 277 extracts items and item values ​​included in the form image data based on the read result data, and generates recognition result data by associating the items and item values ​​with each other, and stores the recognition result data in the recognition result database 260 .

[0085] At this time, the item extraction unit 277 of this embodiment refers to the detail definition information identified by the definition identification unit 276, and extracts the detail item names written in the detail section of the form in association with the values ​​of the detail items.

[0086] In the following description, information that associates the detailed item name written in the detailed portion of the form with the detailed item value may be referred to as detailed information. The detailed information in this embodiment is information included in the recognition result data and is a part of the recognition result data.

[0087] Furthermore, the recognition result data in this embodiment includes information (issuer information) that indicates the issuer of the form, which is written on the form, etc. In other words, the recognition result data includes information that is indicated by the character string written on the form.

[0088] The display screen generation unit 278 uses the recognition result data stored in the recognition result database 260 to generate a confirmation screen for the recognition result.

[0089] The output unit 279 outputs the confirmation screen of the recognition result generated by the display screen generation unit 278 via the communication unit 298. In other words, the output unit 279 causes the confirmation screen of the recognition result generated by the display screen generation unit 278 to be displayed on the web browser of the terminal device 400 via the communication unit 298.

[0090] The definition registration unit 290 of this embodiment includes an operation reception unit 291 , a display control unit 292 , a definition generation unit 293 , and a definition update unit 294 .

[0091] The operation acceptance unit 291 accepts an operation input to a screen displayed on the web browser of the terminal device 400 via the communication unit 298. More specifically, in this embodiment, the communication unit 298 receives information indicating an operation from the terminal device 400, and the operation acceptance unit 291 accepts input of the information received by the communication unit 298.

[0092] The display control unit 292 (first display control unit) instructs the terminal device 400 to display a screen via the communication unit 298. Specifically, the display control unit 292 causes the terminal device 400 to display a screen for generating individual company detailed definition information. The display control unit 292 also generates an XML format file for displaying the screen, and causes the terminal device 400 to display an input screen on the web browser of the terminal device 400 via the communication unit 298.

[0093] In this embodiment, the display control unit 292 displays a screen for generating individual company detailed definition information on the terminal device 400, but is not limited to this. The device on which the display control unit 292 displays the screen may be the server device 200 or a device other than the terminal device 400, and may be any display device having a display function.

[0094] The definition generating unit 293 generates individual company detail definition information based on information inputted from the terminal device 400 to the input screen, and stores the information in the detail definition database 250 .

[0095] The definition update unit 294 updates the individual company detailed definition information by reflecting the modifications to the individual company detailed definition information already registered in the detailed definition database 250.

[0096] The communication unit 298 of the present embodiment is a function realized by, for example, the network I / F 209 or the like, and transmits and receives data between the server device 200 and an external device. Specifically, for example, the communication unit 298 transmits a notification to the terminal device 400 instructing it to display a screen in response to control by the display control unit 292.

[0097] Next, functions of the terminal device 400 of this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining functions of the terminal device.

[0098] The functions of each unit, which will be described later, are realized by a CPU 401 included in the terminal device 400 reading and executing a program stored in a ROM 402 or the like.

[0099] The terminal device 400 of this embodiment includes an input receiving unit 431 , a display control unit 432 , and a communication unit 433 .

[0100] The input receiving unit 431 receives an input for a screen displayed on the web browser of the terminal device 400 by the display control unit 432 .

[0101] The display control unit 432 (second display control unit) controls the display of the display 418 of the terminal device 400. Specifically, the display control unit 432 causes the display 418 (display unit) to display a screen in accordance with an instruction from the server device 200.

[0102] The communication unit 433 is a function realized by the long-distance communication circuit 412, the antenna 412a of the long-distance communication circuit 412, etc., and transmits and receives data between the terminal device 400 and other devices.

[0103] The operation of the form recognition system 100 of this embodiment will be described below with reference to Fig. 11. Fig. 11 is a sequence diagram illustrating the operation of the form recognition system.

[0104] In the form recognition system 100, when the image forming apparatus 300 receives a request from a user to start an application for performing form recognition (step S1101), the image forming apparatus 300 starts the application (step S1102).

[0105] Next, when the image forming apparatus 300 receives authentication information input from the user (step S1103), the image forming apparatus 300 transmits the authentication information to the server apparatus 200 and makes an authentication request (step S1104).

[0106] Upon receiving the authentication request, the server device 200 performs authentication using the authentication processing unit 222, and notifies the image forming device 300 of the result (step S1105). Note that the operation shown here is that in which the user is authenticated.

[0107] Next, the image forming apparatus 300 receives an instruction to scan the form (step S1106), scans the form, and obtains form image data (step S1107). Next, the image forming apparatus 300 transmits the form image data to the server apparatus 200 (step S1108).

[0108] In the server device 200, when the input receiving unit 221 receives the input of the form image data, the authentication processing unit 222 passes this form image data to the form recognition unit 270 (step S1109). In the example of Fig. 9, the form image data is passed to the form recognition unit 270 via the authentication processing unit 222, but this is not limiting. The form image data may be passed to the form recognition unit 270 without passing through the authentication processing unit 222.

[0109] When the form recognition unit 270 of the server device 200 receives the form image data, the job list generation unit 272 registers the job in the job list (step S1110). Next, the form recognition unit 270 stores the form image data in the form image database 230 by the image data storage unit 271 (step S1111).

[0110] Next, the form recognition unit 270 acquires form image data to be recognized from the form image database 230 by the image data acquisition unit 273 (step S1112), and executes a process of recognizing the form image (step S1113). The process of step S1112 will be described in detail later.

[0111] Next, the form recognition unit 270 stores the recognition result data generated by the item extraction unit 277 in the recognition result database 260 (step S1114).

[0112] In the document recognition system 100, when the terminal device 400 receives a browser display instruction input from a user via the input receiving unit 431 (step S1115), the terminal device 400 makes a browser display request to the server device 200 via the communication unit 433 to display the recognition result data (step S1116).

[0113] Furthermore, the terminal device 400 accepts authentication information input from the user via the input accepting unit 431 (step S1117), and transmits this authentication information to the server device 200 via the communication unit 433 (step S1118).

[0114] The server device 200 authenticates the user through the authentication processing unit 222 (step S1119), notifies the result to the terminal device 400, and causes the terminal device 400 to display a list of the job lists (step S1120). Note that the process shown here is for when the user is authenticated.

[0115] Next, when the terminal device 400 receives a job selection from the job list via the input receiving unit 431 (step S1121), the terminal device 400 sends a viewing request to the server device 200 via the communication unit 433, regarding this selection as an instruction to view the recognition results of the form image (step S1122).

[0116] In response to the viewing request, the server device 200 retrieves the recognition result data stored in the recognition result database 260 (step S1123), generates a recognition result confirmation screen and displays it on the browser of the terminal device 400 (step S1124), and allows the user to confirm the recognition result on the displayed recognition result confirmation screen.

[0117] Next, when the terminal device 400 receives an input of an instruction to register individual company detail definition information through the input receiving unit 431 (step S1125), the terminal device 400 transmits a request to register individual company detail definition information to the server device 200 through the communication unit 433 (step S1126).

[0118] When the server device 200 accepts this registration request, the display control unit 292 of the definition registration unit 290 causes the terminal device 400 to display a registration screen for the individual company detailed definition information (step S1127).

[0119] Next, when the terminal device 400 accepts input of the individual company detailed definition information (step S1128), it transmits the input information to the server device 200 (step S1129). The server device 200 generates individual company detailed definition information by the definition generating unit 293 based on the information transmitted from the terminal device 400 (step S1130), stores it in the detailed definition database 250 (step S1131), and ends the process. The process by the definition registration unit 290 will be described in detail later.

[0120] Here, a job list generated by the job list generating unit 272 of this embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of a job list.

[0121] When the server device 200 of this embodiment receives form image data from the image forming device 300, the job list generating unit 272 stores the form image data in association with a job ID as a job list.

[0122] The job list 101 of this embodiment has information items such as a job ID, a user ID, a form image file path, a recognition result file path, and a status.

[0123] The value of the item "job ID" is an identifier for identifying a job. In other words, the value of the item "job ID" is an identifier for identifying the form image data received from the image forming apparatus 300.

[0124] The value of the item “user ID” is an identifier for identifying the user who has logged in to the document recognition system 100 .

[0125] The value of the item "image file path" is information indicating the location where the form image data is stored. The value of the item "recognition result file path" is information indicating the location where the recognition result data of the form image is stored.

[0126] The value of the item "status" indicates the progress of recognition of the form image, in other words, the state of the job.

[0127] In this embodiment, there are four values ​​for the item "status": unprocessed state, draft saved state, confirmed state, and external output state. Each state is as follows.

[0128] Unprocessed state (state 1): This is the state immediately after the recognition result data has been obtained by form recognition, and the recognition results have not yet been confirmed.

[0129] Draft saved state (state 2): The state in which the recognition result data is in the middle of being confirmed, etc., and the recognition result data has not yet been finalized.

[0130] Confirmed state (state 3): The state in which the confirmation of the recognition result data and other operations are completed and the data is stored in the recognition result database 260 as confirmed information.

[0131] External output completed state (state 4): A state in which the output of the recognition result data to an accounting system or the like linked with the document recognition system 100 has been completed.

[0132] In the server device 200 of this embodiment, when it receives form image data from the image forming device 300, the job list generation unit 272 assigns a job ID to the form image data, associates it with the user ID acquired as authentication information, and adds a record to the job list 101. Then, when the form image data is stored in the form image database 230, the job list generation unit 272 adds information indicating the storage destination as the value of the item "image file path."

[0133] Furthermore, when the recognition result data is stored in the recognition result database 260, the job list generation unit 272 adds information indicating the storage destination as the value of the item "recognition result file path" and sets the value of the item "status" to "confirmed."

[0134] In this embodiment, for example, a job list generated by the job list generating unit 272 may be displayed. In this case, the user can check the progress of recognition of the form image via the job list screen displayed based on the job list.

[0135] Next, the processing of the form recognition unit 270 of this embodiment will be described with reference to Fig. 13. Fig. 13 is a first flowchart for explaining the processing of the form recognition unit. Fig. 13 shows details of the processing of step S1113 in Fig. 11. Fig. 13 also shows the processing when multiple form image data are received from the image forming device 300.

[0136] In this embodiment, when the form recognition unit 270 accepts the selection of one job from the job list (step S1301), the preprocessing unit 274 performs preprocessing for form recognition on the form image data acquired by the image data acquisition unit 273 (step S1302). Specifically, the preprocessing is, for example, a correction process for aligning the orientation of the form image.

[0137] Next, the form recognition unit 270 cuts out characters from the form image in an area where the characters are formed (hereinafter, a character field) and performs character recognition (step S1103) using the character recognition unit 275. At this time, the character recognition unit 275 also obtains coordinates indicating the position of the cut-out characters.

[0138] Next, the form recognition unit 270 generates and holds the reading result data based on the positional relationship of each character string recognized by the character recognition unit 275 (step S1304). Specifically, the form recognition unit 270 recognizes characters whose mutual distance is within a preset threshold as one character string based on the coordinates of each recognized character, and generates the reading result data by associating the character string with information such as coordinates indicating the position of the area where the character string is formed.

[0139] Next, the document recognition unit 270, by using the definition specification unit 276, refers to information for specifying the biller included in the read result data (hereinafter, biller information) and the business type determination database 240 to specify the business type of the biller (step S1305).

[0140] Next, the document recognition unit 270 determines, via the definition specification unit 276, whether or not the individual company detail definition information 251 corresponding to the billing source information exists in the detail definition database 250 (step S1306).

[0141] In step S1306, if the individual company detailed definition information 251 exists, the document recognition unit 270 causes the item extraction unit 277 to extract detailed information based on the individual company detailed definition information 251 (step S1307), and proceeds to step S1111, which will be described later. The process of step S1107 will be described later in detail.

[0142] If the individual company detailed definition information 251 does not exist in step S1306, the definition identifying unit 276 determines whether or not the detailed definition database 250 contains business type detailed definition information 252 corresponding to the business type identified in step S1305 (step S1308).

[0143] In step S1307, if the business type detailed definition information 252 exists, the document recognition unit 270 causes the item extraction unit 277 to extract detailed information based on the business type detailed definition information 252 (step S1309), and proceeds to step S1311, which will be described later. The process of step S1309 will be described later in detail.

[0144] In step S1308, if the business type detailed definition information 252 does not exist, the document recognition unit 270 causes the item extraction unit 277 to extract detailed information based on the general detailed definition information 253 (step S1310), and proceeds to step S1311, which will be described later. The process of step S1310 will be described later in detail.

[0145] The document recognition unit 270 determines whether the extraction of the detailed information has been successful (step S1311). Specifically, the document recognition unit 270 determines whether the detailed information extracted in any one of the processes in steps S1307, S1309, and S1310 exists.

[0146] If extraction of the detailed information fails in step S1311, the document recognition unit 270 notifies the terminal device 400 or the image forming device 300 of an error (step S1312), and ends the process.

[0147] In step S1311, if the extraction of the detailed information is successful, the document recognition unit 270 causes the item extraction unit 277 to generate recognition result data by merging the extracted detailed information, and stores the data in the recognition result database 260 (step S1313).

[0148] Next, the document recognition unit 270 causes the job list generation unit 272 to register a storage destination for the recognition result data in the job list 101 (step S1314).

[0149] Next, the document recognition unit 270 causes the display screen generation unit 278 to display on the terminal device 400 a confirmation screen for the recognition result generated using the recognition result data (step S1315), and ends the process.

[0150] 13, the process up to displaying the recognition result data has been described as a series of operations, but the present invention is not limited to this. The form recognition unit 270 may display the recognition result data after receiving a request to view the recognition result data from the terminal device 400.

[0151] Next, the extraction of detailed information by the item extraction unit 277 of this embodiment will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a second flowchart for explaining the processing of the document recognition unit. Fig. 15 shows the details of the processing of step S1307 in Fig. 13.

[0152] In the document recognition unit 270 of this embodiment, when the read result data is held, the item extraction unit 277 extracts the coordinate values ​​of each character (step S1401). Next, the item extraction unit 277 identifies a keyword (character string) from the coordinates of each extracted character, and identifies the coordinate values ​​corresponding to the keyword (step S1402).

[0153] Next, the item extraction unit 277 searches for the detailed item name included in the individual company detailed definition information 251 with the specified keyword (step S1403), and determines whether or not there is a search result (step S1404). Specifically, the item extraction unit 277 determines whether or not there is a detailed item name that matches the specified keyword in the individual company detailed definition information 251.

[0154] In step S1404, if there is no corresponding detailed item name, the document recognition unit 270 ends the process. Also, in step S1404, if there is a corresponding detailed item name, the specified keyword is set as the detailed item name of the detailed part, and the keywords arranged vertically below the specified keyword are extracted as item values ​​of the detailed item name (step S1405), and the process ends.

[0155] The process of step S1309 in FIG. 13 is the same as the process of FIG. 14 except that the detailed definition information to be referenced in step S1403 in FIG. 14 is the business type detailed definition information 252, and therefore a description thereof will be omitted.

[0156] Fig. 15 is a third flowchart for explaining the processing of the document recognition unit. Fig. 15 shows details of the processing of step S1310 in Fig. 13. In other words, Fig. 15 shows the processing for extracting detail information by referring to general-purpose detail definition information 253 when neither individual company detail definition information 251 nor industry detail definition information 252 corresponding to the billing source exists.

[0157] The processes in steps S1501 and S1502 in FIG. 15 are similar to the processes in steps S1401 and S1402 in FIG. 14, and therefore will not be described.

[0158] When the item extraction unit 277 identifies the keyword and the coordinate value, it determines whether or not a table exists in the form image data (step S1503). If a table does not exist in step S1503, the item extraction unit 277 proceeds to step S1510, which will be described later.

[0159] In step S1503, if a table is present, item extraction unit 277 extracts the coordinate values ​​of each cell in the table (step S1504). At this time, item extraction unit 277 may assign identification information (ID) for identifying each cell to each cell. Furthermore, the identification information may be assigned, for example, as a column ID for identifying a column of the table and a row ID for identifying a row of the table. In this way, by assigning a column ID and a row ID, a cell can be identified by a combination of the column ID and the row ID.

[0160] Next, item extraction unit 277 identifies keywords contained in each cell of the table, and stores them in association with the coordinate values ​​of the cells (step S1505).

[0161] Next, the item extraction unit 277 identifies the keywords contained in each cell in the first row (top row) of the table (step S1506). Next, the item extraction unit 277 searches the general-purpose detail definition information 253 for the keywords contained in each cell in the first row (step S1507).

[0162] Next, the item extraction unit 277 determines whether or not there is a detail item name that matches the keyword contained in each cell in the first row in the general-purpose detail definition information 253 (step S1508). If there is no detail item name that matches the keyword contained in each cell in the first row in step S1508, the item extraction unit 277 ends the process.

[0163] In step S1508, if there is a detailed item name in the first row that matches the keyword contained in each cell, item extraction unit 277 treats the keyword contained in each cell in the first row as the detailed item name, extracts the keyword contained in the column that contains this keyword as the item value of the detailed item (step S1509), and terminates the process.

[0164] In step S1503, if a table is not present, the form recognition unit 270 identifies a region of the detail portion in the form image (step S1510). Specifically, the form recognition unit 270 may identify, for example, a region in which keywords are arranged vertically as a region of the detail portion.

[0165] Next, the document recognition unit 270 uses the item extraction unit 277 to identify a keyword in the first line in the area identified in the description section, and stores the identified keyword in association with its coordinate value (step S1511). The keyword in the first line in the identified area is the keyword located at the top of the identified area.

[0166] Next, the item extraction unit 277 searches the general-purpose detail definition information 253 for the keyword identified in step S1511 (step S1512). Next, the item extraction unit 277 determines whether or not a detail item name that matches the identified keyword exists in the general-purpose detail definition information 253 (step S1513).

[0167] If there is no matching detail item name in step S1513, the document recognition unit 270 ends the process.

[0168] In step S1513, if a matching detailed item name exists, the item extraction unit 277 obtains the width of the area circumscribing the keyword from the coordinate values ​​of the keyword identified in step S1511, and extracts the keywords lined up vertically within this width as item values ​​of the detailed item (step S1514), and terminates the process.

[0169] 15, if a table is present in the form image, the coordinate values ​​of the cells are extracted, but vertical lines may be used instead of tables. In this case, the form recognition unit 270 determines whether or not vertical lines are present in the form image, and if vertical lines are present, identifies the keywords located between the vertical lines and the coordinates of the keywords.

[0170] Then, if a detail item name that matches the topmost keyword among the keywords between the vertical lines exists in the general detail definition information 253, the document recognition unit 270 will set the second or subsequent keyword between the vertical lines as the value of the detail item.

[0171] The processing of the document recognition unit 270 will be further described below with reference to Figures 16 and 17. Figure 16 is a first diagram for explaining extraction of detail information.

[0172] Figure 16 is the first diagram for explaining the extraction of detail information. The invoice 161 shown in Figure 16 includes billing source information 161a that identifies the billing source, and a detail section 161b in which data of the same format is repeatedly printed vertically for each detail item. In the invoice 161 shown in Figure 16, the billing source information 161a is the name of the billing source, and the detail section 161b is written in a table format.

[0173] In this embodiment, the invoice 161 is read by the image forming device 300 to generate document image data, and the document recognition unit 270 processes the document image data to generate recognition result data.

[0174] 17A and 17B are diagrams for explaining extraction of detailed information. Information 171 shown in Fig. 17A is the result of character recognition performed by the character recognition unit 275 on the form image data showing the invoice 161.

[0175] Information 171 shows an example in which a character field is cut out from detail portion 161b of invoice 161, and extracted from the results of recognizing characters in the character field and the coordinates indicating the positions of the characters.

[0176] The character recognition unit 275 of this embodiment recognizes a character string by connecting adjacent characters from this information 171. Specifically, the character recognition unit 275 may recognize a combination of characters in which the distance between the characters is within a predetermined number of pixels as one word.

[0177] 17A is information indicating a character string recognized by the character recognition unit 275 and an area of ​​the character string, and is data indicating the result of the character recognition unit 275 reading the form image.

[0178] Furthermore, the character recognition unit 275 recognizes an area divided by vertical and horizontal lines from the form image data as one cell, and assigns identification information (cell ID) for identifying the cell to each cell.

[0179] Information 173 shown in FIG. 17(A) is information indicating the association between the area of ​​each cell recognized from the detail portion 161b and the cell ID.

[0180] In FIG. 17A, for example, the area in which the keyword "product name" is recognized is specified by the height and width in the information 172, with the position indicated by the X coordinate and the Y coordinate as the reference.

[0181] Also, it can be seen from information 173 that the area in which the keyword "product name" was recognized is included in cell ID "2". Therefore, the keyword "product name" is associated with cell ID "2" as a keyword included in detail portion 161b. Note that, in information 173, a cell ID is associated with an area in which the keyword "product name" was recognized, but this is not limiting. The cell ID may be specified by a column ID that specifies the column in which the keywords are arranged vertically and a row ID in which the keywords are arranged horizontally.

[0182] In this manner, the form recognition unit 270 of this embodiment associates keywords with cells from the form image data.

[0183] Information 174 shown in Fig. 17(B) is detailed information of the detailed portion 161b generated based on the reading result data corresponding to the invoice 161. Therefore, the detailed information 174 shown in Fig. 17(B) is a part of the recognition result data of the entire document image data.

[0184] Detailed information 174 shown in FIG. 17(B) is generated by item extraction unit 277, and has information items such as column ID, item name, column coordinates, row ID, item value, and detailed item designation.

[0185] The value of the item "Column ID" is identification information that identifies a column included in the details portion 161b (table), and the value of the item "Detail Item Name" is a keyword in each cell of the top row of the table. The value of the item "Column Coordinates" indicates the coordinates of the column. In this embodiment, the coordinates of a column are defined as the coordinate values ​​of the top left point and the bottom right point of the detail column in the form image.

[0186] The value of the item "row ID" is identification information that identifies a row, and the value of the item "item value" indicates a detailed item and its corresponding item value.

[0187] The value of the item “detailed item designation” indicates whether or not to register the corresponding item name and the coordinates indicating the position of the item name in the individual company detailed definition information 251. The coordinates indicating the position of the item name are indicated by, for example, a combination of a column coordinate and a row ID.

[0188] In the detailed information 174, "product name" which is the value of the item "detailed item specification" is associated with "product name" which is the value of the item "detailed item specification." Also, assume that the coordinates indicating the position of "product name" are the column coordinates.

[0189] In this case, the “Product Name” and the coordinates of the “Product Name” columns are registered in the individual company detailed definition information 251 as the value of the “Product Name” item, which is the value of the “Detailed Item Name” item, and the value of the “Extracted Information” item.

[0190] 17B, ​​the detailed information 174 includes, as information items, column coordinates, column ID, and row ID, but is not limited to this. The detailed information 154 only needs to include, as information items, detailed item names and item values, and other items may not be included in the detailed information 174.

[0191] In this embodiment, when detailed information 174 is displayed on the confirmation screen for the recognition result, only detailed item names and item values ​​need to be displayed.

[0192] The generation of the detailed information 174 by the item extraction unit 277 will be specifically described below.

[0193] First, a case will be described in which the item extraction unit 277 generates the detailed information 174 by referring to the individual company detailed definition information 251. The following description specifically shows the processing in FIG.

[0194] In this case, for example, when the keyword "product name" and the keyword coordinate value are specified in the information 173, the item extraction unit 277 searches the individual company description definition information 251 for the keyword "product name".

[0195] Here, since the keyword "product name" exists in the individual company detailed definition information 251 (see FIG. 8), the item extraction unit 277 extracts the keyword "product name" as a detailed item name. In addition, the item extraction unit 277 extracts keywords arranged under the keyword "product name" obtained from the keyword coordinate value of the keyword "product name" as item values ​​of the detailed items.

[0196] At this time, the item extraction unit 277 extracts coordinates (column ID, column coordinates) indicating the position of the keyword "item name" and associates it with the keyword "item name" which is the item name. The item extraction unit 277 also associates the keyword (e.g., "A") extracted as an item value with the coordinates (column ID, row ID) indicating the position of this keyword (item value).

[0197] Next, a case where the item extraction unit 277 generates the detailed information 174 by referring to the business type detailed definition information 252 will be described.

[0198] In this case as well, when the keyword "product name" and the keyword coordinate value are identified in the information 173, the item extraction unit 277 searches the industry detailed definition information 252 for the keyword "product name". Since the keyword "product name" exists in the industry detailed definition information 252 (see FIG. 7), the item extraction unit 277 extracts the keyword "product name" as a detailed item name. In addition, the item extraction unit 277 extracts the keywords arranged under the keyword "product name" obtained from the keyword coordinate value of the keyword "product name" as item values ​​of the detailed items.

[0199] Here too, the item extraction unit 277 associates the keyword "item name" which is the detailed item name with the coordinates indicating the position of the keyword "item name." In addition, the item extraction unit 277 associates the keyword extracted as the item value with the coordinates indicating the position of this keyword (item value).

[0200] Next, a case will be described where the item extraction unit 277 generates the detail information 174 by referring to the general-purpose detail definition information 253. The following description specifically shows the process of FIG.

[0201] In this case, for example, when the keyword "product name" and the keyword coordinate value are specified in the information 173, the item extraction unit 277 judges whether or not a table exists in the specification portion 161b. Here, since the specification portion 161b is in a table format as shown in FIG. 14, a table exists.

[0202] Therefore, item extraction unit 277 extracts the coordinate values ​​of each cell included in the table showing detail portion 161b from information 173, and stores the coordinate values ​​of each cell in association with the keyword in the cell.

[0203] In the example of FIG. 17, the coordinate values ​​of cell ID "2" and the keyword "product name", the coordinate values ​​of cell ID "7" and the keyword "product price", and so on, are stored in association with each other.

[0204] Next, the item extraction unit 277 identifies keywords included in the top row of the table. Here, the keywords "item name" and "item price" are identified as the keywords included in the top row.

[0205] Next, the item extraction unit 277 searches the general-purpose detail definition information 253 for the specified keyword. Here, the keyword "product name" does not exist in the general-purpose detail definition information 253 (see FIG. 6). Therefore, the item extraction unit 277 does not extract an item value for the keyword "product name".

[0206] In this manner, in this embodiment, the detailed item names are first extracted from the detailed section 161b by referring to the individual company detailed definition information 251. Therefore, according to this embodiment, detailed item names and item values ​​used uniquely by each billing party can be associated and extracted. In other words, according to this embodiment, detailed information can be extracted from the detailed section of the form by referring to the individual company detailed definition information 251.

[0207] Furthermore, in this embodiment, if individual company detailed definition information 251 does not exist, detailed item names and item values ​​are extracted by referring to industry detailed definition information 252 corresponding to the industry of the biller. Therefore, according to this embodiment, even if individual company detailed definition information 251 does not exist, detailed items specific to the industry of the biller can be associated with and extracted from item values. In other words, according to this embodiment, detailed information can be extracted from the detailed information section of a form by referring to industry detailed definition information 252.

[0208] In this embodiment, if the individual company detail definition information 251 and the industry detail definition information 252 do not exist, reference is made to the general-purpose detail definition information 253. Then, in this embodiment, if the detail item name described in the detail part 161b does not exist in the general-purpose detail definition information 253, the detail item name to be newly extracted and information indicating the position where the detail item name is described in the form image can be registered as the individual company detail definition information 251.

[0209] The registration of the individual company detailed definition information 251 by the definition registration unit 290 will be described below with reference to FIG. 18 and FIG.

[0210] 18 is a first flowchart illustrating the process of the definition registration unit 110. FIG 18 shows the details of the process from step S1124 to step S1131 in FIG 9.

[0211] The server device 200 of this embodiment determines whether or not a request for registering the recognition result has been received from the terminal device 400 on a confirmation screen of the recognition result by the form recognition unit 270 (step S1801). Specifically, the server device 200 determines whether or not an instruction has been given to store the recognition result data displayed on the confirmation screen in the recognition result database 260.

[0212] In step S1801, if a request for registering the recognition result has not been accepted, the server device 200 proceeds to step S1805, which will be described later.

[0213] In step S1801, when a request to register the recognition result is received, the server device 200 judges whether the recognition result data has been corrected on the confirmation screen (step S1802). Note that the correction of the recognition result data in this embodiment mainly refers to the correction of the recognition result data of the detail portion.

[0214] If the recognition result data has not been corrected in step S1802, the server device 200 ends the process.

[0215] In step S1802, if the recognition result data has been corrected, the definition registration unit 290 determines whether the individual company detail definition information 251 has been used in the recognition of the form image data by the definition generation unit 293 (step S1803). Specifically, the definition generation unit 293 determines whether the individual company detail definition information 251 associated with the billing source exists.

[0216] In step S1803, if the individual company detailed definition information 251 is not used, the definition registration unit 290 proceeds to step S1806, which will be described later.

[0217] In step S1803, if the individual company detailed definition information 251 is used, the definition update unit 294 updates this individual company detailed definition information 251 stored in the detailed definition database 250 (step S1804), and ends the process.

[0218] In step S1801, if a request for registering the recognition result data is not received, the server device 200 determines whether a request for registering the form definition (individual company detailed definition information 251) is received from the terminal device 400 (step S1805).

[0219] In step S1805, if a request for registering a form definition has not been accepted, the server device 200 returns to step S1801.

[0220] In step S1805, when a request to register a form definition is accepted, the definition registration unit 290 causes the definition generation unit 293 to generate the individual company detailed definition information 251 (step S1806). The process of step S1806 will be described later in detail.

[0221] Next, the server device 200 accepts input of billing source information from the terminal device 400 (step S1807). Next, the definition generating unit 293 associates the input billing source information with the individual company detail definition information 251, registers (stores) it in the detail definition database 250 (step S1808), and ends the process.

[0222] The generation of the individual company detailed definition information 251 by the definition generating unit 293 will be described below with reference to FIG.

[0223] When generating the individual company detail definition information 251, the definition registration unit 290 of this embodiment displays an extraction guide for specifying an area from which to extract detail information (detail item name and detail item value) on the detail portion of the form image displayed on the confirmation screen of the recognition result in a superimposed manner. Then, in this embodiment, when the position of the extraction guide is set in the detail portion of the form image displayed on the confirmation screen, the detail information is extracted from the detail portion based on the extraction guide.

[0224] The extraction guide of this embodiment is an image in which an area showing a specific detailed item name corresponds to an area showing the value of the detailed item shown by the specific detailed item name. In other words, it is an extraction guide image. The extraction guide image includes images of lines forming rectangles arranged two-dimensionally in rows and columns, and text images showing the specific detailed item names associated with each rectangle.

[0225] In this embodiment, the extraction guide image is described as an extraction guide, but is not limited to this. The extraction guide may be generated by drawing lines forming a rectangle arranged two-dimensionally in rows and columns, or a figure composed of multiple highly transparent rectangles on the screen. For example, when the confirmation screen is displayed in a browser, the extraction guide can be generated by causing the browser to draw lines and figures using JavaScript.

[0226] The predetermined item name in this embodiment is a name of an item that is assumed to be included in a general description and is determined in advance. Specifically, the predetermined item name may be selected from the item names included in the general-purpose item definition information 253, for example.

[0227] In this embodiment, an operation is performed on the confirmation screen to associate an area in the extraction guide image indicating a specific detail item name with an area indicating the value of the detail item, and an area in the detail section indicating the detail item name with an area indicating the value of the detail item.

[0228] In other words, in this embodiment, the confirmation screen accepts an operation to set the position of the extraction guide image in the detail portion of the form image. Therefore, the confirmation screen of this embodiment can be said to be an example of a setting screen for setting the position of the extraction guide image.

[0229] When this operation is performed, the definition registration unit 290 sets an area for extracting detailed information based on the extraction guide image.

[0230] Then, the definition registration unit 290 of this embodiment generates individual company detail definition information 251 based on this extraction guide image, and stores it in the detail definition database 250 .

[0231] 19 is a second flowchart illustrating the process of the definition registration unit 1806 in FIG.

[0232] In the definition registration unit 290 of this embodiment, the operation reception unit 291 receives an operation requesting setting of a region of a detailed portion on a confirmation screen of a recognition result displayed on the terminal device 400 (step S1901).

[0233] Next, the definition registration unit 290 causes the display control unit 292 to display an extraction guide image superimposed on the detailed portion of the form image on the confirmation screen (step S1902). The extraction guide image will be described in detail later.

[0234] Next, the definition registration unit 290 determines whether or not an operation has been performed on the extraction guide image displayed on the confirmation screen in the terminal device 400 through the operation reception unit 291 (step S1903). If an operation has been performed on the extraction guide image in step S1903, the definition registration unit 290 proceeds to step S1906, which will be described later.

[0235] In step S1903, if no operation has been performed, the definition registration unit 290 judges whether the display control unit 292 has accepted an operation on a display component that instructs the user to increase or decrease the number of lines in the extraction guide image, which is displayed on the confirmation screen (step S1904). The display component that instructs the user to increase or decrease the number of lines in the extraction guide image is displayed on the confirmation screen together with the extraction guide image. Details of this display component will be described later.

[0236] In step S1904, if an operation to instruct to increase or decrease the number of rows in the extraction guide image has not been received, the definition registration unit 290 proceeds to step S1907, which will be described later.

[0237] In step S1904, when an operation to instruct to increase or decrease the number of rows in the extraction guide image is accepted, the definition registration unit 290 causes the display control unit 292 to change the number of rows in the extraction guide image in response to the operation (step S1905).

[0238] Next, the definition registration unit 290 causes the display control unit 292 to change the display of the extraction guide image in response to the operation accepted in step S1903 (step S1906).

[0239] Next, the definition registration unit 290 determines whether or not an instruction to extract detailed information has been received from the operation reception unit 291 (step S1907). If an instruction to extract has not been received in step S1907, the definition registration unit 290 returns to step S1903.

[0240] In step S1907, if an extraction instruction is received, the document recognition unit 270 causes the item extraction unit 277 to extract the detail item names and detail item values ​​within the area indicated by the extraction guide image, and reflects these in the detail information display column on the confirmation screen (step S1908).

[0241] Next, the definition registration unit 290 generates the individual company detail definition information 251 by the definition generation unit 293 (step S1909) and ends the process. Specifically, the definition generation unit 293 associates the detail item name indicated by the extraction guide image, the keyword extracted as the detail item from the detail part of the form image, and information indicating the position of the keyword on the form, to generate the individual company detail definition information 251.

[0242] A display example of the confirmation screen of the recognition result will be described below with reference to Fig. 20 to Fig. 26. Fig. 20 is a first diagram showing a display example. A screen 501 shown in Fig. 20 is an example of the confirmation screen of the recognition result displayed on the terminal device 400 in step S1124 of Fig. 11, for example.

[0243] The screen 501 includes a display field 502, a display field 503, and display components 508 and 509. A form image indicated by the form image data is displayed in the display field 502. In the example of Fig. 20, the form is an invoice, and an area 504 indicating the details section in the form image is included.

[0244] As shown in Fig. 20, the detail portion in the form image is represented in a table format. In other words, the detail portion in the form image is an example of an image in a table format.

[0245] A display field 503 displays recognition result data resulting from performing form recognition on form image data indicating the form image displayed in the display field 502 by the form recognition unit 270 .

[0246] The display column 503 includes display columns 505, 506, and 507. The display column 505 displays the value of the item "status" of the job list 101. The display column 506 displays billing source information. The billing source information is a part of the recognition result data.

[0247] The detailed information is displayed in the display field 507. Specifically, a list of values ​​of detailed items included in the detailed information is displayed in the display field 507. The display field 507 includes display fields 511, 512, 513, 514, 515, and 516. The display field 507 also includes a display component 510.

[0248] The display component 510 is a display component for instructing the display of an extraction guide image. In other words, the display component 510 is a display component for notifying the server device 200 of an instruction to set areas for item names and item values ​​to be extracted from a tabular image.

[0249] Display field 511 displays the value of the detail item "Item Name" displayed in area 504 showing the detail details in the form image. Display field 512 displays the value of the detail item "Item Code" displayed in area 504. Display field 513 displays the value of the detail item "Item Price" displayed in area 504.

[0250] Display field 514 displays the value of the detail item with the detail item name "Unit Price" displayed in area 504. Display field 515 displays the value of the detail item with the detail item name "Quantity" displayed in area 504. Display field 516 displays the value of the detail item with the detail item name "Order Number" displayed in area 504.

[0251] In this embodiment, item values ​​are thus extracted from items corresponding to the respective detailed item names in the detailed portion of the form image, and displayed in display fields 511, 512, 513, 514, 515, and 516. That is, the items corresponding to these detailed item names are the items to be extracted (extraction target items). Moreover, the extraction target items may be fixed items, or may be changeable by the user.

[0252] 20 shows a case where the extraction of the recognition result data from the form image data is successful. Therefore, the values ​​of all the detail item names in the detail section displayed in the area 504 are displayed in the display field 507.

[0253] A display component 508 is a display component for changing the status value of the job list 101 from an unprocessed state to a draft saved state. When the display component 190 is operated, the display column 505 changes from "unprocessed" to "draft saved".

[0254] The display component 509 is a display component for setting the status value of the job list 101 to the confirmed state. In the example of Fig. 20, the recognition result data is not corrected because extraction of the recognition result data from the form image data was successful. Therefore, in the example of Fig. 20, when the display component 509 is operated, the status value of the job list 101 is set to the confirmed state, and the individual company detailed definition information 251 is not generated.

[0255] The form recognition unit 270 of this embodiment may store, on the confirmation screen, correction history information indicating that the recognized image data has been corrected, and operation history information indicating that the display component 510 has been operated. Furthermore, when the display component 509 is operated and neither the correction history information nor the operation history information is stored, the form recognition unit 270 may set the status value of the job list 101 to the confirmed state.

[0256] 21 is a second diagram showing a display example. A screen 501A shown in FIG.

[0257] Display field 502A displays a form image indicated by the form image data, and the form image includes area 504A indicating the details section (table format). In addition, in the details section of area 504A, there are no detailed item names in column 541. For this reason, in the detailed information displayed in display field 507A of display field 503A, values ​​of the detailed items corresponding to column 541 of the details section are not displayed.

[0258] Furthermore, even if a character string other than the line item name "Order Number", such as "Order ID", is entered in column 541 of the details section, it is not recognized as the line item name "Order Number", and therefore the value of the line item corresponding to column 541 of the details section is not displayed.

[0259] Specifically, in display field 507A, display field 516 is left blank. In this embodiment, when display field 516 is left blank, a generic item name may be displayed as a hint to support the user in inputting data into display field 516. In the example of Fig. 21, "order number" is displayed in display field 516 as a hint.

[0260] When display component 510 is operated in display field 507A, screen 501A transitions to screen 501B.

[0261] 22 is a third diagram showing a display example. A screen 501B shown in FIG.

[0262] A message 521 prompting the user to specify a column (extraction area) that associates the detailed item names and detailed item values ​​in the detailed section, and a display component 522 for canceling the operation of specifying the column are displayed in display field 520. When display component 522 is operated in display field 520, screen 501B transitions to screen 501A.

[0263] In response to an operation on the display component 510 on the screen 501A, an extraction guide image 550 is displayed in the display field 502B. In response to an operation on the display component 510, the display field 502B also displays a display field 551, a display component 552, and a display component 553 together with the extraction guide image 550.

[0264] In display field 551, a list of detail item names associated with the area indicated by extraction guide image 550 is displayed by extraction guide image 550. That is, a list of items to be extracted is displayed. These items to be extracted correspond to detail items 511, 512, 513, 514, 515, and 516 included in display field 507. Display component 552 is a display component for operating to increase or decrease the number of lines in the extraction guide image.

[0265] The display component 553 is a display component for reflecting in the display field 507 the value of the detail item indicated by the detail item name associated with the area indicated by the extraction guide image 550 .

[0266] In other words, the display component 553 is a display component for instructing to extract and associate item names and item values ​​in an extraction area indicated by an extraction guide image from an image of the detail portion. When an operation is performed on the display component 553 in the terminal device 400, the item extraction unit 277 of this embodiment is deemed to have received an instruction to extract item names and item values.

[0267] Upon receiving an extraction instruction, the item extraction unit 277 extracts keywords in the area indicating the item name (detailed item name) in the extraction area and keywords in the area indicating the item value (detailed item value) in the tabular image (detailed portion).

[0268] 22, it can be seen from the extraction guide image 550 that the order number, product name, product code, quantity, unit price, and amount are predetermined as detailed item names associated with the area indicated by the extraction guide image 550. These detailed item names are, for example, item names commonly used in the detailed sections of forms including invoices, delivery notes, purchase slips, etc., and are predetermined.

[0269] In the example of Fig. 22, the extraction guide image 550 includes a rectangular image of the same size as the image of the detailed portion of the area 504A, and an image of lines arranged in the rectangle at a position overlapping the ruled lines of the detailed portion. In other words, the extraction guide image 550 is an image that overlaps the outer frame and the ruled lines of the table-format image showing the detailed portion. In other words, the extraction guide image 550 is an image that includes a rectangle that surrounds the columns included in the detailed portion.

[0270] The area indicated by the image surrounding the column in the extraction guide image 550 corresponds to the detail item name indicated by the text image displayed in this area. The text image indicating the detail item name is displayed as a part of the extraction guide image 550.

[0271] In addition, when displaying a character image, the display control unit 292 of this embodiment refers to the recognition results using the general-purpose detail definition information 253 and displays the character image at a position corresponding to the detail item name of the detail section indicated by area 504A.

[0272] Specifically, character image 561 indicates "product code", and this character image 561 is displayed at a position corresponding to column 571 of the detail item name "product code" in the detail portion in area 504A. Therefore, the area indicated by column 571 in the image of the detail portion is an area associated with the detail item name "product code" by extraction guide image 550.

[0273] Similarly, text image 562 indicates "Product Name", and this text image 562 is displayed at a position corresponding to column 572 of the line item name "Product Name" in the description section in area 504A. Therefore, the area indicated by column 572 in the image of the description section becomes an area associated with the line item name "Product Name" by extraction guide image 550. Moreover, keywords displayed in cells in the second and subsequent rows of column 572 are set as values ​​of the line item "Product Name".

[0274] Furthermore, text image 563 indicates "order number." This text image 563 is displayed in a position corresponding to column 573 of the detail item name "remarks" in the details section in area 504A. In this state, the area indicated by column 573 is associated with the detail item name "order number."

[0275] In this embodiment, when an operation is received to move the rectangular image 581 overlapping with the column 573 in the extraction guide image 550 to a position overlapping with the area indicated by the column 541 in the detail portion, the area indicated by the column 541 is associated with the line item name "order number." In this case, the character image 563 displayed within the rectangle also moves together with the rectangular image 581.

[0276] In this embodiment, an operation on the display component 552a of the display components 552 is received to increase the number of lines in the extraction guide image 550. In this embodiment, it is preferable that the number of lines in the extraction guide image 550 is operated so that it becomes the same as the number of lines in the detail portion in the area 504A. Note that when the display component 552b is operated, the number of lines in the extraction guide image 550 is decreased.

[0277] 23 is a fourth diagram showing a display example. In display field 502B of screen 501B shown in FIG.

[0278] In this embodiment, this operation causes the area indicated by the column 541 in the details section to correspond to the line item name "order number" in the extraction guide image 550. In other words, the keywords displayed on the second and subsequent lines in the column 541 in the details section are set to the value of the line item "order number."

[0279] In this manner, in this embodiment, an extraction guide image 550 is displayed in which predetermined areas of specific detail item names correspond to areas of detail item values ​​in accordance with the rows and columns of the image of the detail portion contained in the form image, which is a tabular image.

[0280] In this embodiment, when an operation to move a rectangular image associated with a specific detail item name included in the extraction guide image 550 is accepted, the rectangular image is moved. In this embodiment, the area in the image of the detail portion where the rectangular image is superimposed is set as an area where the specific detail item name associated with the rectangular image and the value of the detail item are displayed.

[0281] In this embodiment, when display component 553 is operated in the state shown in FIG. 23, the keywords displayed from the second line onwards in column 541 of the details section are extracted as the value of the detail item "order number" and reflected in display field 516 (see FIG. 21).

[0282] Then, the definition registration unit 290 generates the individual company detail definition information 251 by the definition generation unit 293. Specifically, the definition generation unit 293 acquires information indicating the position of the top cell of each column included in the extraction guide image 550 in the form image, and the detail item name associated with this column by the extraction guide image 550. The information indicating the position of the top cell is cell coordinates or the like.

[0283] Next, the definition generation unit 293 associates the acquired detailed item name with the value of the item “detailed item name” of the individual company detailed definition information 251, and associates the acquired cell coordinates with the value of the item “extracted information” of the individual company detailed definition information 251.

[0284] In addition, if the detail item name associated with the area indicating the column in the detail section by the extraction guide image 550 is different from the detail item name displayed in the detail section, the definition generation unit 293 sets the detail item name in the detail section as a keyword for the item “extraction information” of the individual company detail definition information 251.

[0285] 23, for example, in column 571, the detailed item name in the detailed section and the detailed item name associated by extraction guide image 550 are both "product code." Therefore, the definition generating unit 293 sets the value of the item "detailed item name" of individual company detailed definition information 251 to "product code," and sets the value of the item "extracted information" to the cell coordinates of the top cell of column 571 in the form image.

[0286] The definition generating unit 293 may include the keyword "product code" in the value of the item "extracted information".

[0287] In column 572, the detailed item name associated by extraction guide image 550 is “Product Name”, and the detailed item name in the detailed section is “Product Name”. Therefore, the definition generating unit 293 sets the value of the item “Detailed Item Name” in the individual company detailed definition information 251 to “Product Name”, and sets the value of the item “Extracted Information” to the cell coordinate of the top cell in column 572 and the keyword “Product Name” displayed in column 572 in the detailed section.

[0288] In column 573, the detailed item name associated by extraction guide image 550 is "order number," and there is no detailed item name in the detailed portion. Therefore, the definition generating unit 293 sets the value of the item "detailed item name" of individual company detailed definition information 251 to "order number," and sets the value of the item "extracted information" to the cell coordinates of the top cell in column 541.

[0289] In the above manner, the definition generating unit 293 generates the individual company detail definition information 251 corresponding to each column included in the extraction guide image 550.

[0290] As described above, in this embodiment, when an operation to set the position of an extraction guide image in a tabular image is received, the area in the tabular image where the extraction guide image overlaps becomes the extraction area from which the item names and item values ​​associated by the extraction guide image are extracted.

[0291] In other words, in this embodiment, an extraction guide image including a rectangular image surrounding each column of the detail section represented in tabular format is displayed, and by moving the rectangular image, the detail item names that are pre-associated with the columns of the extraction guide image are associated with the areas indicated by each column of the detail section.

[0292] Therefore, according to this embodiment, when setting an extraction region for extracting the detail item name and the detail item value for the image of the detail portion shown in a table format, there is no need to adjust the column spacing, etc. Also, according to this embodiment, since the extraction guide image includes multiple rectangular images associated with the specific detail item names, extraction regions corresponding to multiple detail item names can be set on one screen.

[0293] In this manner, according to this embodiment, an area for extracting item names and item values ​​can be set with a simple operation in a tabular image. 23, the extraction guide image 550 is displayed to include the detailed item name, but is not limited to this. The extraction guide image 550 does not need to include an area for the detailed item name. In this case, the extraction guide image 550 only needs to include an area for the detailed item value, and the text image 561 etc. may be displayed on the first line of the area for the detailed item value.

[0294] When display component 509 is operated, screen 501B in Fig. 23 transitions to screen 501C. Fig. 24 is a fifth diagram showing a display example.

[0295] 24, a display field 523 is displayed in a display field 520. An input field 524 and a display component 525 are displayed in the display field 523.

[0296] The input field 524 is an input field for inputting information about the billing source. The display component 525 is a display component for instructing storage of the individual company detail definition information 251 generated based on the extraction area set by the extraction guide image 550 in the detail definition database 250.

[0297] In this embodiment, when the display component 525 is operated, the individual company detail definition information 251 generated by the definition generating unit 293 is stored in the detail definition database 250 .

[0298] 25 is a sixth diagram showing a display example. A screen 501D shown in FIG.

[0299] A form image is displayed in display field 502C, and area 504B in the form image shows an image of the details section. In the details section shown in area 504B, there is no column in which the detailed item name "unit price" corresponds to the value of the detailed item "unit price."

[0300] Therefore, in display column 507B of display column 503B, display column 514 in which the value of the detailed item name "unit price" is displayed is blank.

[0301] When display component 510 is operated in display field 507B, screen 501D transitions to screen 501E.

[0302] 26 is a seventh diagram showing a display example. A screen 501E shown in FIG.

[0303] In the display field 502D, an extraction guide image 550 is superimposed on the image of the detail portion indicated by the region 504B. Also, in the display field 502D, a display field 551, a display component 552, and a display component 553 are displayed.

[0304] At this time, extraction guide image 550 has columns corresponding to all the detailed item names displayed in display field 551. Therefore, in the image of the detailed portion, in column 574 displaying the detailed item name "remarks", a rectangular image associated with the detailed item name "unit price" in extraction guide image 550 and character image 264 indicating "unit price" are displayed superimposed.

[0305] In this embodiment, in this case, by selecting a detail item name that does not exist in the detail section from the list of detail item names displayed in display field 551, the image showing the area associated with the non-existent detail item name can be erased from extraction guide image 550.

[0306] Fig. 27 is an eighth diagram showing a display example. Screen 501F shown in Fig. 27 is an example of a screen when an operation to select "unit price" in display field 551 on screen 501E is accepted.

[0307] In this embodiment, in response to this operation, display column 502D changes the display mode of the detail item name "unit price" in display column 551 to be different from the display modes of the other detail item names. Specifically, in this embodiment, the detail item name selected in display column 551 may be displayed in gray out.

[0308] Furthermore, on screen 501F, the rectangular image superimposed on column 574 in the image of the details portion and text image 564 indicating "unit price" are erased from extraction guide image 550. When the operation to select "unit price" is received again in display field 551, the rectangular image superimposed on column 574 in the image of the details portion that was once deleted and text image 564 indicating "unit price" are displayed again. In this way, the user can specify which area corresponding to which detail item name is to be displayed in extraction guide image 550 by operating display field 551. That is, the area corresponding to the item to be extracted that is displayed in extraction guide image 550 can be increased or decreased in response to the user's operation.

[0309] In this manner, in this embodiment, by deleting a detailed item name previously associated with the extraction guide image 550, it is possible to erase a rectangular image indicating an area associated with this detailed item name (an extraction area of ​​an item to be extracted associated with this detailed item name) in the extraction guide image 550. Therefore, in this embodiment, it is possible to adjust the number of rectangular images included in the extraction guide image 550 according to the detailed item names included in the detailed portion or their number.

[0310] Furthermore, when the display component 553 is operated with the detail item name deleted, the definition registration unit 290 may generate individual company detail definition information 251 including information indicating that the detail item name has been deleted (that the item to be extracted has been deleted). In this way, when the extraction guide image 550 is displayed for other form images of the same issuer, the deleted detail item name can be identified by referring to the individual company detail definition information 251, and the extraction guide image 550 can be displayed with the rectangular image indicating the area corresponding to the detail item name deleted.

[0311] Fig. 28 is a ninth diagram showing a display example. In Fig. 28, another example of an operation on the extraction guide image 550 will be described.

[0312] FIG. 28A shows an example in which an extraction guide image 550 is displayed superimposed on the detailed portion of a form image displayed on a confirmation screen.

[0313] 28(A), in the image of the details section, the top cell of column 575 displays the line item name "Item Name," and the cells below are blank. Also, in the image of the details section, the top cell of column 576 is blank, and the cells below it display the values ​​"A" and "B."

[0314] An image of a column 582 associated with the detailed item name "product name" in the extraction guide image 550 is displayed superimposed on the column 575 of images of the detailed item.

[0315] FIG. 28(B) is a diagram for explaining an operation on the extraction guide image 550 displayed in the state of FIG. 28(A).

[0316] 28, the item name "item name" displayed in the top cell of column 575 actually corresponds to the values ​​displayed in the second and subsequent cells from the top of column 576. In this embodiment, in such a case, an operation to widen the width of column 582 of extraction guide image 550 is accepted.

[0317] In FIG. 28B, an image of column 582a in which the width of column 582 of extraction guide image 550 has been expanded is displayed superimposed on columns 575 and 576 of the images of the detail portion.

[0318] By this operation, in Figure 28 (B), the areas of columns 575 and 576 in the image of the detail portion that overlap with the image of column 582a in extraction guide image 550 can be made into extraction areas from which the detail item name "Product Name" and the value of the detail item "Product Name" are extracted.

[0319] In the above embodiment, the detail portion of the form image is in a table format with ruled lines, but is not limited to this. The detail portion may be displayed in a manner in which the background color of each column is different. In this case, the display control unit 292 may detect an area with different background colors as one column.

[0320] In the above embodiment, the extraction guide image is displayed so as to overlap the column of the detail portion including the item name and the item value, but the present invention is not limited to this. For example, the extraction guide image may be displayed so as to overlap only the area of ​​the item value of the column of the detail portion. In other words, the user may set only the extraction area of ​​the item value in the extraction guide image, and specify the extraction area of ​​the item name based on the extraction area of ​​the set item value.

[0321] Fig. 29 is a tenth diagram showing a display example. Fig. 29 shows a case where the details in the form image displayed on the confirmation screen are displayed as a multi-column details including multiple detailed item names in one example.

[0322] Fig. 29(A) shows a detail portion 504C in the form image displayed on the confirmation screen. In Fig. 29(A), the extraction guide image 550 is not displayed.

[0323] In the details section 504C shown in Fig. 29(A), the top cell of column 577 displays the detail item name "Product Code", and the cell below that displays the detail item name "Product Name". Also, the third cell from the top of column 577 displays the value of the detail item name "Product Code", and the cell below that displays the value of the detail item name "Product Name". In other words, in the details section 504C, two lines show one detail.

[0324] FIG. 29B shows an example in which an extraction guide image 550 is superimposed on a detail portion 504C in a form image displayed on a confirmation screen.

[0325] In the detail section 504C shown in Fig. 29(B), a green extraction guide image 550 is displayed for setting an extraction area for each item in one detail. These extraction guide images 550 are independent of each other in one detail and do not link with each other. In addition, the position and size of each extraction guide image 550 can be set individually.

[0326] Specifically, for example, the character image "Product Name" can be moved below the character image "Product Code," and the width of the column associated with the character image "Price" can be set to match the width of the columns associated with the character images "Quantity" and "Unit Price."

[0327] In the example of Fig. 29(B), when the position and size of one detail are changed, the positions and sizes of the second and subsequent details are changed in the same way. Here, for the second and subsequent details, the extraction positions may be calculated automatically based on the coordinate values ​​and size of each detail item name set in the first detail.

[0328] 29(B), a display component 552A is displayed in the details portion 504C. The display component 552A is a display component for operating to increase or decrease the number of details in the extraction guide image 550 in the details portion 504C.

[0329] In the details section 504C, when the number of details is set to two or more, the extraction guide images 550 displayed for each item in one detail are linked to each other. For example, when the size or position of the column of the character image "product code" is changed, the size or position of the extraction guide images 550 displayed for the other detail item names is automatically changed.

[0330] If the number of details is set to two or more, after this change, it is not possible to move the position of the text image "Product Price" down to the text image "Product Name" or to move the text image "Unit Price" to the right of the text image "Quantity".

[0331] On the other hand, in the details section 504C, when the number of details is set to "1", the extraction guide images 550 displayed for each item in one detail are not linked to each other. Therefore, the extraction guide image 550 can be freely operated, for example, by moving the position of the text image "product price" below the text image "product name" or by moving the text image "unit price" to the right of the text image "quantity".

[0332] Furthermore, in the example of Fig. 29(B), a dotted line image 550A may be displayed in the field of the item value of the column of the detail portion 504C, and the field to be extracted may be specified by operating on the dotted line image 550A. Note that, although the dotted line image 550A is used in Fig. 29(B), the type of line is not limited to a dotted line.

[0333] By displaying dotted line image 550A in this manner, only the portion in which detailed information is displayed is extracted, and blank portions are not extracted, so that there is no need to perform the "increase / decrease columns" operation.

[0334] In this embodiment, by displaying the extraction guide image 550 as described above for the detail portion 504C displayed in the form of a multi-column detail, it is possible to set an extraction region for each detail.

[0335] In the above embodiment, the table-format image on which the extraction guide image is displayed is described as an image of the detailed portion of a form image, but the present embodiment is not limited to this. The present embodiment can be applied to images other than form images as long as the image is a table-format image in which the names of multiple items to be used can be guessed in advance. Specifically, the present embodiment can be applied to images of report cards, resumes, and the like, for example.

[0336] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0337] Additionally, the devices described in the embodiments are merely representative of one of multiple computing environments for implementing the embodiments disclosed herein.

[0338] In one embodiment, server apparatus 200 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, server apparatus 200 may include multiple computing devices configured to communicate with each other.

[0339] Furthermore, the server device 200 can be configured to share the disclosed processing steps in various combinations. For example, the processes performed by the server device 200 can be performed by other server devices. Similarly, the functions of the server device 200 can be performed by other server devices. Also, the server device and each element of the other server devices can be integrated into one server device or separated into multiple devices.

[0340] The database in the statement may be generated by the learning effect of machine learning. Also, by classifying keywords that may be included in the description of transaction details and account items by machine learning, it is not necessary to use a correspondence table.

[0341] Here, machine learning is a technology that allows a computer to acquire human-like learning capabilities, where the computer autonomously generates algorithms necessary for judgments such as data identification from training data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; the learning method for machine learning is not important.

[0342] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form. [Explanation of symbols]

[0343] 100 Document Recognition System 200 Server device 210 Storage section 220 Form recognition processing unit 230 Form Image Database 240 Industry Determination Database 250 Item definition database 260 Recognition Result Database 270 Document Recognition Unit 271 Image data storage unit 272 Job List Generation Unit 273 Image data acquisition unit 274 Pretreatment section 275 Character recognition section 276 Definition Specification Part 277 Item extraction part 278 Display screen generation section 279 Output section 290 Definition Registration Section 291 Operation reception section 292 Display control section 293 Definition generator 294 Definition update section 298 Communications Department 300 Image forming device 400 Terminal Equipment 500 Core Systems [Prior art documents] [Patent documents]

[0344] [Patent Document 1] JP 2016-51339 A

Claims

1. a display control unit that displays, on a terminal device, a setting screen for setting an extraction area for extracting each item value of one or more predetermined items to be extracted from a tabular image, the setting screen on which an extraction guide indicating the extraction area corresponding to each of the one or more items to be extracted is displayed on the tabular image; an operation receiving unit that receives an operation for setting a position of the extraction guide on the setting screen, the setting screen includes a list of the extraction target items that can be selected and accepted, The display control unit is When a selection of the extraction target item in the list is accepted, the information processing system erases the extraction guide corresponding to the extraction target item selected from the setting screen.

2. The operation reception unit is accepting an operation to designate the item to be extracted on the setting screen; The display control unit is 2 . The information processing system according to claim 1 , further comprising: displaying the extraction guide indicating the extraction area corresponding to each of the specified extraction target items among the one or more extraction target items.

3. The display control unit is 3. The information processing system according to claim 1, wherein the extraction guide is displayed to indicate the extraction area in which the item names and item values ​​of each of the one or more extraction target items are extracted, and in which an area indicating the item names and an area indicating the item values ​​corresponding to each of the one or more extraction target items are associated.

4. the tabular image is an image of a detail portion of a form image, 4. The information processing system according to claim 3, wherein the item name and the item value are a detailed item name included in the details section and a detailed item value indicated by the detailed item name.

5. 5. The information processing system according to claim 4, further comprising an item extraction unit which, upon receiving an instruction to extract the detail item name and the detail item value from the detail portion, extracts and associates, from the image of the detail portion, a keyword of an area indicated by the detail item name in the extracted area with a keyword of an area indicating the detail item value.

6. An information processing system as described in claim 4 or 5, further comprising a definition registration unit that generates detail definition information that associates information indicating the position of the area indicated by the detail item name in the image of the detail portion with a keyword for the area indicated by the detail item name, and stores the information in a memory unit.

7. The operation reception unit is The information processing system according to claim 3 , further comprising: a user interface configured to enable a user to access the extracted information from the extracted information, the user interface being configured to enable a user to access the extracted information from the extracted information.

8. The extraction guide comprises: a column that associates a plurality of the predetermined item names with the predetermined item values; The display control unit is The information processing system according to claim 3 , wherein the extraction guide is displayed so that a width of a column included in the extraction guide matches a width of a column of the table shown by the tabular image.

9. The display control unit is a display component for performing an operation to increase or decrease the number of lines included in the extraction guide together with the extraction guide; 9. The information processing system according to claim 8, further comprising: a list of the predetermined item names displayed on the terminal device.

10. The display control unit:

2. The information processing system according to claim 1, wherein a display mode of the extraction target item selected in the list is made different from a display mode of other extraction target items in the list.

11. An information processing system as described in claim 6, wherein when the extraction guide is erased by selecting the item to be extracted in the list, the detail definition information includes information indicating that the selected item to be extracted has been deleted.

12. An information processing system having a server device and a terminal device, The server device includes: a first display control unit that causes a terminal device to display a setting screen for setting an extraction region for extracting each item value of one or more predetermined extraction target items from a tabular image, the setting screen including an extraction guide indicating the extraction region corresponding to each of the one or more extraction target items being displayed on the tabular image; a communication unit that transmits an instruction to display the setting screen to the terminal device and receives information indicating an operation on the setting screen; an operation receiving unit that receives an operation for setting a position of the extraction guide on the setting screen, The terminal device A display unit; an input receiving unit for receiving instructions from the server device; a second display control unit that causes the display unit to display the setting screen in response to an instruction received by the input receiving unit, the setting screen includes a list of the extraction target items that can be selected and accepted, The first display control unit, When a selection of the extraction target item in the list is accepted, the information processing system erases the extraction guide corresponding to the extraction target item selected from the setting screen.

13. An information processing method by a computer, comprising: displaying, on a terminal device, a setting screen for setting an extraction region for extracting each item value of one or more predetermined extraction target items from a tabular image, the setting screen having an extraction guide showing the extraction region corresponding to each of the one or more extraction target items displayed on the tabular image; accepting an operation for setting a position of the extraction guide on the setting screen; the setting screen includes a list of the extraction target items that can be selected and accepted, When a selection of the extraction target item is accepted from the list, the extraction guide corresponding to the extraction target item selected from the setting screen is erased.

14. displaying, on a terminal device, a setting screen for setting an extraction region for extracting each item value of one or more predetermined extraction target items from a tabular image, the setting screen having an extraction guide showing the extraction region corresponding to each of the one or more extraction target items displayed on the tabular image; accepting an operation for setting a position of the extraction guide on the setting screen; the setting screen includes a list of the extraction target items that can be selected and accepted, The information processing program causes a computer to execute a process of erasing the extraction guide corresponding to the extraction target item selected from the setting screen when a selection of the extraction target item in the list is accepted.

15. A display control unit that displays on a terminal device a setting screen for setting an extraction area for extracting each item value of one or more predetermined items to be extracted from a tabular image, the setting screen having an extraction guide showing the extraction area corresponding to each of the one or more items to be extracted displayed on the tabular image; and an operation receiving unit that receives an operation for setting a position of the extraction guide on the setting screen, The display control unit is displaying the extraction guide indicating the extraction area for extracting the item names and item values ​​of each of the one or more extraction target items, the extraction guide indicating the extraction area being associated with an area indicating the item names and an area indicating the item values ​​corresponding to each of the one or more extraction target items; The operation reception unit is an information processing system that accepts an operation to move, on the tabular image, a position of a column that associates the predetermined item name and the predetermined item value included in the extraction guide.

16. An information processing system having a server device and a terminal device, The server device includes: a first display control unit that causes a terminal device to display a setting screen for setting an extraction region for extracting each item value of one or more predetermined extraction target items from a tabular image, the setting screen including an extraction guide indicating the extraction region corresponding to each of the one or more extraction target items being displayed on the tabular image; a communication unit that transmits an instruction to display the setting screen to the terminal device and receives information indicating an operation on the setting screen; an operation receiving unit that receives an operation for setting a position of the extraction guide on the setting screen, The terminal device A display unit; an input receiving unit for receiving instructions from the server device; a second display control unit that causes the display unit to display the setting screen in response to an instruction received by the input receiving unit, The first display control unit, displaying the extraction guide indicating the extraction area for extracting the item names and item values ​​of each of the one or more extraction target items, the extraction guide indicating the extraction area being associated with an area indicating the item names and an area indicating the item values ​​corresponding to each of the one or more extraction target items; The operation reception unit is an information processing system that accepts an operation to move, on the tabular image, a position of a column that associates the predetermined item name and the predetermined item value included in the extraction guide.

17. A method for processing information by a computer, comprising: a display step of displaying on a terminal device a setting screen for setting an extraction area for extracting each item value of one or more predetermined items to be extracted from a tabular image, the setting screen having an extraction guide showing the extraction area corresponding to each of the one or more items to be extracted displayed on the tabular image; An operation receiving step of receiving an operation for setting a position of the extraction guide on the setting screen; the display step includes displaying the extraction guide indicating the extraction area in which an item name and an item value of each of the one or more extraction target items are extracted, the extraction guide indicating the extraction area being associated with an area indicating the item name and an area indicating the item value corresponding to each of the one or more extraction target items; The operation reception procedure includes: An information processing method comprising: receiving an operation for moving, on the tabular image, a position of a column that associates the predetermined item name and the predetermined item value included in the extraction guide.

18. A display process for displaying on a terminal device a setting screen for setting an extraction area for extracting each item value of one or more predetermined items to be extracted from a tabular image, the setting screen having an extraction guide showing the extraction area corresponding to each of the one or more items to be extracted displayed on the tabular image; an operation receiving process for receiving an operation for setting a position of the extraction guide on the setting screen, The display process includes: executing a process of displaying the extraction guide indicating the extraction area in which an item name and an item value of each of the one or more extraction target items are extracted, the extraction guide indicating the extraction area being associated with an area indicating the item name and an area indicating the item value corresponding to each of the one or more extraction target items; The operation reception process includes: an information processing program that causes a user to execute a process of accepting an operation to move, on the tabular image, a position of a column that associates the specified item name and the specified item value included in the extraction guide;

Images