Information Processing System and Program

Abstract

To reduce workload of a user required for classifying images compared with a case where the user manually sets a reference for classifying images.SOLUTION: When a form image 302 about a "financing agreement" is newly input, determination information (form feature information) is used to determine that the type of the newly input form image 302 is the "financing agreement". More specifically, when the form image 302 about the "financing agreement" is newly input, a type determination unit determines whether or not the form image 302 has form feature information associated with an expression image 202. Then, if the form image 302 has the form feature information associated with the expression image 202, the type determination unit determines that the type of the form image 302 is the "financing agreement."SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an information processing system and a program.

Background Art

[0002] Patent Document 1 discloses a process of, when a plurality of key documents are specified, specifying the dependency of the processes specified for each of the plurality of key documents from the key documents and related documents related to the processes, and determining the execution order of the processes specified for each of the plurality of key documents using the specified dependency.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The image data includes various images. The object of the present invention is When the image data includes a predetermined image, it is intended to execute a process associated with the predetermined image on the image data.

Means for Solving the Problems

[0005] The invention according to claim 1 includes a processor, and the processor acquires input image data and, when the acquired image data includes a predetermined image, executes the process associated with the predetermined image on the image data. An information processing system That is, when the acquired image data includes the predetermined image and the image data has a feature registered in advance, the processor does not execute the process associated with the predetermined image, but executes the process associated with the feature on the image data. This is an information processing system. The claim 2 The invention according to claim is that, when executing the process associated with the feature, the processor varies the process to be executed according to whether the acquired image data includes a predetermined text. The information processing system according to claim 1 described above. The claim 3The invention described in [reference] is a program for causing a computer to realize a function of acquiring input image data and, when the acquired image data includes a predetermined image, executing a process associated with the predetermined image on the image data. That is, when the acquired image data includes the predetermined image and the image data has a feature registered in advance, the function for executing the process does not execute the process associated with the predetermined image, but executes the process associated with the feature on the image data. This is a program. That's what it is.

Advantages of the Invention

[0006] Claims 1 to 3 According to the invention of [claims], when the image data includes a predetermined image, the process associated with the predetermined image can be executed on the image data. Also, when the image data includes a predetermined image and the image data has a feature registered in advance, the process associated with the predetermined image may not be executed, and the process associated with the feature may be executed on the image data. That's possible.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

MODE FOR CARRYING OUT THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram showing the information processing system 1 of the present embodiment. The information processing system 1 of the present embodiment is provided with a form data receiving unit 10 that receives form data, which is form data, a type determination unit 20 that determines the type of the form represented by the form data, and a determination information generation unit 30 that generates determination information used for determining the type of the form. In addition, the information processing system 1 is provided with a processing execution unit 40 that executes a predetermined process on the form data, and a processing content setting unit 50 that sets the processing content performed by the processing execution unit 40.

[0009] The form data receiving unit 10 receives form data which is data of a form. More specifically, the form data is image data obtained by scanning the form, and the form data receiving unit 10 receives this image data. The type determination unit 20 determines the type of the form represented by the input form data by using the determination information generated by the determination information generation unit 30. More specifically, the type determination unit 20 determines which type the form represented by the form data is by using the prepared determination information.

[0010] When the type determination unit 20 determines the type of the form, the result of this determination (information indicating the type of the form) and the form (form data) are passed to the process execution unit 40. The process execution unit 40 executes a process associated with the type obtained by the type determination unit 20 for the passed form. In this embodiment, for each type of form, the process to be performed is set in advance, and the process execution unit 40 executes a process corresponding to the type of this passed form.

[0011] Note that in this embodiment, the form may be sent back from the process execution unit 40 to the determination information generation unit 30. In this embodiment, during the process in the process execution unit 40, visual confirmation work by a confirmator may be performed, and in this confirmation work, it may be determined that the type determination performed by the type determination unit 20 is incorrect.

[0012] In this case, the form is sent back from the process execution unit 40 to the determination information generation unit 30. More specifically, a notification indicating that the type determination is incorrect is sent from the process execution unit 40 to the determination information generation unit 30. In this case, in this embodiment, as will be described later, the determination information generation unit 30 generates new determination information so that the sent-back form can be determined more correctly. Here, the "form" is a general term for books, vouchers, etc., and refers to a medium such as a sheet of paper with blank spaces for entry. In this embodiment, processing of this form will be described, but the information processing system 1 of this embodiment may also be used for general documents that are not forms.

[0013] FIG. 2 is a diagram showing an example of the hardware configuration of the information processing system 1. The information processing system 1 of this embodiment includes an information processing unit 101, an information storage device 102 for storing information, and a network interface 103 for realizing communication via a LAN (= Local Area Network) cable or the like. In addition, the information processing system 1 is provided with a reception device 104 for receiving an operator's operation and a display device 105 for displaying information.

[0014] The reception device 104 is composed of a keyboard, a mouse, etc. In this embodiment, information input by the operator to the information processing system 1 is received by this reception device 104. The keyboard and the mouse are examples, and the reception device 104 is not particularly limited. In addition, for example, a device that recognizes the operator's voice and receives information input by the operator, a touch panel, etc. may also be used. In addition, the display device 105 is composed of a liquid crystal display, an organic EL display, etc. In this embodiment, various screens described later are displayed on this display device 105.

[0015] The information storage device 102 is realized by an existing information storage device such as a hard disk drive. A hard disk drive is a device that reads and writes data to a non-volatile storage medium in which a magnetic substance is applied to the surface of a disk-shaped substrate. However, the information storage device 102 may also be a semiconductor memory or a magnetic tape.

[0016] The information processing unit 101 includes a CPU (= Central Processing Unit) 111 as an example of a processor, a ROM (= Read Only Memory) 112 in which basic software, BIOS (= Basic Input Output System), etc. are stored, and a RAM (= Random Access Memory) 113 used as a work area. The CPU 111 may be multi-core. Also, the ROM 112 may be a rewritable non-volatile semiconductor memory. The information processing unit 101 is a so-called computer. The information processing unit 101, the information storage device 102, the network interface 103, the reception device 104, and the display device 105 are connected through a bus 106 and signal lines (not shown).

[0017] Here, the program executed by the CPU 111 can be provided to the information processing system 1 in a state stored in a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory. Also, the program executed by the CPU 111 may be provided to the information processing system 1 using communication means such as the Internet.

[0018] In this embodiment, the processor refers to a processor in a broad sense and includes a general-purpose processor (e.g., CPU: Central Processing Unit, etc.) and a dedicated processor (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). Also, the operation of the processor may be achieved not only by one processor but also by a plurality of physically separated processors cooperating. Also, the order of each operation of the processor is not limited to the order described in this embodiment and may be changed.

[0019] In this embodiment, a CPU 111 as an example of a processor executes programs stored in a ROM 112 and an information storage device 102, thereby realizing the above-mentioned form data reception unit 10, type determination unit 20, determination information generation unit 30, process execution unit 40, and process content setting unit 50. Each of the processes described below is performed by the form data reception unit 10, the type determination unit 20, the determination information generation unit 30, the process execution unit 40, and the process content setting unit 50. In other words, each of the processes described below is performed by a CPU 111 as an example of a processor that realizes these functional units. Also, each of the various processes described below may be performed using a single information processing device provided with a processor, or may be performed using a plurality of information processing devices each provided with a processor.

[0020] FIG. 3 is a flowchart showing an example of the flow of processing executed in the information processing system 1. In this embodiment, first, form data is input to the information processing system 1, and the form data reception unit 10 receives this form data (step S101). More specifically, as described above, form data that is image data obtained by scanning a form is input to the information processing system 1, and the form data reception unit 10 receives the form data that is image data. Next, in this embodiment, the type determination unit 20 determines whether the above-mentioned determination information used for determining the type of the form is registered in the information processing system 1 (step S102).

[0021] If it is determined in step S102 that the determination information is not registered, the type of the form cannot be determined. In this case, the type determination unit 20 determines that the type of the input form is unknown (step S103). In this case, in this embodiment, the form (form with unknown type) determined to have an unknown type is displayed on a confirmation screen (described later) (step S104). Then, in this case, a setting (hereinafter referred to as "type determination setting") is made by the operator so that an operation of the operator on this confirmation screen is performed to determine the type of the form (step S105).

[0022] FIG. 4 is a flowchart showing the flow of processing executed when the operator makes a type determination setting in step S105 above. In the present embodiment, when the operator makes a type determination setting, the operator performs an operation on the confirmation screen on which a form determined to have an unknown type is displayed. Specifically, the operator performs an operation on the confirmation screen to first generate a representative image for classifying the form (step S201). In other words, the operator generates a representative image for classifying the image of the form (form image) displayed on the confirmation screen. This generated representative image is displayed on the confirmation screen.

[0023] Then, the operator further performs an operation on this confirmation screen to associate the generated representative image with the form (form with unknown type) determined to have an unknown type (step S202). Specifically, the operator performs so-called drag-and-drop to move the form image representing the form determined to have an unknown type to the representative image to associate the representative image with the form image.

[0024] Next, in the present embodiment, the determination information generation unit 30 analyzes this form image associated with the representative image and acquires information indicating the characteristics of this form image (hereinafter referred to as "form characteristic information"). Here, this form characteristic information is an example of the above determination information, and in this process, the determination information generation unit 30 generates form characteristic information as an example of the determination information (step S203).

[0025] Next, the determination information generation unit 30 associates the generated determination information with the generated representation image (step S204). In other words, the determination information generation unit 30 associates the form feature information with the representation image. In this way, in the present embodiment, determination information used for determining the type of form is generated, and this determination information is associated with the representation image. In other words, in the present embodiment, the determination information is associated with the type of form.

[0026] In the present embodiment, as described above, the operator performs so-called drag-and-drop to move the form image representing the form to the representation image and associate the form image with the representation image. As a result, the form feature information, which is information about the features of the form (form image), is associated with the representation image. In other words, the determination information obtained based on the form is associated with the representation image. With the above, the type determination setting in step S105 shown in FIG. 3 is completed. Thereafter, when a new form is input, the determination information set by this type determination setting is used to determine the type of this new form.

[0027] The process shown in FIG. 3 will be further described. In step S102, when it is determined that the determination information is registered, the type of the form specified by the form data received in step S101 is determined using this registered determination information (step S106). Then, in step S107, it is determined whether the type can be determined.

[0028] If it is determined in step S107 that the type of the form cannot be determined, this form is determined to be a form with an unknown type (step S108). In this case, the processes after step S104 described above are executed. That is, in this case, the form determined to have an unknown type is displayed on the confirmation screen, and then the operator performs the type determination setting.

[0029] On the other hand, in step S107, when it is determined that the type of the form can be determined, the process execution unit 40 performs a process associated with the determined type on the form (form data) (step S109). Next, in this embodiment, it is determined whether there is a rollback for this form on which the process by the process execution unit 40 has been performed (step S110). That is, it is determined whether the checker has determined that the type determination performed by the type determination unit 20 is incorrect. In this embodiment, as described above, there may be a case where a visual confirmation operation by the checker is performed. In this confirmation operation, if it is determined that the type determination is incorrect, the form is rolled back.

[0030] If this rollback is performed, the processes after step S104 are performed. Thereby, also in this case, as above, first, the form determined to have an incorrect type and rolled back is displayed on the confirmation screen. Next, in order to make the type determination of this rolled-back form more accurate, the type determination setting is performed by the operator. In this type determination setting, for example, as described above, generation of a new representative image and association of the form image with this representative image are performed. In addition, in the type determination setting, as will be described later, setting for text determination may also be performed.

[0031] FIG. 5 is a diagram showing a confirmation screen. Referring to FIG. 5 and the figures after FIG. 6, the type determination setting will be described in detail. In this embodiment, as described above, the form image 302 representing the form determined to have an unknown type and the form image 302 representing the rolled-back form are displayed on the confirmation screen 200 shown in FIG. 5. On this confirmation screen 200, in the left column, a representative image 202 representing the type of the form, which is a representative image 202 that has already been generated, is displayed.

[0032] In this embodiment, determination information used for determining the type of form is associated with each of these representative images 202. More specifically, in this embodiment, information about the representative image 202 and the determination information associated with this representative image 202 are registered in the information storage device 102 (see FIG. 2) in a state where they are associated with each other.

[0033] At the top of each of the plurality of form images 302 displayed on the confirmation screen 200, a display of "unknown" is made, indicating that the type of the form represented by the form image 302 is unknown. Each of the form images 302 with the display of "unknown" represents, for example, a form image 302 of a form for which the type could not be determined by the determination information associated with the already generated representative image 202. In addition, each of the form images 302 with the display of "unknown" represents, for example, a form image 302 of a form for which the determined type is incorrect and has been sent back.

[0034] On this confirmation screen 200, first, the operator checks the content of each of the form images 302 with the display of "unknown". The operator also checks whether a representative image 202 corresponding to the content of each of these form images 302 already exists. When the representative image 202 corresponding to the form image 302 does not exist, the operator selects "Add Form Type" indicated by the symbol 5A at the lower left of the confirmation screen 200. Thereby, as shown in FIG. 6 (a diagram showing a generation screen for generating a representative image 202), a generation screen 400 for generating a new representative image 202 is displayed.

[0035] On this generation screen 400, as indicated by the symbol 6A, a column named "Form Name" is displayed at the upper left of the screen. After grasping the content of the form image 302 (see FIG. 5) displayed as "unknown", the operator inputs a specific name into this column via an input device 104 (see FIG. 2) such as a keyboard. In other words, the operator inputs a specific name for identifying each of the types. In FIG. 6, the case where the name "financing contract document" is input is illustrated.

[0036] Also, on this generation screen 400, a selection item 410 of "always perform form determination when the number of samples is small" is displayed, and in this example, the case where this selection item 410 is selected by the operator is illustrated. Also, on this screen, a selection item 420 of "additional learning" is displayed, and the case where this selection item 420 is selected by the operator is illustrated. Note that the details of these selection items will be described later.

[0037] When the operator finishes the operation on the generation screen 400 shown in FIG. 6, the operator selects "OK" in the lower right of the screen. Thereby, the screen is switched, and the confirmation screen 200 is redisplayed. FIG. 7 shows the confirmation screen 200 after redisplay. On this confirmation screen 200 after redisplay, as indicated by reference numeral 7A, a representative image 202 corresponding to the "financing contract document" is generated in the upper left of the screen. In other words, a representative image 202 associated with the type of "financing contract document" is generated. Here, the "representative image" is an image associated with information about the type of form, and when the user views this representative image, it refers to an image in which the user can recognize the information about the type.

[0038] In this embodiment, from this state, the operator performs a drag-and-drop operation to move the form image 302 indicated by reference numeral 7C, which is displayed in the center of the screen, to the representative image 202 corresponding to the "financing contract document". In this example, the form image 302 displayed in the center of the screen is a form image 302 representing the "financing contract document". The operator performs a drag-and-drop operation to move this form image 302 displayed in the center of the screen to the representative image 202 associated with the type of "financing contract document".

[0039] Accordingly, in the present embodiment, the determination information generation unit 30 (see FIG. 1) grasps the features of the form image 302 displayed at the center of the screen, and acquires information about the grasped features (hereinafter referred to as "form feature information") as determination information. Next, the determination information generation unit 30 associates the generated form feature information with the above-described representative image 202 with the type of "financing contract". In other words, the determination information generation unit 30 associates the generated form feature information with the type of "financing contract". Accordingly, in the present embodiment, information about the representative image 202 corresponding to the "financing contract" and form feature information obtained from the "financing contract" are registered in the information storage device 102 (see FIG. 2) in a state where they are associated with each other.

[0040] Note that when grasping the features of the form image 302, the determination information generation unit 30 analyzes the form image 302, for example, obtains a histogram of the form image 302, and grasps this histogram as a feature of the form image 302. Note that the grasping of the features of the form image 302 is not limited to the grasping of this histogram, and other known methods may be used.

[0041] In the present embodiment, in this way, the operator performs an operation of associating the representative image 202 used for classifying the form and the form image 302 representing the form. Then, in the present embodiment, when the information processing system 1 receives this operation, the determination information generation unit 30 acquires form feature information about the form image 302, and associates the acquired form feature information with the representative image 202.

[0042] In the present embodiment, the determination information generation unit 30 generates form feature information representing the features of the form image 302, and this form feature information becomes the determination information. Then, the determination information generation unit 30 associates this form feature information as determination information with the representation image 202. In other words, the determination information generation unit 30 associates this form feature information as determination information with the information representing the type of "financing contract".

[0043] In this embodiment, as described above, an example has been described in which the form feature information is associated with the representation image 202 by moving the form image 302 to the representation image 202. However, the user operations for the association are not limited to this. The movement of the form image 302 is not essential. As long as the information processing system 1 can recognize that there is a relationship between the form image 302 and the representation image 202, the method of association is not particularly limited.

[0044] For example, as shown in FIG. 8 (another display example on the confirmation screen 200), when the right mouse button is clicked after placing the mouse pointer P on the form image 302, the already generated representation image 202 may be displayed. In this example, the representation image 202 selected by the user from the displayed representation images 202 is associated with the form feature information obtained from the form image 302. Each of the representation images 202 shown in FIG. 8 is a representation image composed of characters, and the representation image 202 may be composed of character information in this way. In addition, as the representation image 202, it may be an image that does not include character information and visualizes the character information.

[0045] In this embodiment, hereinafter, when a new form image 302 for a "financing contract" is input, the above-generated determination information (form feature information) is used to determine that the type of the newly input form image 302 is a "financing contract". More specifically, when a form image 302 of a "financing contract document" is newly input, the type determination unit 20 determines whether the above form feature information (form feature information obtained from the "financing contract document") associated with the representation image 202 is possessed by this form image 302.

[0046] And when the form image 302 has the above form feature information associated with the representation image 202, the type determination unit 20 determines that the type of this form image 302 is a "financing contract document". In other words, in the present embodiment, when a form image 302, which is another image, has the feature (feature specified by the form feature information) associated with the representation image 202, the information associated with the representation image 202 is associated with this form image 302. Specifically, in the present embodiment, in this case, information indicating the type, which is associated with the representation image 202, is associated with this form image 302. Specifically, in this case, a "financing contract document", which is information indicating the type, is associated with this form image 302.

[0047] In the present embodiment, as described above, drag and drop is performed, and form feature information corresponding to the "financing contract document" is automatically generated. And in the present embodiment, hereafter, when a form corresponding to the "financing contract document" is newly input, it is not determined that the type of this newly input form is "unknown". This newly input form is determined to have the above automatically generated form feature information and is determined to have a type of "financing contract document".

[0048] Note that in the above, the process of associating the form image 302 located at the center of the confirmation screen 200 (see FIG. 7) with the representation image 202 has been described. However, for other form images 302 located outside the center of the confirmation screen 200, the association with the representation image 202 is similarly performed. Incidentally, when generating the representation images 202 corresponding to the form images 302 on the confirmation screen 200 in sequence, a situation may occur where the representation image 202 corresponding to the form image 302 displayed as "unknown" has already been generated. In this case, the operator moves the form image 302 to the already generated representation image 202 without generating a new representation image 202, and associates the form image 302 with this representation image 202.

[0049] In the present embodiment, on the confirmation screen 200 where the representation image 202 is displayed, as shown in FIG. 9 (a diagram showing another display example on the confirmation screen 200), when the operator selects the representation image 202 by moving the pointer P onto the representation image 202, the display indicated by reference numeral 9A is performed. In the display indicated by this reference numeral 9A, the determination method used by the type determination unit 20 when determining the type is displayed. Here, as an example, the case where "machine learning determination" is displayed as the determination method is illustrated. This "machine learning determination" refers to a method of determining the type of a newly input form based on the above-mentioned form feature information obtained by analyzing the form image 302 and associated with the representation image 202.

[0050] Also, in the present embodiment, on this confirmation screen 200, when the operator selects the representation image 202 by performing a right click with the pointer P positioned on the representation image 202, a display indicated by reference numeral 9B, such as "setting and editing", is performed. When the operator selects this "setting and editing", a setting confirmation screen 380 that enables confirmation of the settings related to the representation image 202 is displayed as shown by reference numeral 9C. On this setting confirmation screen 380, the same screen as the generation screen 400 shown in FIG. 6 is displayed.

[0051] On this setting confirmation screen 380, similar to the generation screen 400 shown in FIG. 6, there is a column for the form name in the upper left of the screen. In this example, the name "financing contract" is displayed in this column. In addition, on this setting confirmation screen 380, it is displayed that the selection item 410 of "always perform form determination when the number of samples is small" is selected, and it is also displayed that the selection item 420 of "additional learning" is selected.

[0052] FIG. 10 is a diagram showing the confirmation screen 200 after the association between the form image 302 and the representative image 202 displayed on the confirmation screen 200 shown in FIG. 7 is performed. Before the association between the form image 302 and the representative image 202 is performed, as shown in FIG. 7, "unknown" was displayed above each of the form images 302.

[0053] On the other hand, when the association is performed, as shown in FIG. 10, above each form image 302, the name associated with the representative image 202 to which the form image 302 is associated is displayed. In other words, above each form image 302, the type of the form represented by each form image 302 is displayed. And in this embodiment, thereafter, on this confirmation screen 200 shown in FIG. 10, the operator selects "OK" in the lower right. Thereby, the processing execution unit 40 executes the processing associated with the type of the form image 302 for each of the form images 302.

[0054] In this embodiment, if at least one form feature information can be associated with the newly generated representative image 202, thereafter, based on this one form feature information, the type of the newly input form can be determined. Specifically, for example, if one form feature information obtained from the form image 302 of the "financing contract" can be associated with the representative image 202 of the "financing contract", the type of the new form corresponding to the "financing contract" can be determined as the "financing contract". However, in this case, since the type determination is made based only on one form feature information obtained from one form, the accuracy of the determination is low and an incorrect determination is likely to be made.

[0055] Therefore, in the present embodiment, in order to make it difficult to make an incorrect determination, a process is performed to allow an operator to confirm the result of the type determination performed by the type determination unit 20 for each newly input document. More specifically, when the above selection item 410 of "always perform document determination when the number of samples is small" is selected by the operator, and the number of document feature information (referred to as "number of information associations") associated with the representative image 202 is smaller than a predetermined threshold value, a process is performed to allow the operator to confirm the result of the type determination. More specifically, a confirmation display 390 shown in FIG. 11 (a diagram showing a confirmation display) is performed to allow the operator to confirm the result of the type determination.

[0056] In this confirmation display 390, a plurality of document images 302 representing the documents for which the type determination unit 20 has made a type determination are displayed. More specifically, in this example, document images 302 each representing a plurality of documents for which the type determination unit 20 has determined that the type is a "financing contract" are displayed. In this confirmation display 390, character information indicating the type, such as "financing contract", is displayed in association with each document. The operator refers to this confirmation display 390 and confirms the result of the determination made by the type determination unit 20 for each document.

[0057] In the present embodiment, when the status of the association of features with the representative image 202 is in a predetermined specific status, the operator is notified of the result of grasping the type of the above document image 302, which is an example of another image.

[0058] In the present embodiment, based on the features (document feature information) associated with the representative image 202, the type of other images such as the newly input document image 302 is grasped. At this time, when the status of the association of features with the representative image 202 is in a predetermined specific status, the operator is notified of the result of grasping the type of the document image 302, which is an example of this other image. More specifically, in the present embodiment, when the number of form images 302 in which features are associated with the representative image 202 is smaller than a predetermined threshold value, the operator is notified of the result of identifying the type of the newly identified form image 302.

[0059] And in the present embodiment, when the operator confirms the result of identifying the type, if there is no error in the result of identifying the type, the operator selects "OK" in the lower right of the screen. In this case, the process execution unit 40 (see FIG. 1) executes a process associated with the type of "financing contract" for each form. In the present embodiment, based on the features associated with the representative image 202, the types of other images such as the above form image 302 are identified, and the process associated with the identified type is performed on this other image.

[0060] In this case, when the status of associating features with the representative image 202 is in a predetermined specific status, the process associated with the above type is not performed on the form image 302 which is the above other image. Specifically, in the present embodiment, when the number of images in which features are associated with the representative image 202 is smaller than a predetermined threshold value, the process associated with the type is not performed on the form image 302.

[0061] In other words, in the present embodiment, when the number of images in which features are associated with the representative image 202 is smaller than a predetermined threshold value, the process associated with the type that is to be performed on the form image 302 is not automatically performed. In the present embodiment, when the status of associating features with the representative image 202 is in a specific status, as described above, when "OK" is selected by the operator and there is permission from the operator, the process associated with the type is performed on the form image 302.

[0062] Also, in the present embodiment, when the operator selects "OK" in the lower right of the screen, the number of information associations is updated to the latest number. Specifically, in this case, the number of invoice images 302 that the operator has confirmed the type of this time is added to the information association number, which is the number of invoice feature information associated with the representative image 202 corresponding to the "financing contract document" (the representative image 202 indicated by reference numeral 11A), and the information association number is updated to the latest number.

[0063] In addition, even if the representative image 202 corresponding to the "financing contract document" already exists, it is assumed that the invoice of the "financing contract document" may be regarded as "unknown" due to a processing error or the like. In this case, the operator moves the invoice image 302 of the invoice determined to be "unknown" to the representative image 202 indicating the "financing contract document" by drag and drop.

[0064] In addition, it is also assumed that an invoice other than the "financing contract document" may be erroneously determined to be the "financing contract document". In this case, the operator performs drag and drop on the invoice image 302 erroneously determined to be the "financing contract document" and moves it to the representative image 202 corresponding to the correct invoice.

[0065] In addition, it is assumed that the type of the invoice is determined to be "unknown" due to the fact that the representative image 202 has not been generated yet, and accordingly, the invoice image 302 with the type being "unknown" is displayed. In this case, as described above, after the user generates a new representative image 202, the invoice image 302 displayed as "unknown" is associated with this new representative image 202.

[0066] FIG. 12 is a diagram showing a display screen after the "OK" displayed in the confirmation display 390 of FIG. 11 is selected by the operator. In other words, it is a diagram showing the display screen when the operator gives an instruction to start the process by the process execution unit 40. Here, assume a case where the operator gives an instruction to start the execution of the process in a state where the selection item 410 (see FIG. 6) of "always perform invoice determination when the number of samples is small" is selected and the above information association number is small.

[0067] In this case, as shown in FIG. 12, information about the number of images is displayed. More specifically, in this case, the processing execution unit 40 outputs information about the number of images for which the association of features with the representative image 202 has been performed. Accordingly, as shown in FIG. 12, information about the number of these images is displayed. More specifically, in this example, information about the required number of samples for the form image 302 is displayed. In other words, it is displayed that the number of information associations is small and has not reached a predetermined threshold value.

[0068] In the present embodiment, when the number of images for which the association of features with the representative image 202 has been performed is smaller than a predetermined threshold value, the processing execution unit 40 outputs information indicating that the number of images for which the association of features with the representative image 202 needs to be performed is insufficient. And in the present embodiment, this information is displayed. Note that, as the information about the number of images for which the association of features with the representative image 202 has been performed, information about the insufficient number of images may be displayed in this way, or the number of images for which the association has been performed itself may be displayed.

[0069] In the display in FIG. 12, information about the number of form images 302 that the operator must check the type of is displayed. In other words, in the display in FIG. 12, information about the number of form images 302 that the user needs to visually check the type of is displayed. In other words, in the display in FIG. 12, information about the number of form feature information that needs to be associated with the representative image 202 is displayed.

[0070] In the present embodiment, this display shown in FIG. 12 is performed until the number of form images 302 associated with the representative image 202 reaches a predetermined threshold value. In the present embodiment, when the number of form images 302 associated with the representative image 202 has not reached a predetermined threshold value, the processing by the processing execution unit 40 is not started unless this display shown in FIG. 12 is passed through.

[0071] Also, in the display shown in FIG. 12, as shown by reference numeral 12A, a display for receiving from the operator the cancellation of the selection regarding the selection item 410 is performed. Specifically, a display for receiving from the operator the cancellation of the selection regarding the selection item 410 of "always perform form determination when the number of samples is small" is performed. In the present embodiment, when an operation by the operator is performed on this display and an instruction to cancel the selection is given, thereafter, the confirmation display 390 shown in FIG. 11 is not performed. Also, in this case (when an instruction to cancel the selection is given), thereafter, the processing by the processing execution unit 40 is automatically performed. In other words, in this case, even when there is no instruction to start the processing from the operator, the processing by the processing execution unit 40 is started.

[0072] FIG. 13 is a diagram showing another example of a screen after "OK" displayed in the confirmation display 390 shown in FIG. 11 is selected. In the present embodiment, even when the selection item 410 of "always perform form determination when the number of samples is small" is selected, when "OK" is selected in a state where the number of samples is complete, the screen shown in FIG. 13 is displayed.

[0073] In this screen shown in FIG. 13, a display indicating that the number of samples has been satisfied is performed. In other words, in this screen, a display indicating that the number of form images 302 that need to be associated with the representative image 202 has reached a predetermined threshold value is performed. In this way, when the number of samples is satisfied, thereafter, the confirmation display 390 shown in FIG. 11 is not performed.

[0074] Also, when the number of samples is satisfied in this way, the display shown in FIG. 12 is not performed hereafter. That is, the display regarding the number of images in which the features are associated with the representative image 202 is not performed. In the present embodiment, when the number of images in which the features are associated with the representative image is smaller than a predetermined threshold value, as described above, the information regarding the number of images is output and displayed. On the other hand, when the number of images in which the features are associated with the representative image is not smaller than the predetermined threshold value, the information regarding the number of images is not output, and the display of the information regarding the number of images is not performed.

[0075] In the present embodiment, as shown in FIG. 6, there is also a selection item 420 called "additional learning". In the present embodiment, when this selection item 420 is selected, when the type of the newly input form image 302 is automatically determined, or when the form image 302 is associated with the representative image 202 by the operator, the form feature information is acquired from this form image 302.

[0076] Then, new determination information is generated in consideration of the acquired form feature information. More specifically, in addition to the form feature information acquired in the past, new determination information associated with the type of this form image 302 is generated including the acquired form feature information.

[0077] In the present embodiment, the type of the newly input form image 302 is determined automatically or by the operator. In this case, when the selection item 420 called "additional learning" is selected, new determination information is generated based on both the form feature information obtained from this newly input form image 302 and the form feature information already obtained.

[0078] More specifically, in the present embodiment, as described above, except when the type of the form is determined not to be "unknown", basically, the type of the form image 302 is automatically determined. In this case, when the selection item 420 "additional learning" is selected, new determination information is generated from the form feature information of the form image 302 for which the type has been automatically determined and the form feature information obtained in the past.

[0079] Also, for a form whose type is determined to be "unknown", the operator associates the form image 302 with the representation image 202, whereby the type of the form image 302 is substantially determined. Also in this case, when the selection item 420 "additional learning" is selected, new determination information is generated from the form feature information of the form image 302 for which the type has been determined by the operator and the form feature information obtained in the past.

[0080] FIG. 14 is a diagram showing a generation screen 400 when an operator generates a representation image 202 corresponding to a "lease contract" different from the above-mentioned "financing contract". More specifically, in this example, FIG. 400 shows the generation screen 400 when a form image 302 for a "lease contract" different from the "financing contract" is input to the information processing system 1 (see FIG. 1).

[0081] Also, in FIG. 14, the generation screen 400 is shown when the determination information corresponding to the "lease contract" has not yet been registered and the type of the form image 302 for the "lease contract" is determined to be "unknown". Also, in FIG. 14, the generation screen 400 when the operator generates the representation image 202 corresponding to this "lease contract" is shown.

[0082] In this example, on this generation screen 400, as indicated by reference numeral 14A, a display indicating that similar forms have already been registered is provided. In other words, a display indicating that similar determination information has been registered is provided. More specifically, in this example, the form of the "loan agreement" and the form of the "lease agreement" are similar. Accordingly, the form feature information (information for determination) of the "loan agreement" and the form feature information of the "lease agreement" are similar, and as a result, a display indicating that similar information for determination has already been registered is made.

[0083] In this example, although the similarity between these two pieces of form feature information is determined to be less than a predetermined first threshold value and the two are different from each other, a case where this similarity is greater than a second threshold value (< the first threshold value) is illustrated. Here, in the present embodiment, when new form feature information is obtained from a form determined as "unknown", as described above, this new form feature information is associated with the corresponding representation image 202. At this time, the similarity between the already registered form feature information and this new form feature information is determined.

[0084] As described above, when the similarity is less than the first threshold value, since the above two pieces of form feature information are somewhat different, hereinafter, the type of the form of the "lease agreement" input to the information processing system 1 can be determined tentatively. However, as described above, when the similarity is greater than the second threshold value, the "lease agreement" may be misjudged as the "loan agreement", or the "loan agreement" may be misjudged as the "lease agreement", and misjudgment is likely to occur.

[0085] In the present embodiment, when associating new form feature information with a new representation image 202, if form feature information similar to this new form feature information already exists, as indicated by reference numeral 14A in FIG. 14, a notification indicating that a similar form exists is made.

[0086] Furthermore, in the present embodiment, a process for making misjudgment less likely to occur is executed. Specifically, in the present embodiment, when the operator selects "infer determination method" indicated by reference numeral 14B on this screen shown in FIG. 14, as shown in FIG. 15 (a diagram showing a reference screen referred to by the operator), information for determination for determining the type of the "lease agreement" is notified to the operator. More specifically, in this embodiment, when the operator selects "Speculate on the determination method" indicated by reference numeral 14B in FIG. 14, determination information for determining the form of the "lease contract" is generated, and this determination information is notified to the operator.

[0087] Specifically, in this example, as determination information for determining the form of the "lease contract", information indicating that text determination is to be performed and the content of the text determination process are generated, and as shown in FIG. 15, this information is notified to the operator. More specifically, in this example, as determination information, an example is shown where determination information such as "when the text obtained by reading with OCR is the 'lease contract', the type of this form is determined to be the 'lease contract'" is generated and this determination information is notified to the operator.

[0088] When performing this notification regarding text determination, the determination information generation unit 30 first extracts a coordinate region R in which the difference from the form of the "loan contract" among the forms of the "lease contract" is particularly large. Then, the determination information generation unit 30 acquires the text information included in this coordinate region R of the "lease contract". Thereby, in this embodiment, text information of the "lease contract" is obtained. And in this case, the determination information generation unit 30 generates determination information such as "when the text obtained by reading with OCR is the 'lease contract', the type of this form is determined to be the 'lease contract'" as described above.

[0089] Then, on this screen shown in FIG. 15, when "OK" in the lower right of the figure is selected by the operator, hereafter, for the "lease contract", the type of the form will be determined by machine learning determination and text determination. In other words, in this embodiment, when determining the type of the form of the "lease contract", not only machine learning determination but also text determination is performed, and then it is determined whether the form is a "lease contract" form.

[0090] More specifically, when determining the type of form of the "lease contract document", after determining whether the text "lease contract document" is included in a predetermined specific part, it is determined whether it is a form of the "lease contract document". Here, the "text determination" refers to a method of determining the type of form based on whether a predetermined registered text is included in the form.

[0091] In this embodiment, the determination method can be changed. For example, by the operator performing an operation on the screen shown in FIG. 16 (a diagram showing the screen referred to by the operator when changing the determination method), deletion of the determination method or selection of the determination method can be performed. Specifically, on this screen, by selecting the location indicated by reference numeral 16A, deletion of the determination method can be performed. Specifically, in this example, although machine learning determination and text determination are displayed, by selecting the location indicated by reference numeral 16A, deletion of one or both of this machine learning determination and text determination can be performed.

[0092] Also, in this example, selection of the determination method can be performed. By selecting the location indicated by reference numeral 16B, selection of the machine learning determination can be performed. Also, in this example, by selecting the location indicated by reference numeral 16C, selection of the text determination can be performed. Also, in the above, the case where text determination is automatically added has been described as an example. However, when the "+" indicated by reference numeral 16D is selected, a screen for adding text determination is displayed (not shown), and the operator can manually add text determination. Also, in this embodiment, by selecting "+", addition such as determination by registered file name, determination by barcode / QR, and determination by partial image, which will be described later, can also be performed.

[0093] FIG. 17 is a diagram showing a priority table. In this embodiment, information about the order when determining the type of form is registered in advance in this priority table 480. Note that this priority table 480 is stored in the information storage device 102 (FIG. 2). In this embodiment, the operator sets priorities in advance for each type of form, and these priorities are registered in the priority table 480. The type determination unit 20 determines the type of the form according to the priorities registered in the priority table 480. More specifically, when there is an input of form data, the type determination unit 20 determines the type of the form in the order of the priorities registered in the priority table 480.

[0094] Specifically, in this embodiment, when there is an input of a form (form data), first, the type determination unit 20 performs "text determination, machine learning determination" associated with priority 1 to determine whether the input form is a "lease contract". Specifically, the type determination unit 20 determines whether the input form has the above-mentioned form feature information obtained from the "lease contract" and whether the text "lease contract" is included in the input form. And when the type determination unit 20 satisfies these two conditions, it determines that the type of the form is a "lease contract". In this case, the process execution unit 40 (see FIG. 1) executes a process corresponding to the type of "lease contract" for this form.

[0095] In other words, the type determination unit 20 determines whether the input form, which is an example of other images, has the features (feature information) associated with the representative image 202 of the "lease contract". In addition, the type determination unit 20 determines whether the text information of "lease contract", which is an example of pre-registered information, is included in this input form. And when the input form has the features associated with the representative image 202 of the "lease contract" and the text information of "lease contract" is included in this input form, the type determination unit 20 associates the information associated with the representative image 202 of the "lease contract" with this input form. Specifically, in this case, the type determination unit 20 associates the information of "lease contract" with the input form. In other words, in this case, the type determination unit 20 associates the information representing the type of the form, namely "lease contract", with the input form. Next, in this case, the processing execution unit 40 executes processing corresponding to the type of "lease contract" on the input form.

[0096] Note that at this time, as described above, when the selection item 410 of "always perform form determination when the number of samples is small" is selected and the number of form feature information associated with the representative image 202 is smaller than a predetermined threshold value, as shown in FIG. 11, a confirmation display 390 is performed. Also, at this time, when the selection item 420 of additional learning is selected, the form feature information obtained from the input form ("lease contract") is also added, and new determination information is generated.

[0097] In the present embodiment, as described above, the form feature information of the "financing contract" and the form feature information of the "lease contract" are similar. If only the form feature information is used to determine the "lease contract", there is a risk of misjudging it as a "financing contract". On the contrary, with a configuration in which text determination is also performed as in the present embodiment, such misjudgment is less likely to occur. In the present embodiment, the case of determining the "lease contract" using text determination and machine learning determination has been described, but it is not limited thereto, and it may be determined whether it is a "lease contract" using only text determination.

[0098] Next, the processing when it is not determined that the type of the form is a "lease contract" in the determination at priority 1 will be described. In this case, the type determination unit 20 performs "machine learning determination" associated with priority 2 to determine whether the type of the form is a "financing contract". Specifically, the type determination unit 20 determines whether the input form image 302 has the above-described form feature information obtained by associating the form image 302 of the "financing contract" with the representation image 202.

[0099] In other words, the type determination unit 20 determines whether the input form image 302 has the form feature information associated with the representation image 202 of the "financing contract". And when the input form image 302 has this form feature information, the type determination unit 20 determines that the type of the form is the "financing contract". In this case, the process execution unit 40 executes a process corresponding to the type of "financing contract" on this form.

[0100] Also, in this case, similar to the above, when the selection item 410 of "always perform form determination when the number of samples is small" is selected, and when the number of form feature information associated with the representation image 202 is less than a predetermined threshold, the confirmation display 390 shown in FIG. 11 is performed. Also, at this time, when the selection item 420 of additional learning is selected, the form feature information obtained from this form image 302 of the "financing contract" is also added, and new determination information is generated.

[0101] In this processing example, for a form of a "lease contract" that may be erroneously determined to be a "financing contract", it is highly likely that it has already been determined as a "lease contract" by the determination based on priority 1. Therefore, in this determination with priority 2, it is unlikely that a "lease contract" will be erroneously determined as a "financing contract".

[0102] Next, the processing when it is not determined that the type of the form is a "financing contract" in the determination with priority 2 will be described. In this case, the type determination unit 20 performs "machine learning determination" associated with priority 3 to determine whether the type of the form is a "driver's license". Specifically, the type determination unit 20 determines whether the input form has the features specified by the form feature information obtained by associating the "driver's license" with the representative image 202. If the input form has this feature, the type determination unit 20 determines that the type of the form is "driver's license". In this case, the process execution unit 40 executes a process corresponding to the type of "driver's license" on this form.

[0103] Also in this case, similar to the above, when the selection item 410 of "always perform form determination when the number of samples is small" is selected and the number of form feature information associated with the representative image 202 is smaller than a predetermined threshold, the confirmation display 390 shown in FIG. 11 is performed. Also, similar to the above, when the selection item 420 of additional learning is selected, the form feature information obtained from this form image 302 of "driver's license" is also added, and new determination information is generated.

[0104] Next, the process when it is not determined that the type of the form is "driver's license" in the determination at priority 3 will be described. In this case, the type determination unit 20 makes a determination based on the "registered file name" associated with priority 4 to determine whether the type of the form is "residence card". Specifically, the type determination unit 20 determines whether the file name of the input form is a pre-registered file name. If this condition is satisfied, the type determination unit 20 determines that the type of the form is "residence card". In this case, the process execution unit 40 executes a process corresponding to the type of "residence card" on this form.

[0105] Next, the process when it is not determined that the type of the form is "residence card" in the determination at priority 4 will be described. In this case, the type determination unit 20 makes a determination based on the "barcode / QR image" associated with priority 5 to determine whether the type of the form is "invoice". Specifically, the type determination unit 20 determines whether a barcode or a QR image exists at a predetermined location of the input form image 302. And when this condition is satisfied, the type determination unit 20 determines that the type of the form is "invoice". In this case, the processing execution unit 40 executes processing corresponding to the type of "invoice" on this form.

[0106] Next, the processing when it is not determined that the type of the form is "invoice" in the determination at priority 5 will be described. In this case, the type determination unit 20 performs determination by "partial image (template matching)" associated with priority 6 to determine whether the type of the form is "design drawing". Specifically, the type determination unit 20 determines whether the input form image 302 contains a pre-registered image. And when the type determination unit 20 satisfies this condition, it determines that the type of the form is "design drawing". In this case, the processing execution unit 40 executes processing corresponding to the type of "design drawing" on this form.

[0107] Next, the processing when it is not determined that the type of the form is "design drawing" in the determination at priority 6 will be described. In this case, the type determination unit 20 determines that the type of the form is "unknown". In this case, the confirmation screen 200 shown in FIG. 5 is displayed. And in this case, as described above, after the representative image 202 is generated by the operator, the form image 302 determined to be "unknown" is associated with this representative image 202. Or, in this case, this form image 302 determined to be "unknown" is associated with any of the already generated representative images 202. There may be cases where it is determined to be "unknown" due to misjudgment. In this case, this form image 302 determined to be "unknown" is associated with any of the already generated representative images 202.

[0108] FIG. 18 is a diagram showing another display example on the confirmation screen 200, which is a confirmation screen 200 in which the rolled-back document is displayed. In this display example, although the document type indicated by reference numeral 18A is a "lease contract", it is determined to be a "financing contract", and this illustrates the case where the document is rolled back. In this case, in the present embodiment, when the operator places the pointer P on the displayed document (document image 302) and selects this document, as shown by reference numeral 18B, the determination method used for the determination of this document is displayed. In this example, "machine learning determination" is displayed as the determination method.

[0109] In the present embodiment, on this confirmation screen 200, when "identification setting edit" displayed by selecting "action" in the upper right is selected by the operator, the screen shown in FIG. 16 is displayed. Thereby, it becomes possible to add a determination method and switch the determination method. And in this case, when an operation on the screen shown in FIG. 16 is performed and, for example, a switch to the above "text determination" is made, thereafter, when a "lease contract" document is input, it is correctly determined as a "lease contract". In this case, the rollback of the document is less likely to occur.

[0110] Furthermore, in the present embodiment, as described above, after switching to "text determination", the operator can immediately perform re-determination of the type. Specifically, in the present embodiment, as shown in FIG. 19 (a diagram showing a display example on the confirmation screen 200), when the operator selects "re-determination" displayed by selecting "action" in the upper right, for the document indicated by reference numeral 19A (the same document as the document indicated by reference numeral 18A in FIG. 18), determination by text determination is performed. Thereby, in this case, as shown by reference numeral 19B, the correct type "lease contract" is displayed.

[0111] Also, on this confirmation screen 200 shown in FIG. 19, similar to the above, when the pointer P is placed on the displayed form, as shown by reference numeral 19E, the determination method used for the determination of this form is displayed. In this example, as described above, a switch from machine learning determination to text determination has been made, and as shown by reference numeral 19E, text determination is displayed.

Explanation of Reference Numerals

[0112] 1... Information processing system, 10... Form data reception unit, 20... Type determination unit, 30... Determination information generation unit, 40... Processing execution unit, 50... Processing content setting unit, 111... CPU, 202... Representational image, 302... Form image

Claims

1. An information processing system comprising a processor, wherein the processor obtains input image data, and when the obtained image data includes a predetermined image, executes processing associated with the predetermined image on the image data. The processor when the obtained image data includes the predetermined image and the image data has a previously registered feature, does not execute the processing associated with the predetermined image, but executes processing associated with the feature on the image data. An information processing system.

2. The information processing system according to claim 1, wherein, when executing the processing associated with the feature, the processor varies the processing to be executed according to whether the obtained image data includes predetermined text.

3. A program for causing a computer to realize a function of obtaining input image data and a function of, when the obtained image data includes a predetermined image, executing processing associated with the predetermined image on the image data, wherein the function of executing the processing when the obtained image data includes the predetermined image and the image data has a previously registered feature, does not execute the processing associated with the predetermined image, but executes processing associated with the feature on the image data. A program. ​ ​ ​ ​

Images