Information processing systems and programs

Abstract

To improve workability of a user upon correction of an extraction result as compared with an arrangement of not displaying a description part where an extraction candidate is described, if description information described in a non-stylized form is extracted.SOLUTION: The present invention is directed to an information processing system having one or a plurality of processors. In the system, the one or the plurality of processors acquires an extraction candidate as a candidate to be extracted as description information corresponding to an item of a non-stylized form, from described information as information described on the non-stylized form, and displays, on a display unit, a described part where the extraction candidate in the non-stylized form is described.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing system and a program.

Background Art

[0002] Patent Document 1 describes an image processing apparatus including: a character recognition unit that recognizes a character string of a specific item in an image to be recognized based on a result of learning using a plurality of images including character strings; an output unit that outputs the image to be recognized and the character string recognition result in a manner in which a correspondence relationship indicating the same specific item can be grasped between the image to be recognized and the character string recognition result by the character recognition unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When extracting the described information recorded on an irregular form, if the location of the irregular form from which this described information was extracted is displayed on the display unit together with the extracted described information, it is possible to grasp from which location of this irregular form the extraction was performed. Here, for example, the described information may be extracted from a location unintended by the user. In this case, the user needs to search for the intended location on the irregular form or input the intended described information in order to correct the extraction result. An object of the present invention is to improve the workability of the user when correcting the extraction result as compared to a configuration that does not display the described location where the extraction candidate is described when extracting the described information recorded on an irregular form.

Means for Solving the Problems

[0005] The invention described in claim 1 is an information processing system comprising one or more processors, wherein the one or more processors obtain extraction candidates from the description information, which is information described in a non-standard form, which is a candidate to be extracted as the description information corresponding to an item in the non-standard form, The extraction candidates are obtained based on the item and multiple extraction conditions set by the user for that item. A priority is assigned to the extraction candidates that satisfy more of the extraction conditions, rather than to the extraction candidates that satisfy fewer conditions, and the assigned priority is displayed on the display unit. The section of the aforementioned non-standard form in which the extraction candidates are described is The aforementioned It is an information processing system that displays information on a display unit. The invention described in claim 2 is, The information processing system according to claim 1, wherein the one or more processors display on the display unit the reason for the priority assigned to the extraction candidate, based on the status of fulfillment of the extraction conditions, along with the location of the extraction candidate. The invention described in claim 3 is, The information processing system according to claim 2, wherein the one or more processors display the conditions that the extraction candidates have not met as the reason for the priority. The invention described in claim 4 is, The information processing system according to claim 2, wherein the one or more processors display the conditions that the extraction candidate has met as the reason for the priority. The invention described in claim 5 is, The information processing system according to claim 1, wherein the one or more processors display the locations of the extraction candidates on the image of the non-standard form. The invention described in claim 6 is, The information processing system according to claim 1, wherein the one or more processors obtain a plurality of extraction candidates from the information described in the unstructured form, and display the location of the extraction candidate specified by the user from among the plurality of extraction candidates on an image of the unstructured form. The invention described in claim 7 is, The information processing system according to claim 1, wherein the one or more processors, upon receiving a selection of extraction candidates by a user, use the selected extraction candidates as the result of extracting the described information corresponding to the items. The invention described in claim 8 is, The information processing system according to claim 1, wherein the one or more processors learn the selected location of the extraction candidate as the location of the description information corresponding to the item. The invention described in claim 9 is, The information processing system according to claim 1, wherein the one or more processors, upon receiving input of the description information corresponding to the item from a user, take the input description information as the result of extracting the description information corresponding to the item, and detect the description location in the image of the non-standard form in which the input description information is written. The invention described in claim 10 is, The information processing system according to claim 1, wherein the one or more processors learn the detected location as the location of the description information corresponding to the item. The invention described in claim 11 is, The information processing system according to claim 1, wherein the one or more processors obtain the extraction candidates based on the items and the extraction conditions set for the items, correct the extraction candidates based on correction instructions from the user, and even if the corrected extraction candidates do not satisfy all of the extraction conditions, the corrected extraction candidates are used as the result of extracting the described information corresponding to the items. The invention described in claim 12 is, The information processing system according to claim 1, wherein the one or more processors display to the display unit that the corrected extraction candidate does not satisfy all of the extraction conditions. The invention described in claim 13 is, An information processing system comprising one or more processors, wherein the one or more processors obtain extraction candidates from the information recorded in the non-standard form, which are information recorded in the non-standard form, which are candidates to be extracted as the recorded information corresponding to an item in the non-standard form, display the location in the non-standard form where the extraction candidates are recorded on the display unit, obtain the extraction candidates based on the item and the extraction conditions set for that item, correct the extraction candidates based on correction instructions from the user, even if the corrected extraction candidates do not satisfy all of the extraction conditions, the corrected extraction candidates are treated as the result of extracting the recorded information corresponding to the item, weights are added to the extraction conditions that match the corrected extraction candidates, and new extraction candidates are obtained from a new non-standard form based on the item, the extraction conditions and the weights of the extraction conditions. The invention described in claim 14 is, This program enables a computer to perform the following functions: to obtain extraction candidates from the information recorded in a non-standard form, which are candidates to be extracted as corresponding information to items in the non-standard form; to obtain the extraction candidates based on the items and multiple extraction conditions set by the user for those items; to assign a priority to the extraction candidates that satisfy more of the extraction conditions, so as to have priority over the extraction candidates that satisfy fewer conditions; to display the assigned priority on the display unit; and to display on the display unit the location in the non-standard form where the extraction candidates are recorded. [Effects of the Invention]

[0006] Claim 1, 14 According to the invention, By using user-defined criteria to determine priorities, candidates that better match the user's requirements can be given higher priority compared to when the system simply determines priorities. . Claim 2 According to this invention, the user can refer to the reason for the priority of the extraction candidates. This makes it easier for users to understand. According to the inventions of claims 3 and 4, the user can refer to the basis for the priority determination, making it easier for the user to understand. Claim 5 According to this invention, when selecting extraction candidates, the user can refer to the section from which the information has been extracted. Claim6 According to the invention, compared with a configuration that displays all the description locations of extraction candidates on an image of an irregular form, it is possible to suppress the display unit from becoming complicated. Claim 7 According to the invention, compared with a configuration in which the description information corresponding to an item is corrected by manual input by a user, the workability of the user can be improved. Claim 8 According to the invention, compared with a configuration that does not learn the description location of the selected extraction candidate, the accuracy of extracting the description information corresponding to an item can be improved. Claim 9 According to the invention, compared with a configuration in which the user designates the description location of the description information input by the user, the workability of the user when changing the description information corresponding to an item can be improved. Claim 10 According to the invention, compared with a configuration that does not learn the description location detected as the changed description information, the accuracy of extracting the description information corresponding to an item can be improved. Claim 11 According to the invention, even when the corrected extraction candidate does not satisfy all of the extraction conditions, the corrected extraction candidate can be held as the description information corresponding to an item. Claim 12 According to the invention, it is possible to notify the user that the corrected extraction candidate does not satisfy the extraction conditions. Claim 13 According to the invention, compared with a configuration that simply extracts description information that matches any of the extraction conditions, it is possible to extract description information that matches an important condition.

Brief Description of the Drawings

[0007] [Figure 1] It is a diagram showing an example of the overall configuration of an information processing system to which this embodiment is applied. [Figure 2] It is a diagram showing a configuration example of an OCR processing server and a user terminal according to this embodiment. [Figure 3] It is a diagram showing an example of the functional configuration of an OCR processing server according to this embodiment. [Figure 4]This figure shows an example of the functional configuration of a user terminal according to this embodiment. [Figure 5] This flowchart shows the process flow in which the OCR processing server according to this embodiment allows the user to confirm the extracted candidates. [Figure 6] This figure shows an example of the display on the user terminal display unit according to this embodiment. [Figure 7] This diagram shows an example of assigning priorities to extraction candidates. [Figure 8] This figure shows an example of how the extracted candidates are displayed according to this embodiment. [Figure 9] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 10] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 11] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 12] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 13] This figure shows another example of how the extraction candidates are displayed according to this embodiment, where (A) shows a state in which the extraction candidates satisfy a specific condition, and (B) shows a state in which the extraction candidates have not satisfied the specific condition. [Figure 14] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 15] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Figure 16] This flowchart shows the process flow when a user terminal according to this embodiment requests correction of the extraction candidates. [Figure 17] This flowchart illustrates the process flow by which the OCR processing server according to this embodiment learns the extraction results. [Figure 18] This flowchart shows the process flow in which a user terminal according to the embodiment inputs the extraction results. [Figure 19] This figure shows another example of the display on the user terminal display unit according to this embodiment. [Figure 20] This figure shows an example of a confirmation screen according to this embodiment. [Figure 21] This figure shows another example of how the extracted candidates are displayed according to this embodiment. [Modes for carrying out the invention]

[0008] Hereinafter, embodiments for carrying out the present invention will be described with reference to the attached drawings. <Embodiment> [Information Processing System 1] Figure 1 is a diagram showing an example of the overall configuration of the information processing system 1 to which this embodiment is applied. The information processing system 1 according to this embodiment is configured such that an OCR (Optical Character Recognition) processing server 10, which processes image data of unstructured forms using OCR, and a user terminal 30, which is operated by a user, are connected via a communication line 80. This communication line 80 is, for example, a line such as the Internet, and is used for information communication between the OCR processing server 10 and the user terminal 30. Furthermore, the information processing system 1 may also include a document management server 20 for managing user documents, an image forming apparatus 40 with the function of reading original documents, and a business server 50 which is the output destination for the results of OCR processing of image data of non-standard forms.

[0009] In this embodiment, "non-standard forms" refer to forms in which the locations for extracting information and the meaning of the extracted information are not predetermined. Examples of non-standard forms include application forms, invoices, delivery slips, and other forms in which the positions of the entry fields are not uniform. The entry fields of non-standard forms may be filled in by hand or by printing. Non-standard forms include, for example, office documents created with office software or other application programs, emails, image data optically scanned from documents, facsimile documents, photographs, accounting data, medical data, databases, and others. Non-standard forms include not only still images but also moving images. Still images include diagrams and illustrations.

[0010] In this embodiment, the OCR processing server 10 is a server that performs OCR processing on image data of a non-standard form and recognizes the characters written on the non-standard form. The OCR processing server 10 extracts candidate text data corresponding to specific items of the non-standard form from the image data of the non-standard form. In other words, the OCR processing server 10 obtains candidates (hereinafter referred to as "extraction candidates") to be extracted from the written information, which is the information written on the non-standard form, as written information corresponding to the items of the non-standard form. In the information processing system 1 shown in Figure 1, the OCR processing server 10 consists of a single server, but it may consist of multiple servers. The OCR processing server 10 may be configured as a so-called cloud server or as an on-premise server.

[0011] In this embodiment, the user terminal 30 is an information terminal device used by the user to confirm the results of OCR processing performed by the OCR processing server 10. The user terminal 30 presents the extraction candidates acquired by the OCR processing server 10 and accepts operations from the user to confirm or correct the results of the OCR processing. The user terminal 30 also displays the location in the non-standard form where the extraction candidates are written (hereinafter referred to as the "written location"). The user terminal 30 is equipped with a display unit 30A, which consists of a liquid crystal display panel or an organic EL (Electro Luminescence) display panel, for displaying images, text information, and the like to the user. Examples of user terminals 30 include computer devices, tablet-type information terminals, smartphones, and other information processing devices.

[0012] In this embodiment, the document management server 20 manages image data of documents to be processed by OCR. The document management server 20 inputs the managed image data to the OCR processing server 10. The image forming apparatus 40 in this embodiment has at least a scanning function to read a document and acquire image data of a document on paper. The image forming apparatus 40 inputs the acquired image data to the OCR processing server 10. The image forming apparatus 40 may also have a printer function, a copy function, a facsimile function, etc. In this embodiment, the document management server 20 and the image forming apparatus 40 are examples of input devices that input image data to be processed by OCR to the OCR processing server 10.

[0013] The business server 50 is responsible for performing specific tasks using the results of the OCR processing performed by the OCR processing server 10. Based on the results of the OCR processing, the business server 50 manages customer information and calculates expenses, among other things. In this embodiment, the business server 50 is an example of an output destination where the OCR processing server 10 outputs the results of OCR processing of image data of non-standard forms.

[0014] Here, we will describe the overview of the information processing system 1 of this embodiment. The OCR processing server 10 performs OCR processing on image data of unstructured forms input from the document management server 20, user terminal 30, image forming apparatus 40, etc., and outputs the results of the OCR processing to a predetermined output destination such as the business server 50. Furthermore, the OCR processing server 10 performs reading definition settings, output settings, and business check settings, as well as splitting image data of non-standard forms based on extraction candidates and accepting confirmation and correction of extraction candidates. In the reading definition settings, you can set the reading range, which is the area from which information from image data is read during OCR processing, for example, reading the item value from near the right of the item. In the output settings, you can set the file format and destination of the output data to be output. In the business check settings, you can set the format, such as the number of characters that can be entered into the output data.

[0015] When the user terminal 30 receives a command from the user to extract information corresponding to a specific item from image data of an unstructured form, the OCR processing server 10 is requested to perform the extraction, and the results of this extraction are displayed. The user terminal 30 displays candidate information corresponding to the specific item in the unstructured form on the display unit 30A. Specifically, for the candidate information corresponding to the specific item, the user terminal 30 displays on the display unit 30A the location of the text data that is the extracted information itself.

[0016] This allows the user to refer to the location of the suggested extraction candidate when correcting the extraction result by the OCR processing server 10 on the user terminal 30. This embodiment aims to improve user efficiency when correcting extraction results by displaying extraction candidates in association with the written information and its location before the extraction results by the OCR processing server 10 are finalized.

[0017] [Hardware configuration] Figure 2 shows an example configuration of the OCR processing server 10 and user terminal 30 according to this embodiment. Each of the OCR processing server 10 and user terminal 30 according to this embodiment includes a control unit 11 that controls the operation of the entire device, a secondary storage unit 12 that records information related to non-standard forms, and a communication unit 13 that transmits and receives information via a communication line 80 (see Figure 1). The control unit 11, the secondary storage unit 12, and the communication unit 13 are connected via buses and signal lines.

[0018] The control unit 11 includes a CPU 11a that controls the entire device as an example of a processor, RAM 11b used as working memory for the CPU 11a, and ROM 11c that stores programs executed by the CPU 11a. The control unit 11 also includes a rewritable non-volatile memory 11d that can retain data even if the power supply is interrupted, and an interface unit 11e that controls various parts such as the communication unit 13 connected to the control unit 11. The non-volatile memory 11d is composed of, for example, SRAM or flash memory backed up by a battery. Furthermore, each process in each device of this embodiment is executed when the control unit 11 reads the program stored in the secondary storage unit 12.

[0019] The secondary storage unit 12 is composed of, for example, a hard disk drive (HDD) or semiconductor memory. The secondary storage unit 12 varies depending on the product form of each device. For example, information about unstructured forms is stored in the secondary storage unit 12. In addition, the secondary storage unit 12 stores programs executed by the control unit 11 and information about extraction conditions for OCR processing.

[0020] In addition, the user terminal 30 includes an input device such as a keyboard or mouse, and a display unit 30A (not shown in Figure 2) which consists of an LCD display or the like. If the input device in the user terminal 30 is a touch panel, it may be provided as an integral part of the display unit 30A.

[0021] [Functional configuration of OCR processing server 10] Figure 3 shows an example of the functional configuration of the OCR processing server 10 according to this embodiment. Each function of the OCR processing server 10 shown in Figure 3 is mainly realized by the CPU 11a of the control unit 11. As shown in Figure 3, the OCR processing server 10 includes an unstructured form acquisition unit 101, a request information acquisition unit 102, an OCR processing unit 103, a priority assignment unit 104, an extraction candidate output unit 105, an extraction result correction unit 106, and an extraction result output unit 107. The OCR processing server 10 also includes an unstructured form database (DB) 108, an extraction candidate DB 109, and a learning DB 110.

[0022] The non-standard form acquisition unit 101 acquires image data of non-standard forms. The non-standard form acquisition unit 101 acquires image data of non-standard forms from the secondary storage unit 12 of its own device or from the communication unit 13 that receives image data of non-standard forms from other devices. Other devices include, for example, a document management server 20, a user terminal 30, and an image forming apparatus 40. The non-standard form document acquisition unit 101 stores the acquired image data of the non-standard form document in the non-standard form document DB 108.

[0023] The request information acquisition unit 102 acquires request information indicating the user's request via the user terminal 30 used by the user. Specifically, the request information acquisition unit 102 causes the user to input a request via the user terminal 30 and acquires the request information input by the user via the communication line 80. Note that the method by which the request information acquisition unit 102 acquires the request information is not limited to the above method.

[0024] The OCR processing unit 103 performs character recognition processing on the image data of the non-standard form document to extract the description information included in the non-standard form document. The OCR processing unit 103 extracts the requested description information based on the description content of the item for the description information requested by the user as an extraction candidate. In the present embodiment, the item itself described in the non-standard form document may be referred to as a "key", and the description information corresponding to the item may be referred to as a "value". In other words, the key is the item name, and the value is the item value.

[0025] As described above, the OCR processing unit 103 performs key-value extraction for extracting a key and a value by performing character recognition processing on the image data of the non-standard form document. For example, in a non-standard form document, characters representing the name corresponding to the value are often described around the key "Name". When the item "Name" is selected, the OCR processing unit 103 identifies the characters "Name" corresponding to the key from the image data of the non-standard form document, and extracts the description information corresponding to the value from the periphery of the location where the "Name" is identified. In addition, when the item "Name" is selected, the OCR processing unit 103 may also extract the description information corresponding to the value from the periphery of the location where "Name", "namae", "Name", etc. having the same meaning as "Name" are identified.

[0026] Furthermore, the OCR processing unit 103 may extract candidate values ​​from the written information corresponding to the location where the key has been identified, based on the extraction conditions set for the item. The extraction conditions set for an item are information that limits the type and format of the value corresponding to the key, and serve as hints for identifying candidate values ​​for extraction. Specifically, the extraction criteria are set so that the value content is, for example, a date, phone number, postal code, date of birth, numerical value, name, phonetic spelling of the name, etc. Other extraction conditions include setting the value format to include, for example, Western calendar year, Japanese calendar year, month and day, numbers greater than or equal to n / less than or equal to n, kanji, hiragana, katakana, alphabet, etc.

[0027] The priority assignment unit 104 assigns a priority to each of the extraction candidates based on the likelihood of the OCR processing. Specifically, the priority assignment unit 104 assigns a priority to the extraction candidates based on the achievement status of the extraction conditions set for the items. Alternatively, the priority assignment unit 104 may assign a priority to the extraction candidates based on the location where the key has been identified.

[0028] The extraction candidate output unit 105 outputs information about the extraction candidates extracted by the OCR processing unit 103 to the user terminal 30. Specifically, the extraction candidate output unit 105 outputs information related to the value of the extraction candidate, including the information to be written, the location to be written, and the priority, to the user terminal 30 via the communication unit 13.

[0029] The extraction result correction unit 106 corrects the extraction candidates corresponding to the extraction results based on the request information acquired by the request information acquisition unit 102. In other words, the extraction result correction unit 106 modifies the information regarding the priority 1 extraction candidates extracted by the OCR processing unit 103 based on the request information acquired by the request information acquisition unit 102. Specifically, the extraction result correction unit 106 makes at least one change to the extraction candidate, such as priority, description information, or description location. As a result, the corrected content is reflected in the extraction result by the OCR processing unit 103.

[0030] The extraction result output unit 107 manages the extraction results from the OCR processing unit 103. The extraction result output unit 107 outputs the extraction results from the OCR processing unit 103 to the device's secondary storage unit 12, user terminal 30, or other devices. The extraction result output unit 107 outputs the extraction candidates output by the extraction candidate output unit 105 with priority 1, or the extraction candidates corrected by the extraction result correction unit 106, as the extraction results from the OCR processing unit 103. The extraction result output unit 107 stores the content output by the OCR processing unit 103 as the extraction result in the learning database 110. Furthermore, if the content output by the OCR processing unit 103 as the extraction result has changed from the content of the extraction candidate output by the extraction candidate output unit 105 with priority 1, the extraction result output unit 107 stores this change in the learning database 110.

[0031] The non-standard form DB108 stores and accumulates image data of non-standard forms acquired by the non-standard form acquisition unit 101. The extraction candidate DB 109 stores the extraction candidates extracted by the OCR processing unit 103. Specifically, the extraction candidate DB 109 stores and accumulates the keys and values ​​extracted by the OCR process, associating them with each other. The learning DB 110 stores and accumulates the extraction results output by the OCR processing unit 103 from the extraction result output unit 107. Specifically, when a correction is made to the extraction result, the learning DB 110 stores and accumulates information such as the value, the location where the key was identified, the positional relationship between the key and the value, and the extraction conditions to which the value corresponds among the extraction conditions set for the item.

[0032] [Functional configuration of user terminal 30] Figure 4 shows an example of the functional configuration of the user terminal 30 according to this embodiment. Each function of the user terminal 30 shown in Figure 4 is mainly realized by the CPU 11a of the control unit 11. As shown in Figure 4, the user terminal 30 includes an operation determination unit 301, a display control unit 302, an extraction candidate management unit 303, and a communication control unit 304.

[0033] The operation determination unit 301 determines whether or not a user operation has been performed on the user terminal 30 via an input device or the like. This user operation includes, for example, an operation to instruct the user to select items for extracting information from image data of an unstructured form, an operation to instruct the user to set extraction conditions for each item, and an operation to instruct the user to select an extraction result.

[0034] The display control unit 302 controls the display mode and other aspects of the display unit 30A. Specifically, the display control unit 302 controls the display mode of the operation screen that presents the extraction candidates displayed on the display unit 30A. The display control unit 302 controls the display mode of the operation screen, for example, to indicate the location of the extraction candidates in the non-standard form.

[0035] The extraction candidate management unit 303 manages information related to the presentation of extraction candidates. In other words, the extraction candidate management unit 303 manages the information used by the display control unit 302. The information managed by the extraction candidate management unit 303 includes, for example, information relating the description information of the extraction candidates to the location where it is described, information indicating the priority assigned to each extraction candidate, and details of the achievement status of the extraction conditions for each extraction candidate.

[0036] The communication control unit 304 processes information transmitted by the communication unit 13 and processes information received. The communication control unit 304, via the communication unit 13, requests the OCR processing server 10 to extract the corresponding information for each item, or requests corrections to the extraction results. Furthermore, the communication control unit 304, via the communication unit 13, extracts information from the OCR processing server 10 to present extraction candidates on the operation screen, and also extracts information of the extraction results. In other words, the communication control unit 304 processes the acquired information and makes decisions.

[0037] [Process to confirm the extraction candidates] This section describes an example of a process where the user is allowed to confirm the extraction candidates before the extraction results are output. Figure 5 is a flowchart showing the process flow in which the OCR processing server 10 according to this embodiment allows the user to confirm the extracted candidates. Figure 5 shows an example where the OCR processing server 10 extracts candidate data from image data of an unstructured form, and the user is allowed to confirm the results via the user terminal 30.

[0038] First, the non-standard form acquisition unit 101 of the OCR processing server 10 acquires image data of the non-standard form (S1). Furthermore, the unstructured forms whose image data has been acquired by the unstructured form acquisition unit 101 are displayed as preview images on the display unit 30A of the user terminal 30. In addition, the user terminal 30 receives instructions from the user to set the items to be extracted and the extraction conditions for the OCR processing of the image data of the unstructured forms acquired by the unstructured form acquisition unit 101.

[0039] Here, we will explain the screen that accepts settings from the user for the items to be extracted and the extraction conditions for OCR processing of image data of non-standard forms. Figure 6 shows an example of the display on the display unit 30A of the user terminal 30 according to this embodiment, and shows an example of a setting screen that accepts the setting of items to be extracted and extraction conditions related to OCR processing.

[0040] The display unit 30A of the user terminal 30 displays an operation screen 300 that accepts user input to instruct the OCR processing server 10 on OCR processing. The operation screen 300 includes a preview screen 310 that displays a preview image 311 of a non-standard form, and a settings screen 320 that accepts settings for items to be extracted by OCR processing. In the example shown in Figure 6, the setting screen 320 displays the item setting image 321 and the condition setting image 322.

[0041] Item setting image 321 is an image that accepts the user's setting of the item name to be extracted. In the example shown in Figure 6, the item settings image 321 shows that the following items are set as targets for extraction: "Application Date", "Name: Phonetic Spelling", "Name", "Date of Birth", "Telephone Number", and "Reception Date". The item names to be extracted may be selected from a pre-configured list of item name candidates, or text information may be entered. Furthermore, as mentioned above, settings may be accepted to extract keys that have similar meanings but different displays for the same item name.

[0042] Condition setting image 322 is an image that accepts the user's settings for filtering items. For example, if condition setting image 322 is selected by the user, a screen will be displayed to accept details of the filtering conditions for the items, and settings will be accepted to limit the type and format of the value corresponding to the key.

[0043] Next, the request information acquisition unit 102 of the OCR processing server 10 acquires from the user terminal 30 the items to be extracted and the extraction conditions for the OCR processing of the image data of the non-standard form acquired by the non-standard form acquisition unit 101 (S2). In other words, the request information acquisition unit 102 acquires request information from the user terminal 30 that indicates the items to be extracted and the extraction conditions for the OCR processing. Then, the OCR processing unit 103 of the OCR processing server 10 performs key-value extraction on the image data of the non-standard form based on the request information acquired by the request information acquisition unit 102 (S3).

[0044] Specifically, for example, the user terminal 30 requests that text data with the item name "Application Date" as the key be extracted as value, with the date being on or after the 15th of each month, or in Western calendar format. In this case, the OCR processing unit 103 uses the information surrounding the location where the item name "Application Date" is identified, the information indicating a date on or after the 15th, and the information in Western calendar format as values ​​to extract candidates.

[0045] Next, the priority assignment unit 104 of the OCR processing server 10 assigns a priority to the extraction candidates, which are the result of key-value extraction by the OCR processing unit 103 (S4). Specifically, the priority assignment unit 104 assigns a priority to the extraction candidates based on the achievement status of the request information transmitted from the user terminal 30.

[0046] Here, we will explain the priority assigned by the priority assignment unit 104 to the extraction candidates. Figure 7 shows an example of assigning priorities to extraction candidates. Figure 7 shows an example where the item name "Application Date" is used as the key, and the extraction condition set for the item requires that the value be a date on or after the 15th of each month, or information written in the Gregorian calendar.

[0047] In the example shown in Figure 7, each extraction candidate is associated with a "priority," a "string" extracted as a value, "coordinates" indicating the location on the non-standard form, and a "condition fulfillment status" indicating whether or not the required information is met. In the "Condition Fulfillment Status" section, the "Key" field will display "○" if the value is located near the location where the key characters are identified, and "×" if the value is not located near the location where the key characters are identified. Additionally, the "Notation Format" field will indicate whether the date is in Western calendar format, and the "Date Period" field will indicate whether the date falls on or after the 15th of each month. In this example, priority is assigned to extraction candidates that satisfy more of the conditions included in the request information, rather than to extraction candidates that satisfy fewer conditions. Additionally, priority is assigned to extraction candidates whose value is located near the location where the key characters are identified, rather than to extraction candidates whose value is not located near the location where the key characters are identified.

[0048] For example, the string "November 8, 2021" is assigned a "priority 1" because it fulfills the most conditions included in the request information and the key characters are located near the identified location. For example, the string "July 19, 1987" fulfills the same number of conditions as the string "November 8, 2021," but the key characters are not located near the identified location. Therefore, the string "July 19, 1987" is assigned a priority of "2" and takes precedence over the string "November 8, 2021."

[0049] Furthermore, for example, the strings "7-31" and "04992-1-6232 2021 / 5 / 15" do not meet the conditions included in the request information, and therefore meet fewer conditions than the strings "November 8, 2021" and "July 19, 1987". For this reason, the strings "7-31" and "04992-1-6232 2021 / 5 / 15" are assigned "Priority 3" and "Priority 4," respectively, from information that the user has not set as request information.

[0050] Next, the extraction candidate output unit 105 of the OCR processing server 10 outputs information about the extraction candidates to the user terminal 30 (S5). Furthermore, the information regarding the extraction candidates output by the extraction candidate output unit 105 is displayed on the display unit 30A of the user terminal 30 for the user to confirm. The user terminal 30 also presents the extraction candidates and accepts instructions from the user to confirm the extraction results.

[0051] Here, we will explain how to display information about the extraction candidates to allow the user to confirm it. Figure 8 shows an example of how extraction candidates are displayed according to this embodiment, and illustrates an example of a detailed screen for presenting detailed information about the extraction candidates to the user. Figure 8 shows an example where the extraction candidates shown in Figure 7 are extracted.

[0052] In the example shown in Figure 8, the operation screen 300 includes a preview screen 310 that displays a preview image 311 of a non-standard form, as well as a detail screen 330 that presents the user with detailed information about the extraction candidates. The preview image 311 displays a highlight image 312. The detail screen 330 displays a partial image 331, an output field 332, and a confirmation button image 333.

[0053] The highlighting image 312 is an image that highlights the location of the extraction candidate. In the display example shown in Figure 8, the highlighting image 312 is placed around the area from which one extraction candidate has been extracted in the preview image 311 of the non-standard form. Note that the highlighted image 312 is not limited to an image that surrounds the location of the extraction candidate; it may also be an image containing an arrow or other image, or an image containing a text message such as a pop-up.

[0054] In the example shown in Figure 8, the details screen 330 displays information about the extraction candidate that best meets the user's requested conditions and has been assigned a "priority of 1". The details screen 330 is a screen that presents the user with detailed information about the extraction candidates and accepts instructions from the user to confirm the extraction results from the OCR processing unit 103. Furthermore, the user may be given instructions to change the extraction candidates whose information is displayed on the details screen 330, based on operations such as selecting a highlighted image 312 on the preview image 311.

[0055] Partial image 331 is an image of the portion of a non-standard form that contains the extraction candidate information. Output field 332 displays the text data that will be output as a value. Output field 332 may accept input from the user and allow correction of the text data. The confirmation button image 333 is a button image that receives an instruction from the user to confirm the extraction candidates displayed on the details screen 330 as extraction results by the OCR processing unit 103. When the confirmation button image 333 is selected by the user, the user terminal 30 requests the OCR processing server 10 to confirm the extraction results.

[0056] This example assumes that when a candidate with "priority 1" is displayed on the details screen 330, the user selects the confirmation button image 333. The request information acquisition unit 102 of the OCR processing server 10 acquires a request from the user terminal 30 for extraction candidates with priority 1 as the extraction result (S6). In other words, the request information acquisition unit 102 acquires request information from the user terminal 30 indicating a request for extraction candidates with priority 1 as the extraction result. Then, the extraction result output unit 107 of the OCR processing server 10 outputs an extraction candidate with priority 1 as the extraction result based on the request information acquired by the request information acquisition unit 102 (S7), and the process ends.

[0057] By the way, the display of extraction candidates in this embodiment is not limited to the configuration shown in Figure 8, which displays information about the extraction candidates based on priority or uniformly displays the location of the extraction candidates. Other examples of how to display extraction candidates will be described below.

[0058] Figure 9 shows another example of how to display the extraction candidates according to this embodiment, and shows an example where the priority assigned to each extraction candidate is displayed. Figure 9 shows an example where the priority assigned to each extraction candidate is reflected on the preview image 311 of the non-standard form.

[0059] In the example shown in Figure 9, in addition to the highlighting image 312 that emphasizes the location of the extraction candidate, a badge image 313 is displayed that notifies the user of the priority assigned to the extraction candidate using a small badge-like mark. The badge image 313 is displayed in association with the location where the extraction candidate is described. This aims to improve the user's workflow when selecting an extraction candidate from the location on the preview image 311. Note that the badge image 313 is not limited to being displayed on the preview image 311. The badge image 313 may also be displayed on the details screen 330. This allows the user to check information about the extraction candidates on the details screen 330 after referring to the priority.

[0060] Badge image 313 is an image that informs the user of the priority assigned to each extraction candidate. In other words, badge image 313 is an image that informs the user of the priority itself. Furthermore, the badge image 313 is not limited to the numerical representation shown in Figure 9, but may also be represented using pictograms.

[0061] Next, we will explain how to display the extraction candidates on the preview image 311 of the non-standard form, reflecting the reasons for the priority assigned to each extraction candidate. Figure 10 shows another example of how the extraction candidates are displayed according to this embodiment, and shows an example where the reason for priority is displayed.

[0062] In the example shown in Figure 10, in addition to the highlighted image 312 that emphasizes the location of the extraction candidate, a reference image 314 is displayed for the user to refer to, explaining the reason for the priority assigned to the extraction candidate. Reference image 314 is displayed in association with the location where the extraction candidates are described. Reference image 314 may also be displayed on the details screen 330.

[0063] Reference image 314 is an image that informs the user of the reasons for the priority assigned to each extraction candidate. In other words, reference image 314 is an image that informs the user of the details of the priority. Reference image 314 is not limited to showing conditions that the extraction candidates shown in Figure 10 have not met, but may also show conditions that the extraction candidates have met. Furthermore, reference image 314 is not limited to showing details of the condition's achievement status, such as "Priority 1: Date period is different," but may also show an overview of the condition's achievement status, such as "Priority 1: One of the conditions has not been met." Note that reference image 314 is not limited to images such as pop-ups containing text messages, but may also be displayed using pictograms.

[0064] Furthermore, the display mode of the highlighted image 312, which emphasizes the location of the extraction candidate, may be varied according to the priority assigned to each extraction candidate. Figure 11 shows another example of the display of the extraction candidates according to this embodiment, illustrating an example where the display mode of the highlighted image 312 is varied according to priority.

[0065] In the example shown in Figure 11, the highlighted image 312 is displayed according to the priority of the extraction candidates corresponding to the location indicated by the highlighted image 312. The display mode of the highlighted image 312 is made different between the extraction candidate with priority 1 and the other extraction candidates. Specifically, among the multiple extraction candidates, the display mode of the highlighted image 312 indicating the location of the extraction candidate is made different between the candidate that best fulfills the extraction conditions and the other candidates.

[0066] In the example shown in Figure 11, the highlighted image 312 corresponding to the extraction candidate with priority 1 is displayed as a solid line, while the highlighted images 312 corresponding to the other extraction candidates are displayed as dashed lines. However, this is not limited to this; for example, the display mode of the highlighted image 312 may differ between the extraction candidate with priority 1 and the other extraction candidates, such as blinking and solid, or red and blue, or the shape of the image may also differ. Furthermore, the display mode of the highlighted image 312 may be different for each priority level, rather than being separated for priority 1 extraction candidates and other extraction candidates.

[0067] Additionally, warnings may be displayed for extraction candidates that meet specific conditions. Figure 12 shows another example of the display of extraction candidates according to this embodiment, illustrating an example where a warning is displayed depending on the status of achieving the extraction conditions.

[0068] In the example shown in Figure 12, in addition to the highlighting image 312 that emphasizes the location of the extraction candidate, a warning image 315 is displayed to alert the user about extraction candidates that meet specific conditions. Warning image 315 is displayed in association with the location where the extraction candidate is described. Warning image 315 may also be displayed on the details screen 330.

[0069] Warning image 315 is an image used to inform the user about extraction candidates that meet specific conditions. Specifically, warning image 315 is an image used to inform the user about extraction candidates that have fewer conditions met than the threshold, or extraction candidates that do not meet the required conditions. Note that the warning image 315 is not limited to pictograms; it may also be an image containing text messages such as pop-ups.

[0070] In the example described above, when displaying the extraction candidates according to this embodiment, in addition to the highlighted image 312, one of the badge image 313, reference image 314, or warning image 315 is displayed, but the system is not limited to this. In addition to the highlighted image 312, one or all of the badge image 313, reference image 314, or warning image 315 may be displayed in combination.

[0071] Figure 13 shows another example of how the extraction candidates are displayed according to this embodiment, where (A) shows the state in which the extraction candidates satisfy certain conditions, and (B) shows the state in which the extraction candidates have not satisfied certain conditions. Figure 13 shows an example where information about extraction candidates with priority 4 is displayed on the details screen 330.

[0072] In the example shown in Figure 13(A), the detailed screen 330, which displays information about extraction candidates that do not meet all of the extraction conditions, shows both the reference image 314 and the warning image 315. The extraction candidate corresponding to the item "Application Date" displayed on detail screen 330 is a number, but it does not satisfy the extraction condition set for the item, which is "the value must be the information written in Western calendar format."

[0073] Let's assume that the user corrects the text data of the extraction candidates shown in Figure 13(A) in the output field 332 as shown in Figure 13(B). In the example shown in Figure 13(B), the extraction candidate corresponding to the item "Application Date" displayed on the details screen 330 satisfies the extraction condition set for the item, "The value should be the information written in Western calendar format." In this case, the reference image 314, which shows the condition that has not been met, and the warning image 315, which is displayed when a specific condition is met, are not displayed.

[0074] Furthermore, while only one extraction candidate was shown on detail screen 330, this is not limited to that; multiple extraction candidates may also be shown on detail screen 330. Figure 14 shows another example of the display of extraction candidates according to this embodiment, and shows an example of a detailed screen 330 that displays information about multiple extraction candidates to the user. In the example shown in Figure 14, a cursor 200 is displayed on the operation screen 300, indicating a location or image on the screen. The cursor 200 moves based on user input via the input unit and is displayed to point to a location or image on the screen specified by the user.

[0075] In the example shown in Figure 14, the detail screen 330 displays a partial image 331, an output field 332, and a confirmation button image 333 for each extraction candidate. For example, on the details screen 330, multiple extraction candidates will be displayed in an order according to their assigned priority. In addition, for example, on the details screen 330, multiple extraction candidates may be displayed in the order corresponding to the location indicated on the preview image 311.

[0076] Furthermore, in the display example shown in Figure 14, when the cursor 200 is positioned in the area of ​​the detail screen 330 that displays information about one of the extraction candidates, a highlight image 312 indicating the area from which one of the extraction candidates has been extracted is displayed. In other words, when the user specifies the area that displays information about one of the extraction candidates, a highlight image 312 indicating the area from which one of the extraction candidates has been extracted is displayed. This aims to reduce the clutter on the preview image 311 caused by the display of extraction candidates, compared to when all the extraction candidate displays are shown.

[0077] Furthermore, when displaying the extraction candidates, the user may be instructed on the details screen 330 whether to display only one extraction candidate or multiple extraction candidates. Figure 15 shows another example of the display of extraction candidates according to this embodiment, illustrating an example where the user provides instructions to display information about multiple extraction candidates on the detail screen 330.

[0078] In the example shown in Figure 15, the detail screen 330 displays a partial image 331, an output field 332, a confirmation button image 333, and a candidate button image 334. The candidate button image 334 is an image that receives instructions from the user to display multiple extraction candidates on the details screen 330. When the candidate button image 334 is selected by the user, multiple extraction candidates are displayed on the details screen 330, as shown in Figure 14. Specifically, for example, in the detail screen 330 where information about extraction candidates with priority 1 is displayed, if the candidate button image 334 is selected by the user, information about extraction candidates with priority 2 and above will also be displayed on the detail screen 330.

[0079] [Process to correct the extracted candidates] This section describes an example of how to handle situations where a user requests correction of the extraction candidates. Figure 16 is a flowchart showing the process flow in which the user terminal 30 according to this embodiment requests correction of the extraction candidates. Figure 16 illustrates a case where the user terminal 30 does not use the priority 1 extraction candidates extracted from the image data of the non-standard form as the extraction result, but instead instructs the user to make corrections.

[0080] First, the display control unit 302 of the user terminal 30 displays the extraction candidates with priority 1 on the display unit 30A (S11). Specifically, the display control unit 302 displays the text data and location of the value of the extraction candidate on the display unit 30A for the extraction candidate with priority 1.

[0081] Then, the operation determination unit 301 of the user terminal 30 determines whether or not the text data of the priority 1 extraction candidate has been corrected (S12). In other words, it determines whether or not a user operation has been performed to instruct the correction of the text data of the priority 1 extraction candidate. Specifically, when the operation determination unit 301 receives input from the user in the output field 332 of the detailed screen 330 shown in Figure 8, it determines that a user operation has been performed to instruct correction of the text data of the extraction candidate with priority 1.

[0082] If the text data of a priority 1 extraction candidate is corrected (YES in S12), the operation determination unit 301 of the user terminal 30 determines that it has received a user operation to confirm the corrected priority 1 extraction candidate as the extraction result (S13). Specifically, the operation determination unit 301 determines that if the user selects the confirmation button image 333 on the detailed screen 330 shown in Figure 8, it will confirm that the corrected priority 1 extraction candidate will be selected as the extraction result.

[0083] The communication control unit 304 of the user terminal 30 requests the OCR processing server 10 to replace the text data of the priority 1 extraction candidate with the corrected text data (S14), and then terminates the process. In other words, the communication control unit 304 requests the OCR processing server 10 to use the priority 1 extraction candidate with corrected text data as the extraction result.

[0084] In S12, if the text data of the extraction candidate with priority 1 has not been corrected (NO in S12), the operation determination unit 301 of the user terminal 30 determines that it has received a user operation instructing the user to select a location of an extraction candidate other than the extraction candidate with priority 1 (S15). Specifically, the operation determination unit 301 determines that if the user selects a location for another extraction candidate on the preview image 311 of the non-standard form shown in Figure 8, that other extraction candidate has been selected as an extraction candidate with priority 1.

[0085] Next, the operation determination unit 301 of the user terminal 30 determines that it has received a user operation to confirm the other selected extraction candidates as extraction results (S16). The communication control unit 304 of the user terminal 30 requests the OCR processing server 10 to replace the extraction candidate with priority 1 with another selected extraction candidate (S17), and then terminates the process. In other words, the communication control unit 304 requests the OCR processing server 10 to change the priority of the other selected extraction candidate to 1 and to use it as the extraction result.

[0086] [Process for learning the extraction results] This section explains an example of processing when presenting extraction candidates and learning from the confirmed extraction results. Figure 17 is a flowchart showing the process flow in which the OCR processing server 10 according to this embodiment learns the extraction results. Figure 17 shows an example where the extraction results are finalized after corrections have been made to the extraction candidates with priority 1.

[0087] First, the request information acquisition unit 102 of the OCR processing server 10 acquires request information from the user terminal 30 to confirm the extraction result for the OCR processing of the image data of the non-standard form acquired by the non-standard form acquisition unit 101 (S21). If the extraction candidate output unit 105 obtains request information to confirm the extraction result using an extraction candidate output with priority 1, the extraction result output unit 107 outputs the extraction candidate with priority 1 as the extraction result of the OCR processing unit 103. On the other hand, if the extraction result correction unit 106 obtains request information to confirm the extraction result using an extraction candidate that has been corrected, the extraction result output unit 107 outputs the corrected extraction candidate as the extraction result of the OCR processing unit 103.

[0088] Then, the extraction result output unit 107 of the OCR processing server 10 determines whether or not there are any changes from the location of the priority 1 extraction candidate in the content output as the extraction result by the OCR processing unit 103 (S22). Changes to the location of priority 1 extraction candidates include, for example, when the area for key-value extraction of the candidates is changed while the information of priority 1 extraction candidates remains the same. Other cases where changes to the location of priority 1 extraction candidates include, for example, when a different extraction candidate is selected than the priority 1 extraction candidate, or when something other than an extraction candidate is used as the extraction result.

[0089] If there are any changes in the content output as an extraction result by the OCR processing unit 103 from the location of the extraction candidate with priority 1 (YES in S22), the extraction result output unit 107 of the OCR processing server 10 registers the details of these changes as learning data for the reading position of the non-standard form (S23). In other words, the extraction result output unit 107 stores the changed location in the learning DB 110 as learning data for the area in which extraction candidates corresponding to the items are extracted as key values. On the other hand, if there are no changes to the location of the priority 1 extraction candidate in the output of the extraction result by the OCR processing unit 103 (NO in S22), the training data for the region from which the extraction candidate is extracted as a key value will not be registered.

[0090] Next, the extraction result output unit 107 of the OCR processing server 10 determines whether or not there are any changes from the information described in the extraction candidates to the content output by the OCR processing unit 103 (S24). Examples of situations where the information in the extracted candidates changes include cases where user input is received in output field 332 (see Figure 8) and the text data is modified.

[0091] If there are any changes in the information output as an extraction result by the OCR processing unit 103 from the information of the extraction candidate (YES in S24), the extraction result output unit 107 of the OCR processing server 10 registers the details of these changes as learning data for character recognition (S25) and terminates the process. In other words, the extraction result output unit 107 stores the modified information in the learning DB 110 as learning data for character recognition of non-standard forms and terminates the process. On the other hand, if the content output as an extraction result by the OCR processing unit 103 is unchanged from the information of the extraction candidate (NO in S24), the character recognition training data is not registered and the process ends.

[0092] This aims to improve the accuracy of key-value extraction when corrections are made to extraction candidates with priority 1. It also aims to improve the accuracy of character recognition processing when corrections are made to extraction candidates with priority 1.

[0093] Furthermore, although not explained above, in this embodiment, even if the corrected text data or other selected extraction candidates do not satisfy all of the extraction conditions, the corrected extraction candidates may be output as extraction results by the OCR processing unit 103. For example, training data can be registered in such a way that weights are added to the extraction conditions among the extraction conditions set for an item that match the corrected extraction candidate. This allows new extraction candidates to be obtained from a new non-standard form based on the item, the extraction conditions, and the weights of the extraction conditions.

[0094] [Handling when no extraction candidates exist] Next, we will explain an example of how to handle the case where no extraction candidates are presented on the user terminal 30. Figure 18 is a flowchart showing the process flow in which the user terminal 30, according to the embodiment, inputs the extraction results. Figure 18 shows an example where no candidate items were extracted from the image data of a non-standard form.

[0095] First, the display control unit 302 of the user terminal 30 notifies the user via the display unit 30A that there are no extraction candidates corresponding to the item (S31). Specifically, the display control unit 302 displays a text message on the display unit 30A indicating, for example, that no extraction candidates exist. Note that notification that no extraction candidates exist for an item is not limited to images or other visual representations; sound, light, or other visual means may also be used.

[0096] Here, we will explain how to display a notification to the user when there are no extraction candidates corresponding to an item. Figure 19 shows another example of the display on the display unit 30A of the user terminal 30 according to this embodiment, and shows an example of a notification image that informs the user that there are no extraction candidates corresponding to an item. Figure 19 shows an example where key-value extraction was performed on the item "Reception Date," but no extraction candidates were obtained.

[0097] In the example shown in Figure 19, highlighting images such as 312 (see Figure 8) that emphasize the location of the extraction candidates are not displayed; instead, a notification image 350 is displayed informing the user that there are no extraction candidates corresponding to the item. Notification image 350 is an image used to inform the user that there are no extraction candidates corresponding to the item the user has selected. Specifically, notification image 350 is an image that uses a text message to inform the user that there are no extraction candidates corresponding to the item the user has selected and the reason why. Furthermore, the notification image 350 may include a text message prompting user action, as shown in Figure 19.

[0098] In the example shown in Figure 19, since there are no extraction candidates corresponding to the item selected by the user, the text message "Not found on the report" is displayed in the area of ​​partial image 331. Also, since there is no text data corresponding to the value, the output field 332 is blank. The notification image 350 is displayed in association with the details screen 330. Furthermore, the notification image 350 is displayed using a combination of pictograms, as shown in Figure 19. The notification image 350 may, for example, be displayed near the center of the operation screen 300, or it may be displayed using only pictograms.

[0099] Then, the operation determination unit 301 of the user terminal 30 determines that it has received a user operation instructing the confirmation of the extraction result (S32). Specifically, the operation determination unit 301 determines that it has received a user operation instructing the user to confirm the extraction results when the user selects the confirmation button image 333 on the detailed screen 330 shown in Figure 19. Based on a user operation instructing the user to confirm the extraction results, the display control unit 302 of the user terminal 30 displays a confirmation screen 370 (described later) on the display unit 30A for the user to confirm the extraction results (S33).

[0100] Here, we will explain how to display the extraction results for the user to confirm. Figure 20 shows an example of the confirmation screen 370 according to this embodiment. Figure 20 shows an example where the target item for key-value extraction is "Reception Date," and the extraction conditions are set as "It must be a date" and "It must be in Western calendar format."

[0101] In the example shown in Figure 19, if the user selects the confirmation button image 333 in accordance with the text message in the notification image 350, a confirmation screen 370 will be displayed to allow the user to confirm the extraction results, as shown in Figure 20. The confirmation screen 370 accepts instructions from the user to either finalize the extraction results by leaving the values ​​corresponding to the items blank, or to finalize the extraction results by filling in the values. The confirmation screen 370 displays an input field 371, a cancel button image 372, a verify button image 373, and a save button image 374.

[0102] Input field 371 displays a text message prompting the user to enter a value. Input field 371 also displays the extraction conditions set for the target items for key-value extraction. Input field 371 may accept input from the user and be populated with value text data. The cancel button image 372 is a button image that accepts user input to cancel the confirmation of the extraction results. If the cancel button image 372 is selected by the user, the confirmation screen 370 will be hidden.

[0103] The verification button image 373 is a button image that accepts user input to verify the location of text data entered in input field 371 on image data of an unstructured form. In other words, the verification button image 373 is a button image that accepts user input to detect the location of text data that is around a location where a key can be identified, based on the value information entered. Image 374 of the save button is a button image that accepts instructions from the user to save the item with the corresponding value left blank.

[0104] Then, the operation determination unit 301 of the user terminal 30 determines whether or not to verify the written area (S34). In other words, the operation determination unit 301 determines whether or not a user operation has been performed that instructs the verification of the written area. Specifically, the operation determination unit 301 determines that if the user selects the verification button image 373 on the confirmation screen 370 shown in Figure 20, with text data entered in the input field 371, then the location of the text should be verified.

[0105] If the location of the entry is to be verified (YES in S34), the display control unit 302 of the user terminal 30 displays the result of the verification of the location of the entry on the display unit 30A (S35). Specifically, the display control unit 302 displays an image of the portion of the non-standard form image that was detected as containing text data entered in the input field 371, within the area of ​​the partial image 331 shown in Figure 19. In addition, the text data entered in the input field 371 is displayed in the output field 332 (see Figure 19). If the text data entered in input field 371 cannot be found, the area of ​​partial image 331 may be grayed out, or a text message such as "Not found" may be displayed in the area of ​​partial image 331.

[0106] Next, the operation determination unit 301 of the user terminal 30 determines whether or not to set the location for writing (S36). In other words, the operation determination unit 301 determines whether or not a user operation has been performed to instruct the setting of the location for writing. Specifically, the operation determination unit 301 determines that if the user selects the confirmation button image 333 while a portion of the image of the non-standard form is displayed in the area of ​​the partial image 331 shown in Figure 19, then the entry location should be set. On the other hand, if the user selects the confirmation button image 333 while a portion of the image of the non-standard form is not displayed in the area of ​​the partial image 331, then the operation determination unit 301 determines that the entry location should not be set. In the confirmation screen 370 shown in Figure 20, if the verification button image 373 is selected while text data has been entered in the input field 371, it is determined that a user operation has been performed to instruct the user to verify the entered data.

[0107] If a location for the entry is set (YES in S36), the communication control unit 304 of the user terminal 30 requests the OCR processing server 10 to use the input information and the location for the entry as the extraction result corresponding to the item (S37), and the process ends. If a detected location for the entry is set, this location is registered as training data for the area where extraction candidates corresponding to the item are extracted as key-value pairs. On the other hand, if no location is specified (NO in S36), the communication control unit 304 of the user terminal 30 requests the OCR processing server 10 to extract only the entered information corresponding to the item (S38), and the process ends.

[0108] Furthermore, this example assumes a scenario where, in a case where the location of the entry is not verified, the user instructs the system to save the corresponding value as blank. If the location is not verified in S34 (NO in S34), the operation determination unit 301 of the user terminal 30 determines that it has received a user operation to instruct saving (S39). Specifically, the operation determination unit 301 determines that, on the confirmation screen 370 shown in Figure 20, if the user selects the save button image 374 while the input field 371 is blank, the unit determines that it has received an instruction from the user to save the item with the corresponding value blank. Then, the communication control unit 304 of the user terminal 30 requests the OCR processing server 10 to produce an extraction result in which there is no recorded information or recorded location (S40), and the process ends.

[0109] <Variation> In the example described above, the operation screen 300 displays extraction candidates corresponding to one of the items selected by the user, but this is not limited to this. For example, if the user corrects an extraction candidate corresponding to one item, extraction candidates corresponding to related items associated with that item may be displayed. Figure 21 is a diagram showing another example of the display of extraction candidates according to this embodiment, and shows an example in which extraction candidates corresponding to related items of the corrected extraction candidate items are displayed.

[0110] The example in Figure 21 shows a case where corrections have been made to the extraction candidates for the item "Application Date," and the item "Date of Birth" has been identified as a related item to the item "Application Date." These "related items" are, for example, items that share the same extraction conditions set for each item, or items that share the same classification. Specifically, the corrected item "Application Date" and the related item "Date of Birth" both represent dates, and the type of value corresponding to the key is the same. Furthermore, in the example shown in Figure 21, the highlighting image 312 indicating the location of the item "Application Date" and the highlighting image 312 indicating the location of the item "Date of Birth" are displayed in different ways.

[0111] In each embodiment, the program executed by the CPU 11a, which is an example of a processor, is provided to the control unit 11 while stored on a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory. Alternatively, the program executed by the CPU 11a may be downloaded to the OCR processing server 10 and the user terminal 30 using communication means such as the internet.

[0112] In this embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs: Graphics Processing Units, ASICs: Application Specific Integrated Circuits, FPGAs: Field Programmable Gate Arrays, programmable logic devices, etc.). Furthermore, the operation of the processor may not be performed by a single processor, but may be performed by multiple processors located in physically separate locations working together. Also, the order of the processor's operations is not limited to the order described in this embodiment, and may be changed.

[0113] The processing performed by the information processing system in each embodiment is provided, for example, as a program such as application software. This program can be provided not only by means of communication, but also by being stored on a recording medium such as a CD-ROM.

[0114] (Note) (((1))) An information processing system comprising one or more processors, The one or more processors described above are: From the information recorded in the non-standard form, we obtain candidate extracts that correspond to the items in the non-standard form. The section of the aforementioned non-standard form in which the extraction candidates are described is displayed on the display unit. Information processing system. (((2))) The one or more processors described above are: Based on the content of the items corresponding to the information to be extracted from the aforementioned non-standard form, the extraction candidates are obtained. The information processing system described in (((1))). (((3))) The one or more processors described above are: From the information described in the aforementioned non-standard form, the extraction candidates are obtained based on the items and the extraction conditions set for those items. Based on the status of fulfillment of the extraction conditions, the corresponding section of the extraction candidate is displayed on the display unit. The information processing system described in (((1))) or (((2))). (((4))) The one or more processors described above are: Along with the location of the extraction candidate, the priority assigned to the extraction candidate based on the achievement status is displayed on the display unit. The information processing system described in (((3))). (((5))) The one or more processors described above are: Along with the location of the extraction candidate, the reason for the priority assigned to the extraction candidate based on the achievement status is displayed on the display unit. The information processing system described in (((3))). (((6))) The one or more processors described above are: From the information described in the non-standard form, a plurality of extraction candidates are obtained based on the items and the extraction conditions set for those items. Among the multiple candidates for extraction, the display manner of the described portion of the extraction candidate is made different between the candidate that best fulfills the extraction conditions and the other candidates. The information processing system described in (((3))). (((7))) The one or more processors described above are: The image of the non-standard form displays the corresponding location of the extraction candidate. An information processing system as described in any of (((1))) through (((6))). (((8))) The one or more processors described above are: From the information described in the aforementioned non-standard form, multiple extraction candidates are obtained, The portion of the extraction candidate specified by the user from among the multiple extraction candidates is displayed on the image of the non-standard form. The information processing system described in (((7))). (((9))) The one or more processors described above are: When the user selects the extraction candidate, the selected extraction candidate is used as the result of extracting the information corresponding to the item. An information processing system as described in any of (((1))) through (((8))). (((10))) The one or more processors described above are: The selected location of the extraction candidate is learned as the location of the description information corresponding to the item. The information processing system described in (((9))). (((11))) The one or more processors described above are: When the user inputs the information corresponding to the aforementioned item, the input information is treated as the result of extracting the information corresponding to that item. The image of the non-standard form is used to detect the location where the entered information is recorded. An information processing system as described in any of (((1))) through (((8))). (((12))) The one or more processors described above are: The detected location is learned as the location of the information corresponding to the item. The information processing system described in (((11))). (((13))) The one or more processors described above are: Based on the aforementioned items and the extraction conditions set for those items, the extraction candidates are obtained. Based on correction instructions from the user, the extracted candidates are corrected. Even if the corrected extraction candidate does not satisfy all of the extraction conditions, the corrected extraction candidate shall be considered as the result of extracting the information corresponding to the item. An information processing system as described in any of (((1))) through (((8))). (((14))) The one or more processors described above are: The display unit displays a message to inform the user that the corrected extraction candidate does not meet all of the extraction conditions. The information processing system described in (((13))). (((15))) The one or more processors described above are: Among the extraction conditions mentioned above, weights are added to the extraction conditions that match the corrected extraction candidate. Based on the aforementioned items, the extraction conditions, and the weights of the extraction conditions, new extraction candidates are obtained from a new non-standard form. The information processing system described in (((13))). (((16))) On the computer, A function to obtain extraction candidates, which are candidates to extract as corresponding information items in a non-standard form, from the information recorded in the non-standard form. A function to display on the display unit the location in the non-standard form where the extraction candidates are described, A program to achieve this. [Explanation of Symbols]

[0115] 1…Information processing system, 10…OCR processing server, 11…Control unit, 12…Secondary storage unit, 13…Communication unit, 30…User terminal, 300…Operation screen, 311…Preview image, 312…Highlighted image, 313…Badge image, 314…Reference image, 315…Warning image, 320…Settings screen, 330…Details screen, 331…Partial image, 332…Output field, 333…Confirm button image, 350…Notification image, 370…Confirmation screen

Claims

1. An information processing system comprising one or more processors, The one or more processors described above are: From the information recorded in the non-standard form, we obtain candidate extracts that correspond to the items in the non-standard form. The extraction candidates are obtained based on the item and multiple extraction conditions set by the user for that item. Priority is assigned to the extraction candidates that satisfy more of the extraction conditions than the extraction candidates that satisfy fewer conditions. The assigned priority is displayed on the display unit. The section of the non-standard form in which the extraction candidates are described is displayed on the display unit. Information processing system.

2. The one or more processors described above are: Along with the location of the extraction candidate, the reason for the priority assigned to the extraction candidate, based on the status of fulfillment of the extraction conditions, is displayed on the display unit. The information processing system according to claim 1.

3. The one or more processors The reason for the aforementioned priority is to display the conditions that the extraction candidates have not met. The information processing system according to claim 2.

4. The one or more processors The reason for the aforementioned priority is to display the conditions that the extraction candidates have met. The information processing system according to claim 2.

5. The one or more processors described above are: The image of the non-standard form displays the location of the extraction candidate. The information processing system according to claim 1.

6. The one or more processors described above are: From the information described in the non-standard form, multiple extraction candidates are obtained. The portion of the extraction candidate specified by the user from among the multiple extraction candidates is displayed on the image of the non-standard form. The information processing system according to claim 1.

7. The one or more processors described above are: When the user selects the extraction candidate, the selected extraction candidate is used as the result of extracting the information corresponding to the item. The information processing system according to claim 1.

8. The one or more processors described above are: The selected location of the extraction candidate is learned as the location of the description information corresponding to the item. The information processing system according to claim 1.

9. The one or more processors described above are: When the user inputs the information corresponding to the aforementioned item, the input information is treated as the result of extracting the information corresponding to that item. The image of the non-standard form is used to detect the location where the entered information is recorded. The information processing system according to claim 1.

10. The one or more processors described above are: The detected location is learned as the location of the information corresponding to the item. The information processing system according to claim 1.

11. The one or more processors described above are: Based on the aforementioned items and the extraction conditions set for those items, the extraction candidates are obtained. Based on correction instructions from the user, the extracted candidates are corrected. Even if the corrected extraction candidate does not satisfy all of the extraction conditions, the corrected extraction candidate shall be taken as the result of extracting the information corresponding to the item. The information processing system according to claim 1.

12. The one or more processors described above are: The display unit displays a message to inform the user that the corrected extraction candidate does not meet all of the extraction conditions. The information processing system according to claim 1.

13. An information processing system comprising one or more processors, The one or more processors described above are: From the information recorded in the non-standard form, we obtain candidate extracts that correspond to the items in the non-standard form. The section of the aforementioned non-standard form in which the extraction candidates are described is displayed on the display unit. Based on the aforementioned items and the extraction conditions set for those items, the extraction candidates are obtained. Based on correction instructions from the user, the extracted candidates are corrected. Even if the corrected extraction candidate does not satisfy all of the extraction conditions, the corrected extraction candidate shall be treated as the result of extracting the information corresponding to the item. Among the extraction conditions mentioned above, weights are added to the extraction conditions that match the corrected extraction candidate. Based on the aforementioned items, the extraction conditions, and the weights of the extraction conditions, new extraction candidates are obtained from a new non-standard form. Information processing system.

14. On the computer, A function to obtain extraction candidates, which are candidates to extract as corresponding information items in a non-standard form, from the information recorded in the non-standard form. The extraction candidates are obtained based on the item and multiple extraction conditions set by the user for that item, and the function A function to assign priority to extraction candidates that satisfy more of the aforementioned extraction conditions, so that they are given priority over extraction candidates that satisfy fewer conditions. A function to display the assigned priority on the display unit, A function to display the section of the non-standard form in which the extraction candidates are described on the display unit, A program to achieve this.

Images