Information processing system, information processing method, and information processing program

Abstract

To enter appropriate information into a document input item.SOLUTION: An information processing system 1, which generates a document by embedding input information into one or a plurality of input fields of a document format, comprises: a candidate selection unit 25 which selects one reading data as a candidate of input information related to one input item read from the data medium to be read from among the plurality of reading data corresponding to one input item, each reading data being image data or text data, on the basis of predetermined selection criteria; a first display control unit 26 which displays the reading data selected by the candidate selection unit 25 on a first screen as a candidate for the input information related to the input item; and a second display control unit 27 which, when an input to change the candidate for the input information related to the input item displayed on the first screen is accepted, displays a second screen which can change the candidate of the input information to other data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a document using data read from a medium.

Background Art

[0002] Conventionally, there has been proposed a document creation device for creating documents for issuing ID cards, which includes information reading means for reading image information on the front and back of an ID card, image forming means for forming an image on a predetermined sheet based on the image acquisition information by the information reading means, and control means for controlling the image forming means to form an image of the front and / or back of the ID card in a predetermined area on the sheet (see Patent Document 1).

[0003] In addition, there has been proposed a method (see Patent Document 2) in which the OCR recognition result is compared with the input data by an operator. If they match, it is determined that the OCR recognition process is performed correctly and the recognition result is correct. If they do not match, the OCR recognition result is replaced with the input data by the operator, and further, the input data and the image data targeted for the OCR recognition process are displayed for visual confirmation.

[0004] Furthermore, when valid text data is generated from character image data obtained by optically reading passport travel document information, document data is generated using this text data. When valid text data is not generated, the character image data is displayed by a display means. When the character display of the displayed character image data is usable, document data is generated using the processed image data created by processing the character image data. When the character display of the displayed character image data is difficult to use, a document data generation device for duty-free documents (see Patent Document 3) that generates document data filled in by handwriting has been proposed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] Conventionally, a system is known in which information recorded on a medium such as an identity document is read by a reading device such as a camera, and a form is generated using the read data. In such a system, the data read from the medium is often used as input information for the input items of the form. However, there is a problem that even when the read data is not in the content or data format intended by the user, it is input (described) into the form as it is without the intended content or data format.

[0007] In view of the above problems, the present disclosure aims to input appropriate information into the input items of a form.

Means for Solving the Problems

[0008] An example of the present disclosure is an information processing system that generates a form by embedding input information in one or more input items of a form format, the plurality of read data corresponding to one input item included in the one or more input items, which is read from a medium to be read, each of the plurality of read data being image data or text data, candidate selection means for selecting one read data as a candidate for the input information related to the one input item based on a predetermined selection criterion from among the plurality of read data, first display control means for displaying, on a first screen, the read data selected by the candidate selection means as a candidate for the input information related to the one input item, and second display control means for displaying a second screen that enables the candidate for the input information to be changed to other data when an input for changing the candidate for the input information related to the one input item displayed on the first screen is received.

[0009] The present disclosure can be understood as an information processing apparatus, a system, a method executed by a computer, or a program to be executed by a computer. Further, the present disclosure can also be understood as a recording medium in which such a program is recorded and can be read by a computer or other devices, machines, etc. Here, the recording medium readable by a computer or the like refers to a recording medium that accumulates information such as data and programs by an electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer or the like.

Advantages of the Invention

[0010] According to the present disclosure, it becomes possible to input appropriate information into the input items of the form.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of an information processing apparatus, a system, a method, and a program according to the present disclosure will be described with reference to the drawings. However, the embodiments described below are illustrative of the embodiments, and do not limit the information processing apparatus, system, method, and program according to the present disclosure to the specific configurations described below. In practice, specific configurations according to the implementation mode may be appropriately adopted, and various improvements and modifications may be made.

[0013] In the present embodiment, embodiments will be described when the information processing apparatus, system, method, and program according to the present disclosure are implemented in a system that generates forms using data read from personal identification documents. However, the information processing apparatus, system, method, and program according to the present disclosure can be widely used for technologies for generating forms using data read from a medium, and the application target of the present disclosure is not limited to the examples shown in the embodiments.

[0014] Conventionally, a system is known that reads information (data) recorded (including "described" in "recorded") on a medium such as a personal identification document and generates a form such as an application form by using the read information. In such a system, in many cases, character recognition processing (OCR (Optical Character Recognition) processing) is performed on an image obtained by imaging the front and / or back surfaces of a personal identification document with a camera, so that the information described on the front and / or back surfaces of the personal identification document can be read. And in many cases, the character string recognition result, which is the information read by the OCR processing, is adopted (used) as the input information for the input items of the form.

[0015] However, since the reading accuracy (correct reading rate) of OCR is not 100%, the character string recognition result may include characters that the user did not intend, such as misread characters. Also, when there are foreign characters (characters not registered in the character input software) printed on the surface of media such as identification documents, and it is difficult for OCR to read the foreign characters, the character string recognition result obtained by the OCR process does not include the foreign characters (characters intended by the user). Also, when an OCR error occurs, it may not be possible to obtain the character string recognition result itself. In such cases, there was a problem that reading data that was not in the content or format intended by the user, such as data containing misread characters or data not containing foreign characters printed on the media, was input (described) into the form as input information.

[0016] Therefore, in the system described in this embodiment, by displaying candidates for input information selected based on selection criteria on a first screen and displaying a second screen that allows the candidates for input information to be changed to other data, it is possible to easily change the information to be input into the input item of the form to other data before inputting information into the input item of the form. Also, by displaying at least other reading data (reading data other than the reading data displayed as candidates for input information on the first screen) on the second screen so that it can be selected, it is possible to easily select the information to be input into the input item from among the other reading data. As a result, the user can easily change to using other reading data as the input information. From the above, it becomes possible to input appropriate information into the input item of the form.

[0017] FIG. 1 is a schematic diagram showing the configuration of the system according to the present embodiment. The system according to the present embodiment is a system in which an information processing apparatus 1 is communicably connected to a reading apparatus 9 via a network or other communication means. In the present embodiment, in a system in which a card reader (reading apparatus 9) including a reading unit and an information processing apparatus 1 connected to the card reader by wire or wirelessly are connected, an aspect of implementing the technology according to the present disclosure will be described. However, the system according to the present embodiment may be any information processing system in which a reading apparatus 9 that reads information from a medium (such as an identity document) in which information related to a user is recorded and an information processing apparatus 1 are connected. For example, the technology according to the present disclosure is also applicable in a system in which a scanner apparatus and an information processing apparatus 1 are connected.

[0018] The information processing apparatus 1 is a computer including a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage device 14 such as an EEPROM (Electrically Erasable and Programmable Read Only Memory) or an HDD (Hard Disk Drive), an input device 15 such as a keyboard, a mouse, or a touch panel, an output device 16 such as a display, and a communication unit 17 such as a NIC (Network Interface Card). However, with regard to the specific hardware configuration of the information processing apparatus 1, appropriate omission, replacement, or addition can be made according to the embodiment. Further, the information processing apparatus 1 is not limited to a device composed of a single housing, and may be realized by a plurality of devices using so-called cloud or distributed computing technologies. Note that the information processing apparatus 1 may be any device (computer) such as a tablet terminal, a desktop personal computer, a notebook personal computer, a smartphone, or a mobile phone (feature phone). The information processing apparatus 1 generates a form (form data) by embedding input information in one or more input items of a form format (form form) using the information (read data) recorded on a medium such as an identity document read by the reading apparatus 9.

[0019] The reading device 9 is a computer including a reading unit 91, a camera (face photographing camera) 92, a CPU 93, a ROM 94, a RAM 95, a storage device 96, an input device 97 such as a touch panel, an output device 98 such as a display, and a communication unit 99, etc. However, regarding the specific hardware configuration of the reading device 9, appropriate omissions, substitutions, and additions are possible according to the embodiment. Further, the reading device 9 is not limited to a device composed of a single housing, and may be realized by a plurality of devices using so-called cloud or distributed computing technologies, etc.

[0020] The reading device 9 (reading unit 91) is a device that reads information from a medium (such as an identity document) on which information related to the user is recorded. Hereinafter, "information related to the user" is referred to as "user information". In the present embodiment, an identity document (identity card) is exemplified as a medium on which user information is recorded. An identity document is a document for confirming that the person is the individual (a document for proving identity), and examples include a driver's license, a My Number card, a passport, a residence card, a health insurance cardholder's certificate, a student ID card, a qualification certificate, etc. However, the identity document is not limited to these exemplified documents, and any other document that can confirm that the person is the individual may be used. Further, the medium on which user information is recorded may be other cards, documents, etc. other than identity documents as long as it is a medium on which user information is recorded. Further, the medium on which user information is recorded is not limited to the card format, and may be a medium having a size different from the card size. Further, the medium on which user information is recorded may be made of any material such as plastic or paper. Further, any information may be recorded on the medium. For example, personal information of the user such as name, address, changed address, date of birth, number (such as driver's license number or passport number), expiration date (such as driving period or passport period), face photograph, etc. may be recorded as user information on the medium.

[0021] In this embodiment, an embodiment of generating a form (form data) by reading user information from a medium on which the user information is recorded will be described. However, an embodiment of generating form data based on information other than the user information read from the medium is also applicable to the present disclosure. Therefore, the information recorded on the medium may include information other than the user information.

[0022] In this embodiment, as the reading device 9, a card reader that reads information recorded on a medium is exemplified. The reading device 9 transmits the read information (data) to the information processing device 1. The reading unit 91 is, for example, a camera that photographs a medium such as a camera that photographs the face (for example, the front or back) of an identity document (a camera for photographing a medium), and / or an IC reader that reads information recorded on an IC (Integrated Circuit) chip mounted (built-in) on the medium. The reading device 9 reads the information described on the face of the identity document by performing character recognition processing on the image obtained by the medium photographing camera photographing the face of the identity document. Note that the reading device 9 may be a device capable of reading both sides of the medium (a device having a double-sided automatic reversing function or a device requiring a double-sided simultaneous reading function (a one-pass double-sided simultaneous reading device), etc.), or a device that can only read one side (for example, a single-sided) of the medium (a single-sided reading device). Further, the reading device 9 reads the information recorded on the IC chip by reading the IC chip on which the user information is recorded and mounted on a medium such as an IC card.

[0023] As described above, in the present embodiment, a card reader is exemplified as the reading device 9, but the reading device 9 may be another device other than a card reader. Therefore, the reading device 9 may be, for example, an imaging device (a device that images a medium and acquires an image) such as a scanner, a digital camera, or a camera sensor built in a smartphone / tablet. In this case, in the reading device 9, by performing OCR processing on the captured image, the information described in the medium is read. However, the OCR processing may be performed not by the reading device 9 but by the information processing device 1. In this case, the captured image is transmitted from the reading device 9 to the information processing device 1, and the OCR processing is performed in the information processing device 1 that has received the image.

[0024] As described above, the reading device 9 according to the present embodiment has a function of transmitting data (text data, image data, etc.) read from a medium to the information processing device 1 via a network. Note that the communication means, hardware configuration, etc. of the reading device 9 that can adopt the method according to the present embodiment are not limited to the examples in the present embodiment. Further, in the information processing system according to the present embodiment, when a process using a photo of the user's face such as identity verification is not performed, the reading device 9 does not necessarily need to include a camera (a camera for face photographing) 92. Also, in this case, the reading device 9 does not necessarily need to include an identity verification unit 82 described later.

[0025] FIG. 2 is a diagram showing an outline of the functional configuration of the information processing apparatus and the reading apparatus according to the present embodiment. In the information processing apparatus 1, a program recorded in the storage device 14 is read into the RAM 13 and executed by the CPU 11, and each hardware provided in the information processing apparatus 1 is controlled, whereby a form format storage unit 21, a priority storage unit 22, a form format selection unit 23, a data acquisition unit 24, a candidate selection unit 25, a first display control unit 26, a second display control unit 27, an input reception unit 28, a form generation unit 29, and an output unit 30 are provided. In the present embodiment and other embodiments described later, each function provided in the information processing apparatus 1 is executed by the CPU 11 which is a general-purpose processor, but a part or all of these functions may be executed by one or a plurality of dedicated processors.

[0026] Further, in the reading apparatus 9, a program recorded in the storage device 96 is read into the RAM 95 and executed by the CPU 93, and each hardware provided in the reading apparatus 9 is controlled, whereby a reading unit 81, an identity verification unit 82, and a data transmission unit 83 are provided. In the present embodiment and other embodiments described later, each function provided in the reading apparatus 9 is executed by the CPU 93 which is a general-purpose processor, but a part or all of these functions may be executed by one or a plurality of dedicated processors.

[0027] The form format storage unit 21 stores one or more form formats. A form format is a file that defines the format of a form and includes data indicating the layout of the form and data defining the specifications of items (form items) included in the form. The data indicating the layout of the form defines fixed information described in the form, such as form item names (e.g., "Name", "Address", "Date of Birth", etc.) and form names (e.g., "Request for Copy of Resident Register, etc."), data input fields, and their positions (arrangements). In addition, the data defining the specifications of form items defines, as specifications regarding the input fields for data (input information) related to each form item, for example, character size, name of the font to be used, position, etc. Form items include input items into which variable information (data) such as items related to a name or address is input. In the present embodiment, one or more input items are defined in the form format, and for each input item, an area (a predetermined area within the form (form layout), hereinafter referred to as the "embedded area related to the input item") for embedding input information related to the input item is defined. Note that the form format storage unit 21 may store a form format generated by an external device, or may store a form format such as a form template generated in the information processing apparatus 1. Further, the form format may be any form format, and examples include the format for applying for a resident register for personal application, the format for applying for a resident register for an agent, the format for applying for a certificate of seal registration, etc.

[0028] The priority memory unit 22 stores priority data in which priorities are defined for various types of data. The priority defined in the priority data means the degree (the degree of being prioritized) of preferentially selecting that data as a candidate for input information related to the input item. The priority is predefined by a user such as a system operator. The priority data is data used when performing selection process (A) and selection process (C) described later. When the priorities for various types of data are not made different depending on the type of medium or input item (the same among different types of media or input items), only one piece of priority data defining the priorities for various types of data needs to be stored. On the other hand, when the priorities for various types of data are made different depending on the type of medium, the priority data defining the priorities for various types of data may be stored for each type of medium (or for every two or more types of media). Also, when the priorities for various types of data are made different depending on the input items of the form, the priority data defining the priorities for various types of data may be stored for each input item (or for every two or more input items). Furthermore, when the priorities for various types of data are made different depending on the type of medium and input item, the priority data defining the priorities for various types of data may be stored for each type of medium and each input item (for every one or more types of media and every one or more input items). Thus, the priority data may be stored corresponding to the type of medium and / or input item.

[0029] Note that the data (type of data) for which priorities are defined in the priority data includes different types of data (multiple types of data) in which the data format (for example, image or text, etc.) and / or the data acquisition source (for example, the surface (front or back) of the medium or the IC chip) are different from each other. For example, the data for which priorities are defined includes various types of data that can be obtained from the medium, such as image data (cropped image) and recognition result data (OCR result) obtained from the surface (front, back, etc.) of the medium, and image data and text data obtained from the IC chip. However, the various types of data for which priorities are defined may be any data as long as it can be input information related to the input item. In this embodiment, a priority table is exemplified as the priority data.

[0030] FIG. 3 is a diagram showing an example of a priority table according to the present embodiment. In the priority table shown in FIG. 3, priorities for various data are predetermined. In FIG. 3, as various data for which priorities are determined, there are exemplified a cutout image related to a remarks column cut out from an imaging image of the back surface of a medium, a cutout image cut out from an imaging image of the front surface of the medium, text data in an IC chip, and recognition result data (OCR result) obtained by performing character recognition processing on an imaging image (cutout image) of the front surface of the medium. Hereinafter, an imaging image of the front surface of the medium is referred to as a "front surface image of the medium", an imaging image of the back surface of the medium is referred to as a "back surface image of the medium", a cutout image related to a remarks column cut out from the back surface image of the medium is referred to as a "back surface remarks column image", a cutout image cut out from an imaging image of the front surface of the medium is referred to as a "front surface cutout image", and recognition result data obtained by performing character recognition processing on the front surface image of the medium (front surface cutout image) is referred to as "recognition result data of the front surface". Note that the front surface cutout image can be obtained, for example, from the front surface image of the medium or the IC chip, and may be obtained by either one, or which of the front surface image of the medium and the IC chip to obtain an image from (acquisition source) may be specified in the priority table. In FIG. 3, priorities of 1 (highest priority), 2, 3, and 4 are determined for the back surface remarks column image, the front surface cutout image, the text data in the IC chip, and the recognition result data of the front surface, respectively. Note that the priority table shown in FIG. 3 is, for example, a priority table used when the type of the medium is a driver's license and the input item is an item related to an address (hereinafter referred to as an "address item"). That is, the priority table shown in FIG. 3 may be used as a priority table corresponding to the driver's license which is the type of the medium and the address item which is the input item.

[0031] FIG. 4 is a diagram showing an example of a priority table according to the present embodiment. Similarly to FIG. 3, in FIG. 4, priorities for various data are defined in the priority table. In FIG. 4, as various data for which priorities are defined, text data in the IC chip, cutout images of the surface, and recognition result data of the surface are exemplified. Also, in FIG. 4, priorities of 1 (the highest priority), 2, and 2 are defined for the text data in the IC chip, the cutout images of the surface, and the recognition result data of the surface, respectively. Here, the same priority may be defined for a plurality of data so that the priority of the cutout image of the surface and the priority of the recognition result data of the surface are both 2. Note that the priority table shown in FIG. 4 is, for example, a priority table used when the type of medium is a My Number card and the input item is an address item. That is, the priority table shown in FIG. 4 may be used as a priority table corresponding to the type of medium, which is a My Number card, and the input item, which is an address item.

[0032] Note that the priority data is not limited to a priority table (table format), and may be in another format other than the table format. That is, the priorities for various data may be stored in a format other than the table format. Note that the various data (data types) for which priorities are defined are not limited to the data shown in FIGS. 3 and 4, and may include recognition result data (OCR results) obtained from the image of the remarks column on the back surface, blank data (no input information), and the like.

[0033] The form format selection unit 23 selects a form format used to generate a form (form data). When the form format selection unit 23 receives an input related to the selection of a form format from the user, it selects (extracts) a form format corresponding to the input from among the form formats stored in the form format storage unit 21 as the form format for generating a form. Note that the input reception unit 28, which will be described later, may receive an input related to the selection of a form format from the user. Also, when only one form format is stored in the form format storage unit 21 or when the user does not need to select a form, the information processing apparatus 1 does not necessarily need to include the form format selection unit 23.

[0034] The data acquisition unit 24 acquires various types of information regarding the medium to be read, which is used for generating a form (form data). Specifically, the data acquisition unit 24 acquires data (read data read from the medium) regarding information (such as user information) recorded on the medium to be read, data indicating the type of the medium to be read (data indicating the medium type), and data indicating the result of identity verification. In the present embodiment, these various types of information (data) and the like are acquired from the reading device 9 (data transmission unit 83). The read data, which is data regarding the information recorded on the medium to be read and acquired by the data acquisition unit 24, includes data regarding the information described on the surface of the medium to be read and / or data recorded on the IC chip mounted on the medium to be read. Note that the data regarding the information described on the surface of the medium to be read includes, for example, data of the front surface image and the back surface image of the medium, data of the cut-out image cut out from the front surface image and the back surface image of the medium, and recognition result data obtained by performing OCR processing on the cut-out image, i.e., read data related to the surface of the medium. Also, the data recorded on the IC chip mounted on the medium to be read includes text data and / or image data (such as cut-out images) related to information such as name and address.

[0035] FIG. 5 is a diagram showing an example of a medium according to this embodiment. In the example of FIG. 5, a driver's license is exemplified as the medium. In the case of the driver's license shown in FIG. 5, the data acquisition unit 24 acquires, as recognition result data (character string recognition results), for example, "Tokyo Taro", "1-2-10, △△-cho, □□-shi, 〇〇-ken", "January 1, 1990" etc. for each recording item (reading item) in the medium such as "Name", "Address", "Date of Birth". Further, the data acquisition unit 24 acquires, for example, an image (cut-out image) of a portion where information is described for each recording item, read by the reading unit 91 (medium photographing camera). For example, as a cut-out image for the recording item "Name", an image of the portion (area) where "Tokyo Taro" is described is acquired. Further, the data acquisition unit 24 acquires, for example, as data recorded in the IC chip mounted on the driver's license shown in FIG. 5, the recorded contents (text data) such as "Tokyo Taro", "1-2-10, △△-cho, □□-shi, 〇〇-ken", "January 1, 1990" for each recording item such as "Name", "Address", "Date of Birth", and image data (cut-out image) of a portion where information is described for each recording item.

[0036] Note that in this embodiment, the cut-out images for each recording item read by the reading unit 91 (medium photographing camera) and the cut-out images for each recording item recorded in the IC chip are assumed to be acquired by different cameras, but they may also be acquired by the same camera. Further, as the cut-out image, an image simply cut out from the medium image, an image with enhanced visibility by noise removal processing or the like in the process of OCR processing, an image obtained by performing image processing suitable for printing such as a process of removing or thinning out the background pattern (for enhancing the visibility of characters), or any other processed image may be used.

[0037] Note that any method may be used to obtain the type of the medium to be read by the data acquisition unit 24. For example, a method of obtaining (receiving) the type of the medium acquired (determined) by the reading unit 81 from the data transmission unit 83 (a method of obtaining the type of the medium obtained by determining the type of the medium using the medium to be read) may be used. As another method, a method of obtaining the type of the medium to be read (information on the type of the medium) input by the user who is the user via the input device 15 or the input device 97 may be used. However, these methods of obtaining the type of the medium are merely examples, and as described above, any other method may be used to obtain the type of the medium.

[0038] The candidate selection unit 25 selects (extracts) data to be candidates for input information related to the input items of the form (form format) based on a predetermined selection criterion. In the present embodiment, the candidate selection unit 25 selects only one piece of read data to be a candidate for the input information related to the input item from among a plurality of pieces of read data corresponding to the input item read from the medium to be read. Here, the read data corresponding to the input item is the read data for the recording item (the item recorded on the medium) corresponding to the input item. That is, the read data corresponding to the input item is data indicating the recording information (recording content) for the recording item corresponding to the input item.

[0039] For example, regarding a driver's license, assume a case where the address item, which is an input item on the form (hereinafter referred to as the input item "address"), corresponds to the address item and the address change reserve item recorded on the medium (hereinafter referred to as the recorded item "address" and the recorded item "address change reserve", respectively). In this case, the plurality of read data corresponding to the input item "address" may include, for example, image data (cropped image) of the part where information about the recorded item "address" on the front surface of the driver's license is described, recognition result data (recognized character string) obtained by performing character recognition processing on the cropped image, image data (cropped image) of the part where information about the recorded item "address change reserve" on the back surface of the driver's license is described, recognition result data (recognized character string) obtained by performing character recognition processing on the cropped image, and data (image data and / or text data) about the recorded item "address" read from the IC chip mounted on the driver's license. Thus, each read data is text data (including recognition result data) or image data. Therefore, the candidate selection unit 25 selects text data or image data as a candidate for the input information. However, the candidate selection unit 25 can also make a selection not to embed data (information) in the input item, that is, to leave the embedding area related to the input item blank, based on the selection criteria. In this embodiment, a character string is a sequence (consecutive characters) consisting of one or more characters, and the characters include hiragana, katakana, kanji, alphabets, numbers, symbols, and the like.

[0040] Note that, as in the example of the driver's license described above, the number of record items corresponding to one input item is not limited to a plurality, and only one record item may correspond to one input item. Which record item corresponds to each input item, that is, the correspondence between the input item and the record item, shall be defined by the form format, media definition information, or other definition information. Also, the plurality of read data obtained as the read data corresponding to the input item may all be data related to information described in the same area (predetermined area) of the surface of the medium to be read. For example, the image data (cropped image) of the portion where the information about the record item "address" on the front surface of the driver's license is described, the recognition result data obtained by performing character recognition processing on the cropped image, and the image data (cropped image) and text data of the portion where the information about the record item "address" on the front surface is described, which are obtained from the IC chip, are data related to information described in the same area. Alternatively, among the plurality of read data obtained as the read data corresponding to the input item, there may be included data indicating information described in an area different from other data, such as the image data (cropped image) of the portion where the information about the record item "preliminary address change" on the back surface of the driver's license described above is located.

[0041] Then, only one piece of data selected by the candidate selection unit 25 is displayed as a candidate for the input information related to the input item on the first screen. That is, the candidate selection unit 25 determines the data to be preferentially displayed as a candidate for the input information on the first screen. Note that the candidate selection unit 25 may select a candidate for the input information for only one input item among the input items included in the form (form format), or may select candidates for the input information for a plurality of input items (including all input items).

[0042] The method of selecting data that the candidate selection unit 25 uses as candidates for input information related to input items, that is, the selection criteria used by the candidate selection unit 25 will be described in detail. In the present embodiment, as selection methods, (A) a selection method using priority data, (B) a selection method using the probability of character recognition, and (C) a selection method combining the methods of (A) and (B) are exemplified as three selection methods (three types of selection criteria). The candidate selection unit 25 can use any one of the following methods (A) to (C) for each input item.

[0043] (A) Selection method using priority data The first selection method is a method of selecting candidates for input information using the priority data stored in the priority memory unit 22. That is, the selection criterion is a criterion of selecting, according to the priority data, the reading data to be displayed as candidates for the input information related to the input item on the first screen. For example, in the case of the priority data shown in FIG. 4, the text data in the IC chip has the highest priority. Therefore, when the text data in the IC chip is acquired as the reading data corresponding to the input item, the candidate selection unit 25 selects the text data in the IC chip among the reading data corresponding to the input item as a candidate for the input information. In this way, among the acquired reading data, the data with the highest priority determined in the priority data may be selected as a candidate for the input information. As described above, when the priority data is stored in correspondence with the type of medium (such as the type of ticket) and / or the input item, the candidate selection unit 25 refers to the priority data corresponding to the type of medium to be read and / or the target input item, thereby selecting a candidate for the input information related to the target input item. That is, in this case, the selection criterion is a criterion of selecting, according to the priority data corresponding to the type of medium to be read and / or the target input item, the reading data to be displayed as a candidate for the input information related to the target input item on the first screen. That is, the candidate selection unit 25 can avoid selecting data with lower priority in the priority data as a candidate for the input information or reduce the degree of being selected as a candidate for the input information. On the other hand, the candidate selection unit 25 can select data with higher priority in the priority data as a candidate for the input information or increase the degree of being selected as a candidate for the input information.

[0044] (B) Selection method using the probability of character recognition The second selection method is a method of selecting candidates for input information using the likelihood of character recognition (string recognition result) when the recognition result data, which is the character recognition result (recognized string) obtained by character recognition, contains one or more for the read data corresponding to the input item. That is, the selection criterion is a criterion related to the likelihood of character recognition. Hereinafter, three selection methods (a to c) using the likelihood of character recognition will be exemplified. Note that one of the methods a to c may be used, or a method combining two or more of the methods a to c may be used.

[0045] (a) Method by comparing the reliability score with a threshold This method is a method of selecting candidates for input information by comparing the reliability score indicating the likelihood of the recognition result data (recognized string), which is the result of the OCR process (OCR result), with a predetermined threshold. The reliability score is a score indicating the reliability of the string (text) read by the OCR process, and is represented by a numerical value between 0 and 100 or a numerical value between 0 and 1, indicating that the higher the score, the higher the reliability, and the lower the score, the lower the reliability. When the OCR process is performed by the reading device 9, the data acquisition unit 24 acquires the reliability score from the data transmission unit 83. Note that when the data acquisition unit 24 (information processing device 1) performs the OCR process, the data acquisition unit 24 can acquire the reliability score by performing the OCR process. As a result of comparing the reliability score with a predetermined value that is the threshold, if the reliability score is less than or equal to the predetermined value, it can be determined that the recognized string having the reliability score is unlikely, that is, the character recognition performed to acquire the recognized string is unlikely. In such a case, the candidate selection unit 25 can select the read data (for example, the cut-out image), which is the image data, or other text data other than the recognition result data (for example, the text data acquired from the IC chip) as candidates for the input information because the recognition result data has low reliability. On the other hand, when the reliability score exceeds the predetermined value, the candidate selection unit 25 can select the recognition result data as a candidate for the input information because the recognition result data has high reliability.

[0046] This method can also be used when the read data corresponding to the input item includes a plurality of recognition result data (recognized character strings) obtained by character recognition using a plurality of character recognition engines (OCR engines). For example, assume that the reading device 9 (or the information processing device 1) is equipped with three character recognition engines, and each character recognition engine performs character recognition on the same cut-out image, thereby obtaining three recognized character strings. The cut-out image that is the object of character recognition by each character recognition engine is an image of a predetermined area (area where information is described) corresponding to the input item on the surface of the medium to be read. In this case, when the reliability scores of each of these three recognized character strings exceed a predetermined threshold (or are equal to or higher than the predetermined threshold), the candidate selection unit 25 can select the recognition result data as candidates for the input information because the recognition result data is highly reliable respectively. At this time, among the three recognition result data, the recognition result data with the highest reliability score may be selected as a candidate for the input information. Also, the predetermined value that is the threshold may be an arbitrary numerical value such as 90 or 0.9, or may be a preset fixed value, or may be a variable value that varies depending on the situation.

[0047] Thus, in this method, the criterion regarding the likelihood of character recognition is set such that when the reliability score for the recognition result data is below a predetermined threshold, the recognition result data is less likely to be selected as a candidate for the input information related to the input item compared to when the reliability score for the recognition result data exceeds the predetermined threshold. That is, when the reliability score for the recognition result data is below the predetermined threshold, the candidate selection unit 25 can either not select the recognition result data as a candidate for the input information or reduce the degree to which it is selected as a candidate for the input information. On the other hand, when the reliability score for the recognition result data exceeds the predetermined threshold, the candidate selection unit 25 can either select the recognition result data as a candidate for the input information or increase the degree to which it is selected as a candidate for the input information. Note that the criterion regarding the likelihood of character recognition may be set such that when the reliability score for the recognition result data is less than the predetermined threshold, the recognition result data is less likely to be selected as a candidate for the input information related to the input item compared to when the reliability score for the recognition result data is greater than or equal to the predetermined threshold.

[0048] (b) Method of collating with the address database This method is a method used when the input item is an address item. It is a method of selecting candidates for input information by comparing the address indicated by the recognition result data, which is the read data corresponding to the input item, with the addresses registered in the address database. Here, the address database (not shown) pre-retains information on a plurality of addresses. For example, information on existing addresses such as the latest address information is retained. The address database has, for example, a hierarchical structure with each layer delimited by prefectures, groups, cities, wards, towns and villages, lot numbers, etc., and it is possible to perform a search up to a predetermined delimiter (for example, up to "prefecture group city ward town and village"). However, the address database may have any structure other than a hierarchical structure. When the address indicated by the recognition result data is included in the address database, it can be determined that the recognition result data is reliable (correct). In such a case, since the recognition result data is reliable, the candidate selection unit 25 can select the recognition result data (recognition character string) as a candidate for the input information. On the other hand, when the address indicated by the recognition result data is not included in the address database, since the recognition result data is not reliable, the candidate selection unit 25 can select image data or other text data other than the recognition result data as a candidate for the input information.

[0049] Thus, in this method, the criterion regarding the reliability of character recognition is set such that when the input item is an address item, if the address indicated by the recognition result data is not included in the address database, compared to the case where the address indicated by the recognition result data is included in the address database, the recognition result data indicating the address is less likely to be selected as a candidate for the input information related to the input item. That is, when the address indicated by the recognition result data is not included in the address database, the candidate selection unit 25 can either not select the recognition result data as a candidate for the input information or reduce the degree to which it is selected as a candidate for the input information. On the other hand, when the address indicated by the recognition result data is included in the address database, the candidate selection unit 25 can either select the recognition result data as a candidate for the input information or increase the degree to which it is selected as a candidate for the input information.

[0050] (c) Method for Comparing Character Recognition Results by Multiple Character Recognition Engines This method is used when multiple recognition result data by multiple character recognition engines can be obtained as the read data corresponding to the input item, and it is a method of selecting candidates for the input information by comparing these multiple recognition result data. As described above, the multiple recognition result data by multiple character recognition engines are multiple recognized character strings obtained by performing character recognition on the same predetermined area (partial area corresponding to the input item) on the surface of the medium to be read by each character recognition engine. By comparing these multiple recognized character strings, it is possible to determine the reliability of the recognized character strings. For example, when all of the obtained multiple recognized character strings completely match (are the same character string), or when two recognized character strings obtained by two character recognition engines match by a predetermined ratio (e.g., 95%) or more, these recognized character strings can be determined to be reliable. Also, when three or more character recognition engines are provided, when the majority of all the obtained recognized character strings completely match (are the same character string), these majority recognized character strings can be determined to be reliable. In this way, when the multiple obtained recognized character strings satisfy a predetermined condition (for example, having a predetermined relationship such as the recognized character strings completely matching each other), the candidate selection unit 25 can select candidates for the input information related to the input item from these multiple recognized character strings because the recognized character strings are reliable. On the other hand, when the multiple obtained recognized character strings do not satisfy the predetermined condition, the candidate selection unit 25 can select image data or other text data other than the recognition result data as candidates for the input information because the recognized character strings are not reliable. For example, when three or more character recognition engines are provided, when among all the obtained recognized character strings, although there are recognized character strings that match each other but the majority of all the recognized character strings do not match, image data, etc. may be selected as candidates for the input information.

[0051] Thus, in this method, the criterion regarding the likelihood of character recognition is set such that when a plurality of recognized character strings do not satisfy a predetermined condition, the candidate for input information related to the input item is less likely to be selected from the plurality of recognized character strings compared to the case where the plurality of recognized character strings satisfy the predetermined condition. That is, when the plurality of recognized character strings do not satisfy the predetermined condition, the candidate selection unit 25 can refrain from selecting a candidate for input information from the plurality of recognized character strings or reduce the degree of selecting a candidate for input information from the plurality of recognized character strings. On the other hand, when the plurality of recognized character strings satisfy the predetermined condition, the candidate selection unit 25 can select a candidate for input information from the plurality of recognized character strings or increase the degree of selecting a candidate for input information from the plurality of recognized character strings.

[0052] Here, the predetermined condition is, as described above, for example, that the relationship between the plurality of recognized character strings has a predetermined relationship. The predetermined relationship is, for example, that the plurality of recognized character strings completely match, that the plurality of recognized character strings match by a predetermined ratio or more, that the majority of the plurality of recognized character strings completely match, and the like. When the plurality of recognized character strings match by a predetermined ratio or more, any one of the plurality of recognized character strings may be selected as a candidate for input information, or the recognized character string with a higher reliability score may be selected. Also, when the majority of the plurality of recognized character strings completely match, the recognized character string that matches with the majority may be selected as a candidate for input information. Further, the predetermined condition may be, as described in the method of (a) above, that the reliability score of each of the plurality of recognized character strings exceeds a predetermined threshold (or is equal to or higher than a predetermined threshold).

[0053] Incidentally, the candidate selection unit 25 may select a candidate for the input information by determining an OCR priority indicating the degree of preferentially selecting the recognition result data (OCR result). The OCR priority is a parameter whose degree increases when it is determined that the reliability of the OCR result is high. Therefore, the OCR priority may be determined, for example, by the height of the reliability score of the OCR result, whether the address indicated by the OCR result is included in the address database, whether a plurality of recognition character strings satisfy a predetermined condition, or the like. The candidate selection unit 25 may compare the OCR priority with a threshold value (for example, "medium", "0.8", etc.), and when the OCR priority exceeds the threshold value, select the OCR result as a candidate for the input information with priority over the image data and other text data. In the present embodiment, "high", "medium", and "low" are exemplified as the OCR priority, but the OCR priority may be represented by a numerical value.

[0054] (C) A selection method combining the method of (A) and the method of (B) The third selection method is a method that combines the above-described selection method using the priority data of (A) and the selection method using the reliability of character recognition of (B). For example, when text data in the IC chip is acquired as the read data corresponding to the input item, the text data in the IC chip is selected as a candidate for the input information according to the priority data shown in FIG. 4. On the other hand, when the text data in the IC chip (data defined as priority 1 in FIG. 4) is not acquired as the read data corresponding to the input item, the candidate selection unit 25 selects, according to the priority data shown in FIG. 4, the cut-out image of the surface or the recognition result data of the surface defined as priority 2 as a candidate for the input information. When it is possible to acquire (when acquired) both the cut-out image of the surface and the recognition result data of the surface as the read data corresponding to the input item, the candidate selection unit 25 uses the selection method using the reliability of character recognition of (B) described above to select one of the cut-out image of the surface and the recognition result data of the surface as a candidate for the input information. Thus, in this method, the selection criteria are a combination of the criteria regarding the reliability of character recognition and the criteria for selecting the read data to be displayed as a candidate for the input information related to the input item on the first screen according to the priority data. Note that the candidate for the input information may be selected by combining at least one of the selection methods (A) to (C) described above with any other selection method.

[0055] The first display control unit 26 causes the display unit to display a first screen that displays candidates for the input information related to one or more input items of the form format. The display unit is exemplified by a display or the like and is a type of the output device 16. The first screen is exemplified by a confirmation screen on which the user can confirm the information (candidates for the input information) to be input as the input information in the form format before generating the form. The first display control unit 26 causes the data selected by the candidate selection unit 25 to be displayed on the first screen as a candidate for the input information related to the input item.

[0056] FIG. 6 is a diagram showing an example of the first screen (at the time of first display) according to the present embodiment. The first screen shown in FIG. 6 is a confirmation screen for the user to confirm the information input as input information into the form format before generating (printing) the form. The first screen shown in FIG. 6 is an example of the first screen (the screen at the time of first display) that is displayed for the first time after candidates for the input information are selected by the candidate selection unit 25. On the first screen shown in FIG. 6, for each input item, an item name 31 (31a, 31b, 31c), a frame (column) 32 (32a, 32b, 32c) corresponding to the input item, and data 33 (33a, 33b, 33c) that is a candidate for the input information are displayed in the area within the frame. As candidates for the input information, text data (the text represented by the text data) or image data (the image represented by the image data) is displayed. However, when no candidate (nothing is input) is selected as the candidate for the input information, the area within the frame may be blank (not displayed). On the first screen shown in FIG. 6, the read data obtained from the driver's license shown in FIG. 5 is displayed as a candidate for the input information. For example, for the input item "Address", the item name 31b which is "Address", the solid line frame 32b indicating the address column, and in the area within this frame 32b, a cut-out image (hereinafter referred to as "cut-out image related to the address on the surface") 33b of the portion where the information about the recording item "Address" is described, cut out from the surface image of the medium, is displayed. Note that, as described above, the cut-out image displayed on the first screen as shown in FIG. 6 may be an image simply cut out from the medium image or an image subjected to arbitrary processing. In the example of FIG. 6, for the address item, since the cut-out image on the surface is selected as the candidate for the input information by the candidate selection unit 25, the cut-out image related to the address on the surface is displayed on the first screen. Note that for the input items "Name" and "Date of Birth", recognition result data (recognition character strings) 33a and 33c are displayed on the first screen because the recognition result data is selected as the candidate for the input information by the candidate selection unit 25.

[0057] Also, on the first screen, as shown in FIG. 6, there are buttons for finishing the confirmation before form creation (print confirmation) (the "Cancel" button), buttons for changing the candidates of the input information for each input item (the "Change" button), and a button for proceeding with the generation of the form (the "Next" button). The user can press (select) these buttons by touch operation, mouse operation, or the like. When the "Change" button is pressed, the second screen described later is displayed. Note that the second screen may be displayed in an overlay form on the first screen, or may be displayed in a form that transitions from the first screen to the second screen. When the "Next" button is pressed, for example, the screen may transition to a screen for confirming the form image, or the output process of the form data may be performed.

[0058] Note that the first screen is not limited to the screen configuration shown in FIG. 6, and may have any screen configuration as long as it is a screen on which the input information can be confirmed before form generation. Therefore, for example, the first screen may be an output image confirmation screen on which the data (candidates of the input information), which is the input information, is displayed in a state embedded in the form format, or may be a screen configuration that includes both the screen configuration shown in FIG. 6 and the output image. In the case of such a screen configuration, the user can confirm the output image (description content) of the form on the first screen.

[0059] When the candidates of the input information are changed on the second screen described later, the first display control unit 26 changes the data displayed as the candidates of the input information related to the input item on the first screen in accordance with the content changed on the second screen.

[0060] FIG. 7 is a diagram showing an example of the first screen according to the present embodiment. Also regarding the address field, when recognition result data is selected as a candidate for input information by the candidate selection unit 25, the first screen at the first display is the screen shown in FIG. 7. Further, in the second screen displayed when the change button for the address field in the first screen shown in FIG. 6 is pressed, when the user selects the recognition result data, the display of the first screen is changed to be like the screen shown in FIG. 7. That is, in accordance with the selection of the recognition result data for the address field in the second screen, the first display control unit 26 changes the display of the candidate for the input information related to the address field in the first screen (the input information candidate to be displayed) from the image data shown in FIG. 6 (33b shown in FIG. 6) to the recognition result data (33d shown in FIG. 7).

[0061] The second display control unit 27 causes the display unit to display a second screen that allows the candidate for the input information related to the input field to be changed to other data. In the present embodiment, when the input reception unit 28 receives an input for changing the candidate for the input information related to the input field displayed on the first screen on the first screen, the second screen is displayed. On the second screen, one or more data that can be the input information related to the input field are displayed in a selectable manner. For example, on the second screen, among the plurality of read data corresponding to the input field, the read data not selected as a candidate for the input information by the candidate selection unit 25 is displayed in a selectable manner. Thus, the user can select read data other than the data selected by the candidate selection unit 25 as the input information (candidate for the input information).

[0062] Also, for example, assume a case where the read data corresponding to the input item includes text data such as recognition result data and image data, which are different types of data. In this case, if the read data selected by the candidate selection unit 25 is text data, at least the image data can be displayed on the second screen in a selectable manner. On the other hand, if the read data selected by the candidate selection unit 25 is image data, at least the text data can be displayed on the second screen in a selectable manner. Thus, the user can select data of a format (type) different from the format (type) of the read data selected by the candidate selection unit 25 as input information (candidates for input information).

[0063] Note that the data selected by the candidate selection unit 25 may also be displayed on the second screen in a selectable manner. Thus, for example, when the candidate for the input information is changed from the data once selected by the user through the candidate selection unit 25 to other data and then the candidate for the input information is desired to be returned to the data selected by the candidate selection unit 25 again, it becomes possible to select the said data. When one piece of data is selected by the user on the second screen, the candidate for the input information is changed to this selected one piece of data. Also, as described above, in the present embodiment, different types of data (multiple types of data) with different data formats (e.g., image or text, etc.) and different data acquisition sources (e.g., surface (front or back) of the medium or IC chip) are displayed on the second screen, but multiple pieces of data may be displayed for one data type. For example, as the recognition result data on the surface, multiple pieces of data (multiple recognition result data obtained from multiple character recognition engines) may be displayed on the second screen.

[0064] FIG. 8 is a diagram showing an example (1) of the second screen according to the present embodiment. The second screen shown in FIG. 8 is a screen displayed when the "Change" button related to the address item is pressed by the user on the first screen shown in FIG. 6. In the example of FIG. 8, a plurality of read data 41 (41a, 41b, 41c) corresponding to the address item are displayed on the second screen. Specifically, the image data (cut-out image of the surface) 41a selected by the candidate selection unit 25 and two recognition character strings (OCR results) 41b and 41c which are data not selected by the candidate selection unit 25 are displayed. These two OCR results are recognition result data read by different character recognition engines. "〇〇 Prefecture □□ City △△ Town 1-cho 2-ban 10-go" is the first OCR result by the first character recognition engine, and "〇〇 Prefecture □□ City △△ Town 1-cho 2-ban 70-go" is the second OCR result by the second character recognition engine. Thus, when a plurality of OCR results can be acquired as the read data corresponding to the input item, the user can also select input information (candidates for input information) from the plurality of OCR results on the second screen. However, even when a plurality of OCR results can be acquired, only some of the OCR results among the plurality of OCR results (for example, the OCR result with the highest reliability score, etc.) may be displayed on the second screen.

[0065] Also, as shown in FIG. 8, on the second screen, there are buttons for selecting data to be input information (candidates for input information) (a "Select" button), a button for closing the second screen to stop changing the candidates for input information (a "Cancel" button), and a button for determining the selection of data (a "Confirm Selection" button). The user can press (select) these buttons by touch operation, mouse operation, etc. In the example of FIG. 8, when the "Select" button is pressed, a check mark is displayed beside the text "Select" of the pressed "Select" button. In FIG. 8, since the first OCR result has been selected by the user, a check mark is displayed on the selection button related to the first OCR result. This makes it easier for the user to visually recognize the selected data. When the "Confirm Selection" button is pressed, the change of the input information (candidates for input information) is determined, and as described above, in accordance with the content changed on the second screen, the data displayed as candidates for the input information related to the input item on the first screen is changed.

[0066] Note that when the second display control unit 27 displays a plurality of data that can be input information related to the input item on the second screen, it may display them in the display order according to the priorities of the plurality of data defined in the priority data. For example, the data with a higher priority may be displayed above. For example, as shown in FIG. 8, the data with the highest priority, that is, the image data selected by the candidate selection unit 25, is displayed at the top. In this way, by displaying a plurality of data in the display order according to their respective priorities, it becomes easier for the user to visually recognize which information is appropriate (accurate) to use as the input information.

[0067] Also, the second screen is not limited to the screen configuration shown in FIG. 8, and may have any screen configuration as long as it is a screen on which the input information (candidates for input information) can be changed before form generation. Therefore, for example, when the selection (change) is determined by the user pressing the "Select" button on the second screen, the "Confirm Selection" button may not be displayed on the second screen.

[0068] FIG. 9 is a diagram showing an example (2) of the second screen according to the present embodiment. In the second screen shown in FIG. 9, the display (button) for selecting data is different from that in the second screen shown in FIG. 8. In the example of FIG. 9, radio buttons 51 (51a, 51b, 51c) are displayed as buttons for selecting data. The user can press (select) the radio button 51 by a touch operation, a mouse operation, or the like. Also, in the example of FIG. 9, according to the selection state of the radio button 51, characters such as "print" or "do not print" are displayed next to the radio button 51. Also in FIG. 9, as in FIG. 8, when the first OCR result is selected by the user, the radio button 51b related to the first OCR result is in a selected state, and the character "print" is displayed beside it. Further, as shown in FIG. 9, a display (mark) 53 for enlarging and displaying the image represented by the image data may be displayed on the second screen. By pressing this mark, the image can be enlarged and confirmed in detail. Note that the display for selecting data may be any other arbitrary display (button) such as a check box instead of a radio button.

[0069] FIG. 10 is a diagram showing an example (3) of the second screen according to the present embodiment. Similar to FIG. 8, the second screen shown in FIG. 10 is a screen displayed when the "Change" button related to the address item is pressed by the user on the first screen shown in FIG. 6. In the second screen shown in FIG. 10, the data that can be input information (candidates for input information) displayed is different from the second screen shown in FIG. 8. Specifically, in the example of FIG. 10, similar to FIG. 8, on the second screen, in addition to the image data (cut-out image of the surface) 41a selected by the candidate selection unit 25 and the recognition character string (OCR result) 41b which is the data not selected by the candidate selection unit 25 being displayed, a display (display of the character string "Input with soft keyboard" within a frame) 43 for indicating (explaining) editing or inputting text with the soft keyboard and a display (display of the character string "Do not print (Write by hand)" within a frame) 45 for indicating (explaining) that printing on the form is not performed (writing by hand) are displayed. That is, on the second screen shown in FIG. 10, in addition to the read data 41a and 41b being displayed, character strings (displays) and the like indicating (explaining) other data (text data to be input or edited by the user, blank data (corresponding to not printing), etc.) that can be input information related to the input item are displayed. Note that the display (character string) 43 and the display (character string) 45 are examples and may be any other arbitrary character strings (phrases).

[0070] When the "Select" button related to the display 43 of "Input with Soft Keyboard" is pressed, an editing screen described later is displayed, and the user can edit or input text on this editing screen (see FIG. 11 described later). When the text to be input information (candidates for input information) is determined on the editing screen, for example, this determined text is displayed within the frame instead of the display 43 of "Input with Soft Keyboard" on the second screen shown in FIG. 10. When the "Select and Determine" button is pressed in this state, this input or edited text is determined as a new candidate and is displayed on the first screen (displayed instead of 33b in FIG. 6). Also, when the "Select and Determine" button is pressed while the "Select" button related to the display 45 of "Do Not Print (Write by Hand)" is pressed, it is determined that nothing is input as the candidate for the input information related to the input item, and on the first screen, nothing is displayed in the area corresponding to the input item (for example, blank data is displayed instead of 33b in FIG. 6). Thus, on the screen shown in FIG. 10, as data that can be input information (candidates for input information) related to the input item, in addition to the read data (41a, 41b) corresponding to the input item, other data (data input or edited by the user, blank data, etc.) is displayed in a selectable manner. Note that the editing screen may be displayed in an overlay form on the second screen, or may be displayed in a form that transitions from the second screen to the editing screen. Also, in this embodiment, in addition to the display of data that can be input information related to the input item (display of read data, input or edited text, display indicating blank data), buttons that can select the data are displayed, so that the data is displayed in a selectable manner. However, as long as the data that can be input information is displayed in a selectable manner, it is not limited to this example. For example, the user may be able to select the display of the data that can be input information itself by a touch operation, a mouse operation, or the like.

[0071] FIG. 11 is a diagram showing an example of an editing screen according to the present embodiment. The editing screen shown in FIG. 11 is a screen displayed by selecting soft keyboard input on the screen of FIG. 10. The user can input or edit text in a predetermined area (the area within the frame) 61 using the soft keyboard shown at the bottom of the editing screen shown in FIG. 11, so that desired data can be used as candidates for input information. For example, when the area 61 within the frame is blank when the editing screen is displayed, the user can input desired text into the area 61 within the frame. Further, for example, when text data (recognition result data obtained by OCR processing or text data read from an IC chip) is displayed in the area 61 within the frame when the editing screen is displayed, the user can edit the text data displayed in the area 61 within the frame so as to be the desired text.

[0072] Further, on the editing screen, as shown in FIG. 11, buttons for determining the text input or edited by the user (the "Determine" button) and buttons for canceling (aborting) the text input or editing (the "Cancel" button) are displayed. When the "Determine" button is pressed, the editing screen is closed, and the text displayed (input) in the area 61 within the frame when the "Determine" button was pressed is displayed in the area where the character string 43 of "Input with Soft Keyboard" on the second screen was displayed (displayed instead of the display 43 in FIG. 10). When the "Cancel" button is pressed, the editing screen may be closed and the display on the second screen may be restored.

[0073] The input reception unit 28 receives various inputs from the user. For example, the input reception unit 28 receives an input from the user for changing candidates for input information related to an input item on the first screen. Further, for example, the input reception unit 28 receives an input from the user for determining candidates for input information related to each input item as input information to be embedded in a form format on the first screen. Further, for example, the input reception unit 28 receives an input from the user for changing candidates for input information on the second screen.

[0074] Based on the read data read from the medium to be read (the data acquired by the data acquisition unit 24), the form generation unit 29 generates a form (form data) by embedding input information in one or more input items (the embedding positions of the input information for the input items) of the form format. In the present embodiment, the form generation unit 29 embeds (arranges) data (read data read from the medium, data such as fixed characters, etc.) as the input information of the input items in one or more form formats selected by the form format selection unit 23, thereby generating one or more forms (form data) in which the data is described. In the present embodiment, when the input reception unit 28 receives an input from the user for determining the candidate of the input information related to each input item as the input information to be embedded in the form format, the form generation unit 29 generates a form by embedding the data that is the candidate of the input information in the form format as the input information. Note that any method may be used as the method of embedding the data that is the input information in the embedding position (embedding area). For example, a method of arranging a text object related to the data that is the input information in the underlying data (or the replicated underlying data) that serves as the basis for the layout of the form included in the form format, or in the case of Excel, Word, etc., a method of inputting the data that is the input information in the specified cell (area) may be used. In this way, the form generation unit 29 generates a form by transcribing the information read from the medium into the form format and the like.

[0075] In addition, when the data to be embedded as input information related to an input item is large-sized image data such as an image of the remarks column on the back of a driver's license, it may not fit into the embedding area related to the input item in its original size. Also, although it is possible to embed such large-sized image data into the embedding area by reducing it (fitting it into the embedding area), if it is reduced too much, there may be a problem that the readability of the characters described in the image decreases. Therefore, in the present embodiment, in the case of large-sized image data whose readability of the characters described by embedding in the embedding area related to a predetermined input item becomes low, it is possible to embed it in another area instead of the embedding area. For example, when it is necessary to reduce the image data to a predetermined ratio (e.g., half) or less in order to embed it so that it fits within the embedding area related to the input item, it is presumed that there is difficulty in the readability of the characters described in the image, and the image data is embedded in another area instead of the embedding area. Also, for example, when the image data cannot fit within the embedding area unless it is reduced, the image data may be embedded in another area instead of the embedding area.

[0076] That is, when the data to be embedded as input information in the form generation unit 29 is image data and the image size of the image data does not satisfy a predetermined size condition in relation to the size of the embedding area related to the input item, this image data can be embedded not in the embedding area but in another area (a previously specified area). In the above example, this predetermined size condition is conditional on the reduction rate of the image data required to embed the image data so as to fit within the embedding area related to the input item exceeding a predetermined value (for example, 50%). Note that the predetermined value serving as the threshold for the reduction rate is not limited to 50% and may be any other value (reduction rate). Note that the predetermined size condition is not limited to the conditions described above and may be any condition as long as it is conditional on the image data not being reduced too much (not becoming too low in readability) to be embedded in the embedding area related to the input item. Additionally, the predetermined size condition may be that the image size of the image data is smaller than the size of the embedding area related to the input item (the image data fitting within the embedding area in the non-reduced state), etc.

[0077] In addition, when the form generation unit 29 embeds this image data in an area other than the embedding area related to the input item, a predetermined text (character string) may be embedded in the embedding area of the input information related to the input item. For example, as the predetermined text, it is possible to embed text suggesting that the input information (image data) has been embedded in an area other than the embedding area related to the input item, that is, text for guiding to refer to the other area. Note that this "other area" is not limited to other areas on the same page as the embedding area related to the input item and may be an area within a page different from the embedding area related to the input item.

[0078] FIG. 12 is a diagram showing an example (1) of embedding image data according to the present embodiment into a form format. In FIG. 12, as the form format, a format 71 related to an application form for a family register (invoice for a copy of a family register, etc.) is exemplified. Further, as input information related to the input item "address" in the form (data to be embedded as input information), a cut-out image (image data) of the remarks column on the back of the driver's license (the column where the new address is described) is exemplified. In FIG. 12, a blank space 73 as an embedding area (input column) related to the input item "address" is arranged in the area (right column) within the frame immediately to the right of the frame in which the item name "address" is described. In the example of FIG. 12, the back remarks column image is embedded in an area (embedding area related to the input item "address") 73 that is previously designated to embed the input information related to this input item "address". This is because it is determined that the size of the back remarks column image satisfies a predetermined size condition in relation to the size of the embedding area 73 related to the input item (there is no difficulty in readability even when embedded in the embedding area). Thus, when the image data can be shrunk and embedded in the embedding area related to the input item without difficulty in readability, etc., the image data is embedded in the embedding area.

[0079] FIG. 13 is a diagram showing an example (2) of embedding image data according to the present embodiment into a form format. Also in FIG. 13, similar to FIG. 12, as the form format, the format 71 related to the application form for a family register (invoice for a copy of a family register, etc.) is exemplified. Further, as input information related to the input item "address" of the form (data to be embedded as input information), an extracted image (image data) of the remarks column on the back of the driver's license (the column where the new address is described) is exemplified. In the example of FIG. 13, the back remarks column image is embedded in an area other than the area 73 that is not pre-specified to embed the input information related to the input item "address". Specifically, the back remarks column image is embedded in the margin area 75 on the right side of the area for describing the purpose of use. This is because it is determined that the size of the back remarks column image does not satisfy the predetermined size condition in relation to the size of the embedding area 73 related to the input item (there would be difficulty in readability if embedded in the embedding area). Thus, when it is difficult to readability, etc. if the image data is reduced and embedded so as to fit into the embedding area related to the input item, the image data may be embedded in an area other than the embedding area. Note that, for the other area where the image data is to be embedded when the predetermined size condition is not satisfied, an area having a size that allows the image data to be embedded in a size with no difficulty in readability is specified. Also, for the other area, it may be pre-specified in advance in the form format or other definition information, or the form generation unit 29 may detect the margin area in the form format (layout) and specify it by determining that the predetermined size condition is not satisfied.

[0080] FIG. 14 is a diagram showing an example of embedding a predetermined text into a form format according to the present embodiment. FIG. 14 exemplifies a case where, in the case of the example shown in FIG. 13, an example of a predetermined text, "Refer to Remarks Column", is embedded (printed) in the embedding area (input field) 73 related to the input item "address". As shown in FIG. 14, by embedding texts such as "Refer to Remarks Column" and "Refer to End of Invoice" in the embedding area related to the input item "address", a person who checks the created form can easily confirm if the address information is described in another area.

[0081] The output unit 30 outputs the form data (including the form data after data conversion) generated by the form generation unit 29 and / or outputs (or gives an output instruction) the form based on the form data. Various methods may be used for the output method and output instruction method of the form data or the form based on the form data. For example, the output unit 30 outputs (transmits) the form data (image data, PDF data, etc.) or the form data after data conversion to an external device or external system that is communicably connected to the information processing apparatus 1 via a network or other communication means. Note that the form data after data conversion is, for example, form data (CSV information, Text information, etc.) for which CSV conversion, Text conversion, etc. has been performed. In an external device (for example, a database input system, etc.) that has acquired data such as CSV information, after visual confirmation by the user of the acquired data, etc., the database may be updated based on the data. For example, when the CSV information includes information related to an address change, the information stored in the local government's database may be updated based on the information of the address after the change.

[0082] Also, as another output method, for example, when the information processing apparatus 1 includes the output unit 30 as a printing function, the output unit 30 prints (outputs) a paper-based form based on the form data generated by the form generation unit 29. Also, for example, when the information processing apparatus 1 is communicably connected to a printing device (printer) via a network or other communication means, the output unit 30 transmits the form data to the printing device and then gives an instruction to print a paper-based form based on the form data.

[0083] The reading unit 81 discriminates the type (category) of the medium on which user information is recorded. In the present embodiment, in order to exemplify an identification document as the medium on which user information is recorded, the reading unit 81 determines (acquires) which identification document among various identification documents such as a My Number card, a driver's license, a residence card, and a passport the medium to be read is (the type of the identification document) from the medium. Various methods may be used to discriminate the type of the identification document. For example, a method of discriminating the type of the identification document by analyzing the arrangement of characters, photographs, etc. described in the identification document by reading the identification document (medium) may be used. As another method, a method of acquiring information indicating the type of the identification document recorded on the IC chip by reading the IC chip mounted on the identification document (a method using an IC reader) may be used. As another method, a method of reading an image of the identification document and performing OCR processing on the image to acquire characters indicating the type of the identification document described in the identification document (for example, characters such as "driver's license") may be used. Note that in the present embodiment, the reading unit 81 discriminates (acquires) the type of the identification document using the identification document, but the type of the medium to be read (the type of the identification document) may be acquired by accepting an input (selection input, etc.) of the type of the medium to be read from the user. Note that in the present embodiment, the type of the medium is discriminated in the reading device 9, but the type of the medium may be discriminated in the information processing device 1.

[0084] Further, the reading unit 81 acquires information such as user information recorded on the medium (such as an identity document) by reading the medium. The reading unit 81 is a functional unit corresponding to the reading unit 91. Various methods may be used to acquire the information recorded on the medium (such as an identity document) by the reading unit 81. For example, the reading unit 81 acquires an image of the surface of the medium (hereinafter referred to as a "medium image") by reading the medium with a camera, scanner, or the like. Then, in the reading device 9, by providing a medium definition storage unit (not shown) that stores medium definition information in which the information description positions (regions) of the recorded items are defined for each type of identity document, the reading unit 81 refers to this medium definition information and acquires an image (extracted image) obtained by cutting out a predetermined region (for example, a character recognition region) from the medium image according to the type of the medium to be read. Thus, an extracted image corresponding to the information of the items described on the surface of the medium is obtained. Then, the reading unit 81 may perform OCR processing on the acquired extracted image (partial image) to acquire the information (recognition result data that is text data) described on the surface of the medium. Note that the method of acquiring the extracted image and the recognition result data corresponding to the information (item value) of each item from the medium is not limited to the method described above, and various methods may be used. Also, in this embodiment, the case where the reading device 9 acquires the extracted image and the recognition result data is illustrated, but it is not limited to this case. That is, the reading device 9 may only acquire the medium image. In this case, the information processing device 1 may be provided with the above-described medium definition storage unit, so that the data acquisition unit 24 may acquire the extracted image of each item and perform OCR processing on the extracted image. Note that the medium definition information is not limited to the above-described information, and may include other definition information as long as it is definition information necessary for reading information from each medium. Therefore, the medium definition information may define, for example, a method for accessing an IC chip mounted on the medium (including information for access, etc.) for each medium.

[0085] Also, as another example of a method for acquiring information recorded on a medium by the reading unit 81, a method of acquiring user information recorded on an IC chip by reading the IC chip mounted on the medium may be used. Specifically, when a correct personal identification number (PIN) is input from the user, the reading unit 81 can read user information (text data and / or image data (medium image or an image cut out from the medium image)) from the IC chip mounted on the medium by an IC reader. Further, in the IC chip, for example, a partial image (cut-out image) of a portion where information such as name and address is described may be recorded. For this partial image recorded on the IC chip, if the pixels are not coarse, it is also possible to obtain recognition result data as user information by performing OCR processing on this partial image.

[0086] The personal authentication unit 82 performs personal authentication using the user information recorded on the medium. Various methods may be used for personal authentication. For example, the personal authentication unit 82 collates the face photo data recorded in the personal authentication document (IC chip) with the face photo of the user taken by the camera 92 of the reading device 9. For example, when the face features obtained from each of these face photos are different, it is determined that "personal authentication could not be performed (personal authentication is NG)". Also, for example, as a result of collating the information on the IC chip and the information on the document surface, if the two do not match (when the information (name and address) recorded on the IC chip mounted on the personal authentication document is different from the information (name and address) on the surface of the personal authentication document, when the layout of the surface of the read personal authentication document is the layout of a driver's license but the read personal authentication document does not have an IC chip mounted (a normal driver's license has an IC chip mounted), when the content of the IC chip mounted on the read personal authentication document is the content of a my number card but the layout of the surface of the read personal authentication document is a driver's license, etc.), personal authentication may be determined to be NG. Also, when the age of the user obtained from the medium is less than the specified age, personal authentication may be determined to be NG. In this embodiment, since the reading device 9 includes the personal authentication unit 82, personal authentication processing is performed in the reading device 9. However, personal authentication processing may be performed in the information processing device 1 by having the information processing device 1 include the personal authentication unit 82.

[0087] The data transmission unit 83 transmits various data (information) to the information processing device 1. Specifically, the data transmission unit 83 transmits information such as the type of medium (information related to the medium type) acquired (discriminated) by the reading unit 81, the information read by the reading unit 81 (read data), and information indicating the result of personal authentication by the personal authentication unit 82 to the information processing device 1.

[0088] Hereinafter, six processing flows (six examples) are illustrated as examples of the selection process executed by the candidate selection unit 25 with reference to Figs. 15 to 21. For each input item included in the form, any of these six processing flows can be used. In each flow, either image data or text data is selected as a candidate for input information to be displayed on the first screen by the selection process. However, as shown in Fig. 20, "blank (nothing displayed)" may be selected as a candidate for input information. In addition, the flows illustrated in Figs. 15 to 21 are examples of the selection process, and any processing flow other than those illustrated in Figs. 15 to 21 may be used as long as it uses the method described in the above-mentioned selection processes (A) to (C). Note that the selection process illustrated in each of the flowcharts illustrated in Figs. 15 to 21 is started when the medium is read, an instruction to confirm information to be input in the input item is received, or the like. The flowchart illustrated in Fig. 16 is a flow that can be used when the input item is an address item. The flowcharts shown in Fig. 15 and Fig. 17 to Fig. 21 are flows that can be used for any input item of a form. Fig. 15 to Fig. 17 show an example of selecting input information candidates for the input item "address". Fig. 15 to Fig. 19 show an example of using a driver's license as a medium. Note that it is assumed that read data corresponding to the input item is acquired in the information processing device 1 before the processes shown in the flowcharts of Fig. 15 to Fig. 21 are started. However, if read data such as a reliability score of an OCR result is not used in the selection process, read data corresponding to the input item may be acquired after the selection process.

[0089] 15 is a flowchart showing an outline of the flow of the selection process (1) according to this embodiment. This flowchart corresponds to the selection method (a) described above. In this flowchart, as the read data corresponding to the input item "address", a cut-out image (image data) relating to the address on the front of the driver's license and recognition result data (OCR result) obtained by performing character recognition processing on this image data are illustrated.

[0090] In step S101, it is determined whether the reliability score of the OCR result is equal to or lower than a threshold value. The candidate selection unit 25 determines whether the reliability score of the OCR result, which is the read data corresponding to the input item "address", is equal to or lower than a predetermined threshold value. If the reliability score of the OCR result is not equal to or lower than the predetermined threshold value (No in step S101), the process proceeds to step S102. On the other hand, if the reliability score of the OCR result is equal to or lower than the predetermined threshold value (Yes in step S101), the process proceeds to step S103.

[0091] In step S102, the OCR result is selected. The candidate selection unit 25 selects the recognition result data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". As a result, only the OCR result is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 7). Then, the process shown in this flowchart ends.

[0092] In step S103, an image is selected. The candidate selection unit 25 selects the image data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". As a result, only the image represented by the image data is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 6). Then, the process shown in this flowchart ends.

[0093] FIG. 16 is a flowchart showing an outline of the flow of the selection process (2) according to the present embodiment. This flowchart corresponds to the flowchart of the selection method (b) described above. Also in this flowchart, similar to FIG. 15, as the read data corresponding to the input item "address", an extracted image (image data) related to the address on the front surface of the driver's license and recognition result data (OCR result) obtained by performing character recognition processing on this image data are exemplified.

[0094] In step S201, it is determined whether the OCR result (recognized character string) regarding the address is included in the address database. The candidate selection unit 25 determines whether the address indicated by the recognition result data, which is the read data corresponding to the input item "address", is included in the address database. If the address indicated by the recognition result data is included in the address database (Yes in step S201), the process proceeds to step S202. On the other hand, if the address indicated by the recognition result data is not included in the address database (No in step S201), the process proceeds to step S203.

[0095] In step S202, the OCR result is selected. The candidate selection unit 25 selects the recognition result data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". As a result, only the OCR result is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 7). Then, the process shown in this flowchart ends.

[0096] In step S203, the image is selected. The candidate selection unit 25 selects the image data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". As a result, only the image represented by the image data is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 6). Then, the process shown in this flowchart ends.

[0097] FIG. 17 is a flowchart showing an outline of the flow of the selection process (3) according to the present embodiment. This flowchart corresponds to the flowcharts for the selection methods (a) and (c) described above. In this flowchart, as the read data corresponding to the input item "address", an extracted image (image data) related to the address on the surface of the driver's license and a plurality of recognition result data (a plurality of OCR results) obtained by performing character recognition processing on this image data by a plurality of character recognition engines are exemplified. Among the plurality of OCR results, the prioritized OCR result is hereinafter referred to as the first OCR result. Note that, among the plurality of OCR results, the OCR result with the highest reliability score may be used as the prioritized OCR result, or a prioritized character recognition engine may be determined in advance, and the OCR result obtained from the prioritized character recognition engine may be used as the prioritized OCR result.

[0098] In step S301, it is determined whether the reliability score of the first OCR result is less than or equal to a threshold value. The candidate selection unit 25 determines whether the reliability score of the first OCR result, which is the read data corresponding to the input item "address", is less than or equal to a predetermined threshold value. If the reliability score of the first OCR result is not less than or equal to the predetermined threshold value (No in step S301), the process proceeds to step S302. On the other hand, if the reliability score of the first OCR result is less than or equal to the predetermined threshold value (Yes in step S301), the process proceeds to step S304.

[0099] In step S302, it is determined whether a plurality of OCR results by a plurality of character recognition engines satisfy a predetermined condition. The candidate selection unit 25 determines whether the plurality of OCR results obtained as the read data corresponding to the input item "address" satisfy a predetermined condition. For example, the candidate selection unit 25 determines whether the plurality of OCR results (recognized character strings) are exactly the same. If the plurality of OCR results satisfy the predetermined condition (Yes in step S302), the process proceeds to step S303. On the other hand, if the plurality of OCR results do not satisfy the predetermined condition (No in step S302), the process proceeds to step S304.

[0100] In step S303, the OCR result is selected. The candidate selection unit 25 selects one piece of recognition result data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". For example, the recognition result data related to the prioritized OCR result or the recognition result data with the highest reliability score is selected. From this, only one OCR result is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 7). Thereafter, the processing shown in this flowchart ends.

[0101] In step S304, an image is selected. The candidate selection unit 25 selects image data, which is the read data corresponding to the input item "address", as a candidate for the input information of the input item "address". From this, only the image represented by the image data is displayed on the first screen as a candidate for the input information of the input item "address" (see FIG. 6). Thereafter, the processing shown in this flowchart ends.

[0102] FIGS. 18 and 19 are flowcharts showing an outline of the flow of the selection process (4) according to the present embodiment. This flowchart corresponds to a method combining the selection methods (a) to (c) described above. As described above, this flowchart is a processing flow that can be used for any input item. Hereinafter, the input item for which candidates for input information are selected will be referred to as the "target input item". Further, in this flowchart, as the read data corresponding to the target input item, image data (cropped image) and a plurality of recognition result data (a plurality of OCR results) obtained by performing character recognition processing on this image data by a plurality of character recognition engines are exemplified. Note that, similar to the case of FIG. 17, among the plurality of OCR results, the prioritized OCR result is referred to as the first OCR result.

[0103] In step S401, it is determined whether the reliability score of the first OCR result is less than or equal to a threshold. The candidate selection unit 25 determines whether the reliability score of the first OCR result, which is the read data corresponding to the target input item, is less than or equal to a predetermined threshold. If the reliability score of the first OCR result is not less than or equal to the predetermined threshold (No in step S401), the process proceeds to step S402. On the other hand, if the reliability score of the first OCR result is less than or equal to the predetermined threshold (Yes in step S401), the process proceeds to step S405.

[0104] In step S402, it is determined whether a plurality of OCR results by a plurality of character recognition engines satisfy a predetermined condition. The candidate selection unit 25 determines whether the plurality of OCR results obtained as the read data corresponding to the target input item satisfy a predetermined condition. If the plurality of OCR results satisfy the predetermined condition (Yes in step S402), the process proceeds to step S403. On the other hand, if the plurality of OCR results do not satisfy the predetermined condition (No in step S402), the process proceeds to step S404.

[0105] In step S403, the candidate selection unit 25 determines that the OCR priority indicating the degree of prioritizing the OCR result is "high". In step S404, the candidate selection unit 25 determines that the OCR priority is "medium". In step S405, the candidate selection unit 25 determines that the OCR priority is "low". After steps S403, S404, and S405, the process proceeds to step S406.

[0106] In step S406, it is determined whether the input item is related to an address. The candidate selection unit 25 determines whether the target input item is an address item. If the target input item is an address item (Yes in step S406), the process proceeds to step S407. On the other hand, if the target input item is not an address item (No in step S406), the process proceeds to step S410. Note that when the target input item is not an address item, the process may proceed to step S410 after another determination process (selection process) different from step S407.

[0107] In step S407, it is determined whether the OCR result (recognized character string) is included in the address database. The candidate selection unit 25 determines whether the address (OCR result) indicated by the recognition result data, which is the read data corresponding to the target input item, is included in the address database. Note that the OCR result to be compared with the address database may be all the OCR results corresponding to the target input item, or may be one OCR result such as the first OCR result or the OCR result with the highest reliability score. When the address indicated by the recognition result data is included in the address database (Yes in step S407), the process proceeds to step S408. On the other hand, when the address indicated by the recognition result data is not included in the address database (No in step S407), the process proceeds to step S409.

[0108] In step S408, the candidate selection unit 25 increases the OCR priority by one level. Specifically, when the OCR priority was determined to be "low" in step S405, the OCR priority is changed to "medium", and when the OCR priority was determined to be "medium" in step S404, the OCR priority is changed to "high". Note that when the OCR priority was determined to be "high" in step S403, the OCR priority may remain unchanged at "high" or may be changed to a priority higher than "high". When the OCR priority is a parameter represented by a numerical value, in step S408, the OCR priority is set to a value higher by a predetermined value.

[0109] In step S409, the candidate selection unit 25 decreases the OCR priority by one level. Specifically, when the OCR priority was determined to be "high" in step S403, the OCR priority is changed to "medium", and when the OCR priority was determined to be "medium" in step S404, the OCR priority is changed to "low". Note that when the OCR priority was determined to be "low" in step S405, the OCR priority may remain unchanged at "low" or may be changed to a priority lower than "low". When the OCR priority is a parameter represented by a numerical value, in step S409, the OCR priority is set to a value lower by a predetermined value.

[0110] In step S410, it is determined whether the OCR priority is "medium" or higher. The candidate selection unit 25 determines whether the OCR priority determined by the processes of steps S401 to S409 is "medium" or higher, that is, "medium" or "high" (including the level above "high" if there is a level above "high"). If the OCR priority is "medium" or higher (Yes in step S410), the process proceeds to step S411. If the OCR priority is lower than "medium" (No in step S410), the process proceeds to step S412.

[0111] In step S411, the OCR result is selected. The candidate selection unit 25 selects one piece of recognition result data, which is the read data corresponding to the target input item, as a candidate for the input information related to the target input item. For example, the recognition result data related to the prioritized OCR result or the recognition result data with the highest reliability score is selected. As a result, only one OCR result is displayed on the first screen as a candidate for the input information of the target input item (see FIG. 7). Then, the process shown in this flowchart ends.

[0112] In step S412, an image is selected. The candidate selection unit 25 selects image data, which is the read data corresponding to the target input item, as a candidate for the input information related to the target input item. As a result, only the image represented by the image data is displayed on the first screen as a candidate for the input information of the target input item (see FIG. 6). Then, the process shown in this flowchart ends.

[0113] Figure 20 is a flowchart showing an overview of the flow of the selection process (5) according to the present embodiment. This flowchart corresponds to the selection method of (A) described above and shows an overview of the flow of the selection process using priority data. In this flowchart, a case where priority data is stored for each type of medium and each input item is exemplified. Specifically, a case where the first priority data shown in FIG. 3 is stored corresponding to the case where the medium is a driver's license and the input item is an address item is exemplified. Since the candidate selection unit 25 selects candidates for the input information according to the priorities shown in the priority data of FIG. 3, in the flowchart of FIG. 20, the presence or absence of information is confirmed in descending order of priority in the priority data of FIG. 3, and candidates for the input information are determined. That is, according to the priorities determined in the priority data, the order of confirming the presence or absence of information in steps S505, S507, S509, and S511 in FIG. 20 is determined. Note that priority data other than the first priority data may be stored corresponding to other media and other input items. As described above, this flowchart is a processing flow that can be used for any input item, and hereinafter, the input item for which candidates for the input information are selected will be referred to as the "target input item".

[0114] In step S501, the type (ticket type) of the medium to be read is determined. The candidate selection unit 25 determines the type of the medium to be read by referring to the type of the medium to be read acquired by the data acquisition unit 24. If the medium to be read is not a driver's license ( "other than driver's license" in step S501), the process proceeds to step S502. If the medium to be read is a driver's license ( "driver's license" in step S501), the process proceeds to step S503.

[0115] In step S502, selection processing related to other ticket types is performed. The candidate selection unit 25 performs selection processing related to media types other than a driver's license. In this flowchart, since the selection processing in the case where the media type is a driver's license is illustrated in detail, the detailed description of the selection processing in step S502 is omitted. However, in step S502, any of the selection processing methods (A) to (C) described above may be used. Also, in step S502, selection processing may be further performed according to the type of the media. Then, the processing shown in this flowchart ends.

[0116] In step S503, the type of the target input item is determined. The candidate selection unit 25 determines whether the target input item is an address item. If the target input item is not an address item ( "other than address item" in step S503), the process proceeds to step S504. If the target input item is an address item ( "address item" in step S503), the process proceeds to step S505.

[0117] In step S504, selection processing related to other input items in the case where the media type is a driver's license is performed. The candidate selection unit 25 performs selection processing related to input items other than the address item. In this flowchart, since the selection processing in the case where the input item is an address item is illustrated in detail, the detailed description of the selection processing in step S504 is omitted. However, in step S504, any of the selection processing methods (A) to (C) described above may be used. Also, in step S504, selection processing may be further performed according to the type of the input item. Then, the processing shown in this flowchart ends.

[0118] In step S505, it is determined whether there is information in the reverse side remarks column image. Since the candidate selection unit 25 determines that the reverse side remarks column image has a priority of 1 in the first priority data (priority table shown in FIG. 3) corresponding to the type of medium to be read (driver's license) and the target input item (address item), first, as the read data corresponding to the target input item (address item), it is determined whether the information in the reverse side remarks column image has been acquired (presence or absence of image information). In this embodiment, when the image data itself has not been acquired due to settings, operations, etc., or when the image data has been acquired but nothing is described in the image represented by the image data (it is blank), or when there is information described in the image represented by the acquired image data but the described content is not input information related to the target input item, it is determined that there is no image information. Therefore, in step S505, when the reverse side is not set to be read and thus the reverse side is not read and the image data of the reverse side remarks column itself has not been acquired, or when the image of the reverse side remarks column has been acquired but no information is added to the reverse side remarks column, or when the image of the reverse side remarks column has been acquired but there is a description such as "undetermined (to be described on the reverse side after notification)" in the reverse side remarks column, it is determined that there is no information in the reverse side remarks column image. When it is determined that there is information in the reverse side remarks column image (Yes in step S505), the process proceeds to step S506. On the other hand, when it is determined that there is no information in the reverse side remarks column image (No in step S505), the process proceeds to step S507. Note that the determination of the presence or absence of image information is not limited to the above example, and if the image data has been acquired, it may be determined that there is image information.

[0119] In step S506, the reverse side remarks column image is selected. The candidate selection unit 25 selects the reverse side remarks column image (image data), which is the read data corresponding to the target input item, as a candidate for the input information related to the target input item (address item). As a result, only the reverse side remarks column image is displayed on the first screen as a candidate for the input information related to the target input item. Then, the process shown in this flowchart ends.

[0120] In step S507, it is determined whether there is information on the cut-out image of the front side. Since the cut-out image of the front side has a priority of 2 in the priority table shown in FIG. 3, the candidate selection unit 25, following the confirmation of the presence or absence of the back-side remarks column image (step S505), determines whether a cut-out image related to the address on the front side (the presence or absence of image information) has been acquired as the read data corresponding to the target input item (address item). Note that the method for determining the presence or absence of image information is substantially the same as the method described in step S505, so the description is omitted. If it is determined that there is information on the cut-out image related to the address on the front side (Yes in step S507), the process proceeds to step S508. On the other hand, if it is determined that there is no information on the cut-out image related to the address on the front side (No in step S507), the process proceeds to step S509.

[0121] In step S508, the cut-out image related to the address on the front side is selected. The candidate selection unit 25 selects the cut-out image (image data) related to the address on the front side, which is the read data corresponding to the target input item, as a candidate for the input information related to the target input item (address item). As a result, only the cut-out image related to the address on the front side is displayed on the first screen as a candidate for the input information related to the target input item. Then, the process shown in this flowchart ends.

[0122] In step S509, it is determined whether there is information on the text data in the IC chip. Since the text data in the IC chip has a priority of 3 in the priority table shown in FIG. 3, the candidate selection unit 25, following the confirmation of the presence or absence of the cut-out image related to the address on the front side (step S507), determines whether text data in the IC chip (for example, address information such as "1-2-10, △△-cho, □□-shi, 〇〇-prefecture") has been acquired as the read data corresponding to the target input item (address item) (the presence or absence of text data information). If it is determined that text data corresponding to the target input item (address item) has been acquired from the IC chip (Yes in step S509), the process proceeds to step S510. On the other hand, if it is determined that text data corresponding to the target input item (address item) has not been acquired from the IC chip (No in step S509), the process proceeds to step S511.

[0123] In step S510, the text data in the IC chip is selected. The candidate selection unit 25 selects, as a candidate for the input information related to the target input item (address item), the text data in the IC chip, which is the read data corresponding to the target input item. As a result, only the text represented by the text data in the IC chip is displayed on the first screen as a candidate for the input information related to the target input item. Then, the processing shown in this flowchart ends.

[0124] In step S511, it is determined whether there is information on the OCR result (recognition result data) on the surface. Since the recognition result data on the surface has a priority of 4 in the priority table shown in FIG. 3, the candidate selection unit 25 determines, following the confirmation of the presence or absence of the text data in the IC chip (step S509), whether the recognition result data on the surface has been acquired as the read data corresponding to the target input item (address item) (the presence or absence of information on the recognition result data). If it is determined that the recognition result data on the surface corresponding to the target input item (address item) has been acquired (Yes in step S511), the processing proceeds to step S512. On the other hand, if it is determined that the recognition result data on the surface corresponding to the target input item (address item) has not been acquired (No in step S511), the processing proceeds to step S513.

[0125] In step S512, the recognition result data on the surface is selected. The candidate selection unit 25 selects, as a candidate for the input information related to the target input item (address item), the recognition result data on the surface, which is the read data corresponding to the target input item. As a result, only the text (recognized character string) represented by the recognition result data on the surface is displayed on the first screen as a candidate for the input information related to the target input item. Then, the processing shown in this flowchart ends.

[0126] In step S513, a blank is selected. The candidate selection unit 254 selects, as a candidate for the input information related to the target input item (address item), a blank, that is, to display no data. As a result, on the first screen, no data is displayed as a candidate for the input information related to the target input item. Then, the processing shown in this flowchart ends.

[0127] FIG. 21 is a flowchart showing an outline of the flow of the selection process (6) according to the present embodiment. This flowchart corresponds to the selection method of (C) described above. In this flowchart, a case where priority data is stored for each type of medium and each input item is exemplified. Specifically, a case where the second priority data shown in FIG. 4 is stored corresponding to the case where the medium is a my number card and the input item is an address item is exemplified. Since the candidate selection unit 25 selects candidates for the input information according to the priorities shown in the priority data of FIG. 4, in the flowchart of FIG. 21, the presence or absence of information is confirmed in descending order of priority in the priority data of FIG. 4, and candidates for the input information are determined. Note that priority data other than the second priority data may be stored corresponding to other media and other input items. As described above, this flowchart is a processing flow that can be used for any input item, and hereinafter, the input item for which candidates for the input information are selected will be referred to as the "target input item".

[0128] In step S601, the type (type of ticket) of the medium to be read is determined. The candidate selection unit 25 determines the type of the medium to be read by referring to the type of the medium to be read acquired by the data acquisition unit 24. If the medium to be read is not a my number card ("other than my number card" in step S601), the process proceeds to step S602. If the medium to be read is a my number card ("my number card" in step S601), the process proceeds to step S603.

[0129] In step S602, selection processing related to other ticket types is performed. The candidate selection unit 25 performs selection processing related to media types other than the My Number card. In this flowchart, in order to illustrate in detail the selection processing when the media type is the My Number card, the detailed description of the selection processing in step S602 is omitted. However, in step S602, any of the selection processing methods (A) to (C) described above may be used. Also, in step S602, further selection processing may be performed according to the type of the media. Then, the processing shown in this flowchart ends.

[0130] In step S603, the type of the target input item is determined. The candidate selection unit 25 determines whether the target input item is an address item. If the target input item is not an address item ("other than address item" in step S603), the process proceeds to step S604. If the target input item is an address item ("address item" in step S603), the process proceeds to step S605.

[0131] In step S604, selection processing related to other input items is performed when the media type is the My Number card. The candidate selection unit 25 performs selection processing related to input items other than the address item. In this flowchart, in order to illustrate in detail the selection processing when the input item is an address item, the detailed description of the selection processing in step S604 is omitted. However, in step S604, any of the selection processing methods (A) to (C) described above may be used. Also, in step S604, further selection processing may be performed according to the type of the input item. Then, the processing shown in this flowchart ends.

[0132] In step S605, it is determined whether there is information on text data in the IC chip. Since the second priority data (priority table shown in FIG. 4) corresponding to the type of medium to be read (My Number card) and the target input item (address item) indicates that the text data in the IC chip has a priority of 1, first, the candidate selection unit 25 determines whether the text data in the IC chip has been acquired as the read data corresponding to the target input item (address item) (whether there is information on the text data). If it is determined that the text data corresponding to the target input item (address item) has been acquired from the IC chip (Yes in step S605), the process proceeds to step S606. On the other hand, if it is determined that the text data corresponding to the target input item (address item) has not been acquired from the IC chip (No in step S605), the process proceeds to step S607.

[0133] In step S606, the text data in the IC chip is selected. The candidate selection unit 25 selects the text data in the IC chip, which is the read data corresponding to the target input item, as a candidate for the input information related to the target input item (address item). As a result, only the text represented by the text data in the IC chip is displayed on the first screen as a candidate for the input information related to the target input item. Then, the process shown in this flowchart ends.

[0134] In step S607, a process for selecting either the OCR result or the cut-out image is performed. Since both the cut-out image on the surface and the recognition result data on the surface (OCR result) have a priority of 2 in the priority table shown in FIG. 4, the candidate selection unit 25 selects either the cut-out image on the surface or the recognition result data on the surface (OCR result) as a candidate for the input information. Note that, as a method for selecting either the OCR result or the cut-out image, in step S607, any one of the four processing flows shown in FIGS. 15 to 19 may be executed. However, when the processing flows shown in FIGS. 18 and 19 are executed in step S607, since the process of step S406 has been executed in step S603, the process of step S406 does not need to be executed. Then, the process shown in this flowchart ends.

[0135] Figure 22 is a flowchart showing an overview of the flow of the image embedding process according to the present embodiment. The process shown in the flowchart shown in Figure 22 is started on the occasion of receiving an input for determining, as input information to be embedded in the form format, candidates for input information related to the input item, etc. In Figure 22, the address item is exemplified as the input item.

[0136] In step S701, it is determined whether the information to be embedded is the back reference column image. The form generation unit 29 determines whether the information (data) determined as the input information to be embedded in the form format is the back reference column image (image data). If the information to be embedded as the input information is the back reference column image (Yes in step S701), the process proceeds to step S702. On the other hand, if the information to be embedded as the input information is not the back reference column image (No in step S701), the process proceeds to step S705.

[0137] In step S702, when embedding the back reference column image in the designated area, it is determined whether it is necessary to reduce the back reference column image to half (a predetermined ratio) or less. The form generation unit 29 determines whether it is necessary to reduce the back reference column image to half or less by comparing the size of the back reference column image determined as the input information with the designated area (the embedding area related to the input item). If it is necessary to reduce the back reference column image to half or less (Yes in step S702), the process proceeds to step S703. On the other hand, if it is not necessary to reduce the back reference column image to half or less (No in step S702), the process proceeds to step S705. Note that the "half" which is the size condition used as the threshold value is an example, and as described above, a predetermined size condition may be adopted.

[0138] In step S703, a predetermined phrase (character string) is described in the designated area. The form generation unit 29 embeds a predetermined phrase such as "Refer to the remarks column" in the area designated to embed the input information related to the input item (address item) (see Figure 14). Then, the process proceeds to step S704.

[0139] In step S704, the back-remark column image is embedded in another area. The form generation unit 29 embeds the back-remark column image in an area other than the area designated to embed the input information related to the input item (address item) (see FIG. 13). Thereafter, the processing shown in this flowchart ends. Note that steps S703 and S704 may be in any order.

[0140] In step S705, information is embedded so as to fit within the designated area. The form generation unit 29 embeds the image data (image) such that the image represented by the image data determined as the input information fits within the area designated to embed the input information related to the input item (address item). In the case of image data related to an image larger than the designated area, the size is reduced and then embedded in the designated area. Thereafter, the processing shown in this flowchart ends.

[0141] Note that by omitting the processing of step S701, the image targeted by this flow does not have to be limited to the back-remark column image. Also, although the image embedding process has been described in this flowchart, when the data determined as the input information is text data, the text data is embedded in the area designated to embed the input information related to the input item.

[0142] As described above, according to the present embodiment, by selecting one piece of read data as a candidate for input information based on a selection criterion and displaying it on a first screen, and when receiving an input for changing the candidate displayed on the first screen, displaying a second screen that enables changing of the candidate, it becomes possible to input appropriate information into the input items of the form. For example, when printing personal information read from an identity document onto an application form, it becomes possible to easily change the incorrect printed content before the characters misread during OCR processing of the identity document are printed on the application form, thereby making it possible to input appropriate information into the input items of the application form. Further, in the present embodiment, when the read data selected by the candidate selection unit 25 is image data, at least text data can be displayed on the second screen in a selectable manner, or when the read data selected by the candidate selection unit 25 is text data, at least image data can be displayed on the second screen in a selectable manner. That is, information that is a candidate for input information related to the input item is first displayed on the first screen in a format (such as text or image) according to the selection criterion, and it is possible to display a second screen that can change this format to another format. From this, it becomes possible to easily select data in a format different from the format (text or image) selected according to the selection criterion as a candidate for input information. That is, it becomes possible to easily change the candidate for input information to data in a format different from the format selected according to the selection criterion. From this, it becomes possible to input appropriate information into the input items of the form.

[0143] For example, on the first screen, if a user confirms that the OCR result, which is a candidate for input information, contains misread characters or the like, the user can easily change the candidate for the input information to a cut-out image on the second screen. Also, when the area to be read contains non-standard characters and the user wants to print the non-standard characters as input information, a cut-out image related to the area containing the non-standard characters is selected by the candidate selection unit 25 or by the user on the second screen, enabling the printing of the non-standard characters. Further, even when an OCR error occurs (when character recognition cannot be executed), a cut-out image related to the OCR target area is selected by the candidate selection unit 25 or by the user on the second screen, enabling the creation of an application form using the character string described in the OCR target area. As described above, according to the present embodiment, it is possible to input (print) appropriate personal information on the form. Also, according to the present embodiment, since candidates for input information are selected using selection criteria, it is possible to select more appropriate (accurate) personal information as candidates for input information and input (print) it on the form.

[0144] <Other Embodiments> In the above embodiment, an information processing system in which the information processing device 1 and the reading device 9 are separate devices (housings) is illustrated. However, by providing the information processing device 1 with a reading unit 91 or the like, the functions of the information processing device 1 and the functions of the reading device 9 may be provided by a single device having the same housing. In this case, the data acquisition unit 24 may function as the reading unit 81 and the identity verification unit 82. Also, since the information processing device 1 and the reading device 9 have the same housing, the information processing device 1 (reading device 9) does not need to include the data transmission unit 83.

Explanation of Reference Numerals

[0145] 1 Information Processing Device 9 Reading Device

Claims

1. An information processing system for generating a form by embedding input information in one or more input items of a form format, Candidate selection means for selecting, from a plurality of read data corresponding to one input item included in the one or more input items, which are read from a medium to be read, and each of which is image data or text data, one read data as a candidate for the input information related to the one input item based on a predetermined selection criterion; First display control means for displaying, on a first screen, the read data selected by the candidate selection means as a candidate for the input information related to the one input item; Second display control means for displaying a second screen that allows the candidate for the input information to be changed to other data when an input for changing the candidate for the input information related to the one input item displayed on the first screen is received; An information processing system comprising the above.

2. Further comprising priority storage means for storing priority data in which priorities for various data are defined, The selection criterion is a criterion for selecting, according to the priority data, the read data to be displayed as a candidate for the input information related to the one input item on the first screen. The information processing system according to claim 1.

3. The priority storage means stores the priority data in association with the type of medium and / or input item, The selection criterion is a criterion for selecting, according to the type of the medium to be read and / or the priority data corresponding to the one input item, the read data to be displayed as a candidate for the input information related to the one input item on the first screen. The information processing system according to claim 2.

4. The second display control means displays, on the second screen, a plurality of data that can be the input information related to the one input item in a selectable manner and in a display order according to the priorities for the plurality of data defined in the priority data. The information processing system according to claim 2.

5. The plurality of read data corresponding to the one input item read from the medium to be read includes one or more recognition result data obtained by character recognition, which is text data, The selection criterion is a criterion regarding the likelihood of the character recognition. The information processing system according to claim 1.

6. The criteria regarding the likelihood of the character recognition include a criterion that when the reliability score indicating the likelihood of one piece of recognition result data included in the one or more recognition result data is less than or less than a predetermined threshold, compared with the case where the reliability score of the one piece of recognition result data exceeds or is greater than or equal to the predetermined threshold, the one piece of recognition result data is less likely to be selected as a candidate for the input information related to the one input item. The information processing system according to claim 5.

7. The criteria regarding the likelihood of the character recognition include a criterion that when the one input item is an item related to an address, if the address indicated by one piece of recognition result data included in the one or more recognition result data is not included in an address database that pre - holds a plurality of addresses, compared with the case where the address indicated by the one piece of recognition result data is included in the address database, the one piece of recognition result data indicating the address is less likely to be selected as a candidate for the input information related to the one input item. The information processing system according to claim 5.

8. The plurality of read data corresponding to the one input item read from the medium to be read includes a plurality of recognition result data. The plurality of recognition result data are a plurality of recognized character strings obtained by performing character recognition on a predetermined area on the surface of the medium to be read by a plurality of character recognition engines. The criteria regarding the likelihood of the character recognition include a criterion that when the plurality of recognized character strings do not satisfy a predetermined condition, compared with the case where the plurality of recognized character strings satisfy the predetermined condition, a candidate for the input information related to the one input item is less likely to be selected from the plurality of recognized character strings. The information processing system according to claim 5.

9. The predetermined condition is that the relationship between the plurality of recognized character strings has a predetermined relationship. The information processing system according to claim 8.

10. It further includes priority storage means for storing priority data in which priorities for various data are determined. The plurality of read data corresponding to the one input item read from the medium to be read is text data and includes one or more recognition result data obtained by character recognition. The selection criteria include a criterion related to the certainty of the character recognition and a criterion for selecting the read data to be displayed as a candidate for the input information related to the one input item on the first screen according to the priority data, and are a combined criterion. The information processing system according to claim 1.

11. When an input for determining the candidate for the input information related to each input item as the input information to be embedded in the form format is received, the form generation means further includes generating the form by embedding the candidate for the input information as the input information in the form format. When the data to be embedded as the input information is image data and the image size of the image data does not satisfy a predetermined size condition in relation to the size of the embedding area for the input information related to the one input item, the form generation means embeds the image data in another area instead of the embedding area. The information processing system according to claim 1.

12. When the form generation means embeds the image data in the other area, the form generation means embeds a predetermined text in the embedding area. The information processing system according to claim 11.

13. When the candidate for the input information is changed on the second screen, the first display control means changes the data displayed as the candidate for the input information related to the one input item on the first screen in accordance with the changed content on the second screen. The information processing system according to claim 1.

14. The plurality of read data corresponding to the one input item read from the medium to be read includes at least two of the image data of a partial image cut out from the read image of the surface of the medium to be read, one or more recognition result data obtained by performing character recognition on the partial image, the image data read from an IC (Integrated Circuit) chip mounted on the medium to be read, and the text data read from the IC chip. The information processing system according to claim 1.

15. On the second screen, one or more data that can be the input information related to the one input item are displayed in a selectable manner. When one data is selected from the one or more data on the second screen, the candidate for the input information is changed to the selected one data. The information processing system according to any one of claims 1 to 14.

16. The one or more data that can be the input information related to the one input item include read data that has not been selected as a candidate for the input information by the candidate selection means among the plurality of read data corresponding to the one input item read from the medium to be read. The information processing system according to claim 15.

17. The plurality of read data corresponding to the one input item read from the medium to be read includes at least one text data and one image data. When the read data selected by the candidate selection means is the one text data, the second display control means causes at least the one image data to be displayed on the second screen in a selectable manner. When the read data selected by the candidate selection means is the one image data, the second display control means causes at least the one text data to be displayed on the second screen in a selectable manner. The information processing system according to claim 16.

18. The plurality of read data corresponding to the one input item read from the medium to be read includes at least one recognition result data obtained by character recognition that is text data and one image data. When the read data selected by the candidate selection means is the one recognition result data, the second display control means causes at least the one image data to be displayed on the second screen in a selectable manner. When the read data selected by the candidate selection means is the one image data, the second display control means causes at least the one recognition result data to be displayed on the second screen in a selectable manner. The information processing system according to claim 16.

19. A computer that generates a form by embedding input information in one or more input items of a form format. A candidate selection step of selecting one read data as a candidate for the input information related to the one input item from among the plurality of read data corresponding to the one input item included in the one or more input items, each of which is image data or text data, read from the medium to be read, based on a predetermined selection criterion. A first display control step of causing the first screen to display the read data selected in the candidate selection step as a candidate for the input information related to the one input item; A second display control step of causing a second screen to be displayed that enables the candidate for the input information to be changed to other data when an input for changing the candidate for the input information related to the one input item displayed on the first screen is received; An information processing method for executing.

20. A computer that generates a form by embedding input information in one or more input items of a form format, Candidate selection means for selecting one piece of read data as a candidate for the input information related to the one input item from among a plurality of pieces of read data corresponding to one input item included in the one or more input items, each of which is read from a medium to be read and is image data or text data, based on a predetermined selection criterion; First display control means for causing the first screen to display the read data selected by the candidate selection means as a candidate for the input information related to the one input item; Second display control means for causing a second screen to be displayed that enables the candidate for the input information to be changed to other data when an input for changing the candidate for the input information related to the one input item displayed on the first screen is received; An information processing program that functions as.

Images