Information processing device, control method for information processing device, and control program for information processing device

Abstract

To provide an information processing device or the like capable of recognizing characters written on a document with high accuracy, regardless of whether the characters are regular or irregular, and whether they are handwritten or printed.SOLUTION: An information processing device includes an acquisition unit that acquires an image of a document including printed characters and handwritten characters, a printed character recognition unit that recognizes the printed characters in the image, an identification unit that identifies the position of a keyword in the image based on a recognition result by the printed character recognition unit, a setting unit that sets a target area that is an area including a handwritten character string to be detected based on the position of the keyword and features of the image, an extraction unit that extracts multiple character strings included in the target area, a handwritten character recognition unit that recognizes the handwritten characters of the multiple character strings and calculates a reliability degree of the recognition result, a determination unit that determines the handwritten character string to be detected from the multiple character strings based on the reliability degree, and an output unit that outputs the recognition result by the handwritten character recognition unit for the determined handwritten character string to be detected.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a control program for an information processing device. [Background technology]

[0002] Conventionally, OCR (Optical Character Recoginition / Reader) has been used to read characters written in each field on identification documents such as driver's licenses and health insurance cards from images of the identification documents (for example, Patent Document 1). In recent years, with the development of artificial intelligence technology, the use of AI-OCR, which utilizes AI (Artificial Intelligence) technology such as deep learning to improve the accuracy of character recognition and layout analysis, has become widespread. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-140703 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, some identity verification documents, such as maternal and child health handbooks, are non-standard documents whose formats vary by local government, and contain a mixture of handwritten and typed items. In such cases, even with AI-OCR, it has been difficult to read them accurately.

[0005] There is a demand for highly accurate recognition of characters written on documents, whether they are regular or irregular, and whether they are handwritten or printed. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes an acquisition unit that acquires an image of a document including typed characters and handwritten characters, a typed character recognition unit that recognizes typed characters in the image, an identification unit that identifies the position of a keyword in the image based on the recognition result by the typed character recognition unit, a setting unit that sets a target area that is an area including a handwritten character string to be detected based on the position of the keyword and image features, an extraction unit that extracts multiple character strings included in the target area, a handwritten character recognition unit that recognizes handwritten characters in the multiple character strings and calculates the reliability of the recognition result, a determination unit that determines a handwritten character string to be detected from the multiple character strings based on the reliability, and an output unit that outputs the recognition result by the handwritten character recognition unit for the handwritten character string to be detected determined by the determination unit.

[0007] In an information processing device according to one embodiment of the present invention, an image includes a plurality of handwritten character strings to be detected, and the setting unit may use the position of one target area including one handwritten character string to be detected, which is identified by the position of a keyword, as the position of a keyword when setting other target areas including other handwritten character strings to be detected.

[0008] In the information processing device according to one embodiment of the present invention, the setting section may set the target area based on a ruled line included in the image as a feature of the image.

[0009] In the information processing device according to one embodiment of the present invention, the acquisition unit may further acquire information relating to the type of document, and the identification unit may identify the position of the keyword according to the type of document.

[0010] In an information processing device according to an embodiment of the present invention, the extraction section may combine bounding boxes surrounding characters or character strings included in the target area based on a predetermined condition, and extract the characters as multiple character strings.

[0011] In an information processing device according to one embodiment of the present invention, the setting unit sets an area including a typed character string to be detected as a target area, the typed character recognition unit recognizes typed characters of multiple character strings included in the target area extracted by the extraction unit and calculates the reliability of the typed character recognition results, the determination unit determines a typed character string to be detected from among the multiple character strings based on the reliability of the typed character recognition results, and the output unit may output the recognition result by the typed character recognition unit for the typed character string to be detected determined by the determination unit.

[0012] In the information processing device according to one embodiment of the present invention, the document may be a maternal and child health handbook.

[0013] A control method for an information processing device according to one embodiment of the present invention causes the information processing device to execute an acquisition step of acquiring an image of a document including typed characters and handwritten characters, a typed character recognition step of recognizing typed characters in the image, an identification step of identifying the position of a keyword in the image based on the recognition result from the typed character recognition step, a setting step of setting a target area that is an area including a handwritten character string to be detected based on the position of the keyword and image features, an extraction step of extracting multiple character strings included in the target area, a handwritten character recognition step of recognizing handwritten characters in the multiple character strings and calculating the reliability of the recognition result, a determination step of determining a handwritten character string to be detected from among the multiple character strings based on the reliability, and an output step of outputting the recognition result from the handwritten character recognition step for the handwritten character string to be detected determined by the determination step.

[0014] A control program for an information processing device according to one embodiment of the present invention causes the information processing device to implement an acquisition function for acquiring an image of a document including typed characters and handwritten characters, a typed character recognition function for recognizing typed characters in the image, an identification function for identifying the position of a keyword in the image based on the recognition result by the typed character recognition function, a setting function for setting a target area which is an area including a handwritten character string to be detected based on the position of the keyword and the image features, an extraction function for extracting multiple character strings included in the target area, a handwritten character recognition function for recognizing handwritten characters in multiple character strings and calculating the reliability of the recognition result, a determination function for determining a handwritten character string to be detected from among the multiple character strings based on the reliability, and an output function for outputting the recognition result by the handwritten character recognition function for the handwritten character string to be detected determined by the determination function. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram of the configuration of an information processing system according to one embodiment of the present invention. [Figure 2] FIG. 2 is an example of a functional block diagram of a server (information processing device) according to one embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart showing an example of the operation of the server according to one embodiment of the present invention. [Figure 4] 4(a) to 4(d) are diagrams for explaining an outline of character detection according to one embodiment of the present invention. [Figure 5] FIG. 5 is an example of a data table. [Figure 6] FIG. 6 is an example machine learning pipeline for character recognition according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the invention according to the present disclosure (also referred to as the present invention) will be described using the drawings. Note that the drawings are merely examples, and the present invention is not limited to those shown in the drawings. For example, the number of servers, user terminals, etc., data sets (tables), flowcharts, and documents shown in the drawings are merely examples, and the present invention is not limited to these.

[0017] <System configuration> 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. The information processing system 600 is a system that extracts information for determining whether a user satisfies the conditions for using a predetermined service from an image 10 of a user's identification document. That is, the information processing system 600 may perform character recognition processing on the image 10 to output necessary character information (text information) from the image 10. The identification document may vary depending on the predetermined service to be provided, and may be, for example, an official identification document such as a driver's license, health insurance card, passport, My Number card, resident registration card, maternal and child health handbook, national pension handbook, student ID card, student notebook, etc., as well as an employee ID card, business card, mail, bankbook, various forms, etc.

[0018] The information processing system 600 includes a server (character recognition device) 100, a service server 101, and one or more user terminals 200. The server 100 is an information processing device (character recognition device) that performs character recognition processing. The service server 101 is an information processing device that performs processing related to a predetermined service provided to a user. The service server 101 is connected to the server 100, and passes an image 10 transmitted from the user terminal 200 to the server 100. The server 100 performs character recognition processing on the image of the document 10 transmitted from the user terminal 200, and outputs the result of extracting characters contained in the document 10 (character recognition result).

[0019] Note that, although FIG. 1 illustrates the server 100 and the service server 101 as separate entities, this is not limiting. That is, each function described as being provided by the server 100 may be realized by multiple servers or by a single server. Furthermore, the server 100 may be, for example, a distributed server system that cooperates by communicating via a network, or may be a so-called cloud server. That is, the server 100 is not limited to a physical server, and may also include a virtual server created by software. Furthermore, the server 100 may be any device that is an information processing device that can realize the functions described in each embodiment, and may include, for example, a server device, a computer (for example, but not limited to, a desktop, laptop, tablet, etc.), a communication platform, etc.

[0020] The user terminal 200 and the service server 101 are connected via a network 500. The network 500 includes a wireless network and a wired network, and specifically, for example, the network 500 may be a wireless LAN (WLAN), a wide area network (WAN), code division multiple access (CDMA), long term evolution (LTE), LTE-Advanced, fourth generation communication (4G), fifth generation communication (5G), sixth generation communication (6G) or later mobile communication system, etc. Note that the network 500 is not limited to these examples and may generally be, for example, Bluetooth (registered trademark), an optical fiber line, etc. Alternatively, the network 500 may be a combination of these.

[0021] Although FIG. 1 shows a smartphone as the user terminal 200, the user terminal 200 may be any type of communication terminal capable of transmitting the image 10 to the service server 101.

[0022] Next, the hardware configuration and functional configuration of the server 100 will be described with reference to FIG. (1) Server hardware configuration The server 100 includes a control unit 110 , a communication unit 120 , an input / output unit 130 , and a storage unit 170 .

[0023] The control unit 110 is typically a processor, and includes a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), a microprocessor, etc., and is realized by a logic circuit (hardware) or a dedicated circuit formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc. The control unit 110 reads out a program (software) stored in the storage unit 170 and executes the code and instructions included in the program, thereby performing the functions and methods described in each embodiment.

[0024] The storage unit 170 stores various programs and various data required for the operation of the server 100. The storage unit 170 may include, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 170 may also include memory (random access memory (RAM), read only memory (ROM), etc.) that provides a working area for the control unit 110.

[0025] The communication unit 120 is implemented as hardware such as a network adapter, communication software, or a combination of these. The communication unit 120 transmits and receives various data to and from the service server 101 via the network 500.

[0026] The input / output unit 130 includes an input device that accepts various operations from an administrator or the like on the server 100, and an output unit that outputs processing results processed by the server 100. The input device may include, for example, a keyboard and a microphone, and the output device may include, for example, a display and a speaker.

[0027] (2) Server functional configuration The server 100 may include, as functions realized by the control unit 110, an acquisition unit 111, a character detection unit 112, a printed character recognition unit 113, a handwritten character recognition unit 114, an identification unit 115, a setting unit 116, an extraction unit 117, and a determination unit 118. Of the functional units shown in FIG. 2, functional units that are not essential to the embodiments described below may be omitted. Furthermore, the functions or processes of each functional unit may be realized by machine learning or AI to the extent feasible.

[0028] <Server control flow> The control flow of the server 100 according to one embodiment of the present invention will be described with reference to FIGS. 3 to 5, along with the description of each functional unit of the server 100. FIG.

[0029] The acquisition unit 111 acquires an image 10 of a document including printed characters and handwritten characters (step S11). Hereinafter, the service server 101 will be described as providing a service requiring proof of parent-child relationship, and the image 10 will be described as relating to a birth registration certificate in a maternal and child health handbook. Specifically, the character recognition process according to one embodiment of the present invention may aim to detect the parent's name, the child's name, and the child's date of birth from an image of the birth registration certificate, in order to use the information acquired when signing up for the service to determine whether it is correct. However, the document targeted by the present invention is not limited to the maternal and child health handbook.

[0030] FIG. 4(a) shows an example of image 10, captured from a page of a maternal and child health handbook containing a completed birth registration certificate (hereinafter simply referred to as a "birth registration certificate"). As mentioned above, the format of birth registration certificates is not standardized among municipalities, and the location of the parent's name and child's name varies. Furthermore, some municipalities affix a sticker bearing the birth registration certificate in place of the original birth registration certificate. In this case, the thinness of the sticker paper can cause the text underneath to show through. For these reasons, it is difficult to recognize the parent's name, child's name, and date of birth. Furthermore, factors that complicate character recognition include the mixture of printed and handwritten characters, phonetic readings written above the name, and the presence of circled characters on the same line as the text.

[0031] The printed character recognition unit 113 recognizes printed characters in the image 10 (step S12). Note that, prior to the recognition of printed characters by the printed character recognition unit 113, the character detection unit 112 detects characters in the image 10. Note that the character detection process may use a known character detection method, such as a method based on deep learning (R-CNN: Recrrent CNN (Convolutional Neural Networks), YOLO, SSD) or a method based on a sliding window method. Also, the character recognition process may use a known character recognition method, such as a character recognition model set by prior learning using deep learning, or a pattern matching method.

[0032] The identification unit 115 identifies the position of a keyword in the image 10 based on the recognition result by the printed character recognition unit 113 (step S13). A keyword is a character or object used to estimate the location of a target detection object (here, the parent's name, the child's name, and the date of birth), and serves as an anchor. As shown in FIG. 4(a), on the birth registration certificate, adjacent to the field for entering the parent's name are characters such as "mother," "father," and "name." Additionally, adjacent to the field for entering the child's name are characters such as "child's name" and "birth registration certificate." These keywords may be stored in the storage unit 170 for each type of document.

[0033] FIG. 5 is an example of a data table stored in memory unit 170 that stores keywords when the document is a birth registration certificate. Note that the figure is just an example, and there may be more keywords. The target area will be described later. Identification unit 115 may identify, as keywords 31, characters stored in data table TB10 from among the characters recognized by print character recognition unit 113.

[0034] The setting unit 116 sets a target area, which is an area including the handwritten character string to be detected, based on the position of the keyword 31 and the characteristics of the image 10 (step S14). Here, the data table TB10 of FIG. 5 also stores information regarding the relative positional relationship between the position of the keyword and the target area. For example, the data table TB10 indicates that the target area including the "parent's name," which is the detection target, is located to the right of the keyword "mother" (here, this means the right side when viewing the image 10 from above). Note that the "right side" is used for the purpose of explanation, and in reality, the relative position of the target area with respect to the keyword is stored in the data table TB10 in a format that can be understood by an information processing device.

[0035] 4(b) shows an example of the target area 33 set according to the position of the keyword. The setting unit 116 may use the ruled lines that separate the items and entry fields on the birth registration certificate, which are detected by image processing of the image 10, to set the target area. That is, the target area 33 is detected to the right of the keyword "mother" and is set as a rectangular area (cell) separated by the ruled lines.

[0036] The extraction unit 117 extracts multiple character strings included in the target area (step S15). This will be described with reference to FIG. 4(c). As a first step, the extraction unit 117 may extract character strings included in the target area 33 using bounding boxes. FIG. 4(c) shows an example in which bounding boxes b1 to b4 are extracted. Note that since bounding boxes are known, their description will be omitted. As a second step, the extraction unit 117 extracts multiple character strings by combining at least one or more bounding boxes. Note that the extraction unit 117 may combine bounding boxes surrounding characters included in the target area based on predetermined conditions to extract multiple character strings.

[0037] Here, predetermined conditions for combining bounding boxes are described. First, candidate bounding boxes may be provisionally determined based on their height and position information and then grouped by line. This allows for distinguishing between lines of printed characters and lines of handwritten characters, and for distinguishing between furigana and kanji. Lines may be combined or split based on the line height and relative position from the anchor, and the final line may be selected. The predetermined conditions may be, for example, conditions for predicting the date of birth, separating the furigana from the name, removing circled characters, and combining the first and last names. Specifically, when predicting the date of birth, the year, month, and day are printed characters, and the numbers are handwritten characters. Considering that handwritten characters tend to be larger than printed characters, the combination of characters to be recognized and their heights may be appropriately selected before combining. When separating the furigana from the name, the bounding boxes may be split by deleting the center of the overlapping portion. Furthermore, when removing circled characters, for example, in a maternal and child health handbook, since there are circles for male and female, they may be removed if "male and female" are within the same bounding box, or if "male" cannot be read and only "female" is present. Furthermore, when combining names, whether to combine the bounding boxes may be determined based on the height and position of the bounding boxes.

[0038] For example, in FIG. 4(c), the bounding boxes located on the same row are combined, and the character strings "Nangin" based on only the bounding box b3, "Nangin Hanako" based on the bounding boxes b3 and b4, and "Hanako" based on only the bounding box b4 may be extracted. The character strings "Nangin" based on only the bounding box b1, "Nangin Hanako" based on the bounding boxes b1 and b2, and "Hanako" based on only the bounding box b2 may be extracted. The processing by the extraction unit 117 is similar to the processing for extracting a region of interest (ROI) in object detection.

[0039] The handwritten character recognition unit 114 recognizes the handwritten characters of the multiple character strings extracted by the extraction unit 117 and calculates the reliability of the recognition result (step S16). The handwritten character recognition process may use an existing method, such as a model trained by machine learning on a data set of handwritten characters or a pattern matching method using a dictionary of handwritten characters. The reliability of the recognition result is a numerical value indicating the likelihood of the recognized handwritten characters, and may be, for example, the reliability score used in YOLO. According to one embodiment of the present invention, a score indicating the likelihood (reliability) of the prediction result is used, and if the score is lower than a threshold, the read result may be returned as "no character string." For example, the above logic may be used in areas where blanks are allowed, such as when only one of two parent name entry fields is filled in.

[0040] The determining unit 118 determines a handwritten character string to be detected from among the multiple character strings based on the reliability (step S17). That is, the character string with the highest reliability may be determined as the detection target. The output unit 119 outputs the recognition result by the handwritten character recognition unit 114 for the handwritten character string to be detected determined by the determining unit 118 (step S18).

[0041] In this way, according to one embodiment of the present invention, a target area including a character string to be detected is set from keywords identified by a recognition process for printed characters, which has a lower processing load than that of handwritten character recognition. Therefore, it is possible to estimate the position of the character string to be detected with high accuracy even for documents with different formats.

[0042] The setting unit 116 may use the position of one target area including one handwritten character string of a detection target, which is identified by the position of a keyword, as the position of the keyword when setting another target area including another handwritten character string of a detection target. This will be explained using FIG. 4(b). For example, suppose that the keyword "mother" was not recognized by the printed character recognition unit 113 due to light reflection or the like when the image 10 was captured. In this case, the setting unit 116 may set the target area 33 including the mother's name using the target area 34 set using the keyword "father." In this case, prior information that the name of "mother" will be written above the name of "father" may be used.

[0043] This makes it possible to estimate a target area that includes the character string to be detected even if the accuracy of estimating the position of a keyword using printed characters is low.

[0044] In one embodiment of the present invention, the acquisition unit 111 may further acquire information regarding the type of document, and the identification unit 115 may identify the position of the keyword according to the type of document. That is, information on the anchor keyword may be stored for each document in the storage unit 170, and may be used by the identification unit 115 to identify the position of the keyword. This makes it possible to detect the target character string for various types of documents.

[0045] The above-described processing according to an embodiment of the present invention may be realized by an end-to-end model using deep learning. The end-to-end model is a method of learning input and output in a single pipeline. Figure 6 shows an example of pipeline processing for character recognition according to an embodiment of the present invention.

[0046] When the pipeline 20 receives an input of an image 10, it performs a process of detecting printed characters (S1) and a process of detecting handwritten characters (S4). The detected printed characters are then subjected to a recognition process (S3) and then used to extract a target area (S3). The extracted target area is then subjected to a process of removing noise other than the character string (S4) and a process of refining the target area (S5). The results of the handwritten character detection (S4) may also be used in the refinement of the target area. The refinement process refers to an improvement and division process of the target area, and may include a process of bringing the target area closer to the detection target or a process of dividing overlapping target areas. The handwritten characters included in the target area are then subjected to a recognition process (S6), followed by a filtering process (S7), and the recognition results are output. The filtering process may include the removal of noise such as furigana and circled characters, and correction of date formats.

[0047] While the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art would readily be able to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present invention. For example, the functions included in each component, step, etc. may be rearranged so as not to cause logical inconsistencies, and multiple components, steps, etc. may be combined or separated into one. Furthermore, the configurations described in the above embodiments may be appropriately combined. For example, each component described as being included in server 100 may be realized in a distributed manner across multiple servers.

[0048] The programs of the embodiments of the present disclosure may be provided in a state stored in a storage medium readable by an information processing device. The storage medium may store the programs in a "non-transitory tangible medium." The programs include, for example, software programs and control programs. When the functional units of the server 100 are realized by software, the server 100 functions as an acquisition unit 111, a character detection unit 112, a printed character recognition unit 113, a handwritten character recognition unit 114, an identification unit 115, a setting unit 116, an extraction unit 117, a determination unit 118, and an output unit 119 by the processor executing the programs loaded into the memory.

[0049] The storage medium may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs), application specific ICs (ASICs), etc.), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0050] Furthermore, the program of the present disclosure may be provided to the server 100 via any transmission medium (such as a communication network or broadcast waves) capable of transmitting the program.

[0051] Furthermore, each embodiment of the present disclosure may be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission. Note that the program of the present disclosure may be implemented using, for example, a scripting language such as JavaScript (registered trademark) or Python, or C, Go, Swift, Koltin, Java (registered trademark), or the like.

[0052] According to each aspect of the present disclosure described above, by providing a highly convenient method of personal authentication, it is possible to contribute to achieving Goal 11 of the Sustainable Development Goals (SDGs), "Sustainable cities and communities." [Explanation of symbols]

[0053] 100 Server (information processing device) 111 Acquisition Department 112 Character detection unit 113 Print character recognition section 114 Handwritten Character Recognition Unit 115 Specific section 116 Setting Section 117 Extraction part 118 Decision Section 119 Output section 120 Communications Department 130 Input / output section 170 Storage device 101 Service Server 200 User terminal (terminal device) 500 Network 600 Information Processing Systems

Claims

1. an acquisition unit for acquiring an image of a document containing printed and handwritten characters; a printed character recognition unit that recognizes printed characters in the image; an identification unit that identifies a position of a keyword within the image based on a recognition result by the printed character recognition unit; a setting unit that sets a target area that is an area including a handwritten character string to be detected based on the position of the keyword and the features of the image; an extraction unit that extracts a plurality of character strings included in the target area; a handwritten character recognition unit that recognizes handwritten characters of the plurality of character strings and calculates the reliability of the recognition results; a determination unit that determines the handwritten character string to be detected from among the plurality of character strings based on the reliability; an output unit that outputs a recognition result by the handwritten character recognition unit for the handwritten character string to be detected that has been determined by the determination unit; An information processing device comprising:

2. the image includes a plurality of handwritten character strings to be detected, the setting unit uses the position of one target area including one handwritten character string to be detected, which is specified by the position of the keyword, as the position of a keyword when setting another target area including another handwritten character string to be detected. The information processing device according to claim 1 .

3. the setting unit sets the target area based on a ruled line included in the image as a feature of the image. The information processing device according to claim 1 .

4. The acquisition unit further acquires information regarding the type of the document, the identifying unit identifies a position of a keyword according to the type of the document; The information processing device according to claim 1 .

5. the extraction unit combines bounding boxes surrounding characters or character strings included in the target area based on predetermined conditions to extract the multiple character strings; The information processing device according to claim 1 .

6. the setting unit sets an area including a character string to be detected as the target area; the printed character recognition unit recognizes printed characters of a plurality of character strings included in the target area extracted by the extraction unit, and calculates a reliability of the recognition results of the printed characters; the determination unit determines a print character string to be detected from among a plurality of character strings based on a reliability of a recognition result of the print character; the output unit outputs a recognition result by the print character recognition unit for the print character string to be detected determined by the determination unit. The information processing device according to claim 1 .

7. The information processing device according to claim 1 , wherein the document is a maternal and child health handbook.

8. In the information processing device, an acquisition step of acquiring an image of a document containing printed and handwritten characters; a print character recognition step of recognizing print characters within the image; an identifying step of identifying a position of a keyword within the image based on a recognition result from the printed character recognition step; a setting step of setting a target area that is an area including a handwritten character string to be detected based on the position of the keyword and the features of the image; an extraction step of extracting a plurality of character strings included in the target region; a handwritten character recognition step of recognizing handwritten characters of the plurality of character strings and calculating a reliability of the recognition result; a determining step of determining the handwritten character string to be detected from among the plurality of character strings based on the reliability; an output step of outputting a recognition result of the handwritten character string determined in the determination step for the detection target handwritten character string; A control method for an information processing device that causes the information processing device to execute the above.

9. In the information processing device, an acquisition function for acquiring images of documents containing printed and handwritten characters; a print character recognition function for recognizing print characters in the image; an identifying function for identifying a position of a keyword within the image based on a recognition result by the printed character recognition function; a setting function for setting a target area, which is an area including a handwritten character string to be detected, based on the position of the keyword and the features of the image; an extraction function for extracting a plurality of character strings included in the target area; a handwritten character recognition function for recognizing handwritten characters of the plurality of character strings and calculating the reliability of the recognition result; a determination function for determining the handwritten character string to be detected from among the plurality of character strings based on the reliability; an output function for outputting a recognition result by the handwritten character recognition function for the handwritten character string to be detected determined by the determination function; A control program for an information processing device that realizes the above.

Images