Image processing device, image processing method, and program

Abstract

To reduce a processing time required for character recognition processing on a character region in a document image and appropriately determine the character region to be targeted for character recognition.SOLUTION: An image processing device detects character regions from a document image, and among a plurality of character regions as detected, specifies a target character region which is a target on which character recognition processing is to be performed based on information about the heights and widths of the character regions as well as a positional relationship between a relevant character region and other of the character regions present around the relevant character region, and performs the character recognition processing on the target character region as specified.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technique for extracting text information from image data. [Background technology]

[0002] There is a system that extracts a character string indicating a document title or the like from a document image and automatically assigns the extracted character string to a file name of a file including data of the document image, or to a folder name of a folder storing the file (hereinafter, these are collectively referred to as a "file name"), etc. There is also a system that extracts a character string indicating a name of a person, organization, or object, etc., or an amount, etc. from a document image and automatically transcribes the extracted character string to a database, etc. In these systems, since the position in the document image of a character area including a character string that a user wants to extract from the document image differs for each document image, it may be necessary to perform character recognition processing on all character areas in the document image. However, performing character recognition processing on all character areas in the document image requires an enormous amount of processing time.

[0003] Patent Document 1 discloses a technique for analyzing character attributes such as character size or character color for each character area in a document image and determining the character area for character recognition based on the analyzed attributes. According to the technique disclosed in Patent Document 1, character recognition processing is performed only on necessary character areas in a document image, so that the processing time of the character recognition processing is reduced. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-178007 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a character region to be subjected to character recognition is determined simply based on the character attributes such as the character size or color of the character region, as in the technology disclosed in Patent Document 1, a character region that should be subjected to character recognition processing may be excluded from character recognition. For example, this may occur when a character string that should be a target is included in a character region that is excluded from character recognition because its character size is determined to be smaller than a predetermined threshold value.

[0006] The present disclosure is intended to solve the above-mentioned problems, and aims to appropriately determine character regions to be subject to character recognition while reducing the processing time required for character recognition processing on character regions in a document image. [Means for solving the problem]

[0007] The image processing device of the present disclosure has a detection means for detecting a character area from a document image, an identification means for identifying a target character area, which is the character area on which character recognition processing is to be performed, from among the multiple character areas detected by the detection means, based on information on the height and width of the character area and the positional relationship between the character area and other character areas surrounding the character area, and a character recognition means for performing the character recognition processing on the target character area identified by the identification means. Effect of the Invention

[0008] According to the present disclosure, it is possible to appropriately determine character regions to be subjected to character recognition while reducing the processing time required for character recognition processing on character regions in a document image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing an example of a hardware configuration of the image processing system. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the image processing system. [Diagram 3] FIG. 2 is a sequence diagram showing an example of a processing flow in the image processing system. [Figure 4] 1A and 1B are diagrams illustrating an example of a document image, and character strings to be extracted and character strings not to be extracted. [Diagram 5] 10 is a flowchart showing an example of the flow of a character extraction process in a character extraction unit. [Figure 6] 10 is a flowchart showing an example of the flow of a target determination process for character recognition in a target determination unit. [Figure 7] 11 is a diagram for explaining an example of a display process in the information processing terminal. FIG. [Figure 8] 10 is a flowchart showing an example of the flow of a target determination process for character recognition in a target determination unit. [Figure 9] 13 is a flowchart showing an example of the flow of a target setting process in a target determination unit. [Figure 10] FIG. 1 is a diagram showing an example of a character region including a character string composed of handwritten characters. [Figure 11] 10 is a flowchart showing an example of the flow of a recognition execution area setting process in an area setting unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the disclosed solution.

[0011] [First embodiment] <Image processing system configuration> 1 is a block diagram showing an example of a hardware configuration of an image processing system 100 in the first embodiment. The image processing system 100 includes an image forming apparatus 110, an image processing server 120, an information processing terminal 130, and a storage server 140. The image forming apparatus 110, the image processing server 120, the information processing terminal 130, and the storage server 140 are connected to each other so as to be able to communicate with each other via a network 150 such as a LAN (Local Area Network) or the Internet.

[0012] The image forming apparatus 110 can transmit image data obtained by reading (hereinafter also referred to as "scanning") an original (document) to the image processing server 120 and can request transmission to the storage server 140 via the image processing server 120. In the present embodiment, the image forming apparatus 110 is described as an MFP having a scanning function, a printing function, a copying function, and the like, as an example, but the image forming apparatus 110 is not limited to a MFP. For example, the image forming apparatus 110 may be an apparatus having a scanning function. Here, the scanning function is a function of transmitting image data generated by optically reading an original using a scanner possessed by the image forming apparatus 110 to an external device. The printing function is a function of forming (printing) an image indicated by print data received from the information processing terminal 130 or the like on a recording medium such as paper. The copying function is a function of obtaining a copy of an original by printing an image obtained by reading an original using a scanner on a recording medium.

[0013] In the present embodiment, the image processing system 100 is described as having an image forming apparatus 110, an image processing server 120, an information processing terminal 130, and a storage server 140 as an example, but the configuration of the image processing system 100 is not limited to this. For example, the image forming apparatus 110 may have at least one of the functions of the information processing terminal 130 and the image processing server 120. The image processing server 120 may be connected on a LAN rather than on the Internet. The storage server 140 may be replaced with a mail server or the like. In this case, for example, the storage server 140 may attach image data obtained by reading a document to an e-mail or the like and send it to a specified e-mail address. For example, the image processing system 100 may have at least one of the image processing server 120, the information processing terminal 130, and the storage server 140 for one image forming apparatus 110.

[0014] <Hardware configuration> The image forming apparatus 110 has a printer 101, a scanner 102, and an operation unit 103 as hardware components. In addition to the above hardware components, the image forming apparatus 110 also has a CPU 111, a RAM 112, a HDD 113, a network I / F 114, a printer I / F 115, a scanner I / F 116, an operation unit I / F 117, and an expansion I / F 118. The CPU 111 is connected to the RAM 112, the HDD 113, the network I / F 114, the printer I / F 115, the scanner I / F 116, the operation unit I / F 117, and the expansion I / F 118 via a bus, and can exchange various data with these. The CPU 111 also loads commands (computer programs) read from the HDD 113 into the RAM 112, and executes the commands loaded into the RAM 112 to control the execution of processing of each functional component to be described later. In this embodiment, one CPU 111 executes each process shown in the flowchart described later using one memory (RAM 112 or HDD 113), but is not limited to this. For example, each process may be executed by multiple CPUs, or multiple RAMs or HDDs working together.

[0015] HDD 113 is configured with a hard disk drive or the like, and can store instructions executable by CPU 111, setting values ​​used in image forming apparatus 110, data related to processing requested by a user, and the like. RAM 112 is an area for temporarily storing instructions read by CPU 111 from HDD 113. RAM 112 can also store various data required for executing instructions. For example, in image processing, CPU 111 can perform processing by expanding input image data in RAM 112.

[0016] The network I / F 114 is an interface for performing network communication with an external device in the image processing system 100. The network I / F 114 can notify the CPU 111 of the completion of data reception, and can transmit data on the RAM 112 to the network 150 according to an instruction from the CPU 111. The printer I / F 115 can transmit print data to be printed to the printer 101 according to an instruction from the CPU 111, and can transmit information indicating the printer status received from the printer 101 to the CPU 111. The scanner I / F 116 transmits an image reading instruction instructed by the CPU 111 to the scanner 102, and transmits image data received from the scanner 102 to the CPU 111. The scanner I / F 116 can also transmit information on the status of the scanner 102 received from the scanner 102 to the CPU 111.

[0017] The operation unit I / F 117 can transmit instructions from the user input via the operation unit 103 to the CPU 111 and can display screen information for the user to operate on the operation unit 103. The expansion I / F 118 is an interface that allows an external device to be connected to the image forming apparatus 110. The expansion I / F 118 includes, for example, a USB (Universal Serial Bus) type interface. The image forming apparatus 110 can read data stored in an external storage device such as a USB memory connected to the expansion I / F 118 and write data to the external storage device.

[0018] The printer 101 can print an image indicated by image data received via the printer I / F 115 on paper, and can also transmit information indicating the status of the printer 101 to the printer I / F 115. The scanner 102 can read an original placed on the scanner 102 according to an image reading instruction received via the scanner I / F 116, and transmit image data obtained by the reading to the scanner I / F 116. The scanner 102 can also transmit information indicating the status of the scanner 102 to the scanner I / F 116. The operation unit 103 is an interface for issuing various instructions to the image forming apparatus 110 based on a user's operation. For example, the operation unit 103 is equipped with a touch panel type liquid crystal display, displays an operation screen, and accepts operations from the user.

[0019] The image processing server 120 has a CPU 121, a RAM 122, a HDD 123, and a network I / F 124 as hardware configurations. The CPU 121 controls the entire image processing server 120. The CPU 121 is connected to the RAM 122, the HDD 123, and the network I / F 124 via a bus, and can control the transmission and reception of various data between them. The CPU 121 loads a control program (command) read from the HDD 123 into the RAM 122 and executes it. The HDD 123 is a large-capacity storage device such as a hard disk drive that stores image data, various programs, and various data generated during image processing. In this embodiment, one CPU 121 executes each process shown in a flowchart described later using one memory (RAM 122 or HDD 123), but is not limited to this.

[0020] The information processing terminal 130 has, as a hardware configuration, a CPU 131, a RAM 132, a HDD 133, a network I / F 134, an operation unit I / F 135, and an operation unit 136. The CPU 131 controls the entire information processing terminal 130. The CPU 131 is also connected to the RAM 132, the HDD 133, the network I / F 134, the operation unit I / F 135, and the operation unit 136 via a bus, and can control the transmission and reception of various data between them. The CPU 131 also loads a control program (command) read from the HDD 133 into the RAM 132 and executes it. The operation unit I / F 135 is an interface that transmits instructions from a user input from the operation unit 136 to the CPU 131, and also transmits information on an operation screen to be displayed to the operation unit 136 based on display control by the CPU 131. In this embodiment, one CPU 131 executes each process shown in a flowchart described later using one memory (RAM 132 or HDD 133), but the present invention is not limited to this.

[0021] The storage server 140 has a CPU 141, a RAM 142, a HDD 143, and a network I / F 144 as hardware configurations. The CPU 141 controls the entire storage server 140. The CPU 141 is also connected to the RAM 142, the HDD 143, and the network I / F 144 via a bus, and can control data transmission and reception between them. The CPU 141 also loads a control program (command) read from the HDD 143 into the RAM 142 and executes it. The HDD 143 can store image data received from the image processing server 120. In this embodiment, one CPU 141 executes each process shown in a flowchart described later using one memory (RAM 142 or HDD 143), but is not limited to this.

[0022] <Functional configuration> Fig. 2 is a block diagram showing an example of a functional configuration of the image processing system according to the present embodiment. Specifically, Fig. 2(a) is a block diagram showing an example of a functional configuration of the image forming apparatus 110, Fig. 2(b) is a block diagram showing an example of a functional configuration of the image processing server 120, and Fig. 2(c) is a block diagram showing an example of a functional configuration of the information processing terminal 130. The image forming apparatus 110 has, as its functional configuration, an image reading unit 211, an image printing unit 212, a UI display unit 213, an input receiving unit 214, and an image transmitting unit 215. Each unit of the image forming apparatus 110 as its functional configuration is realized by the CPU 111 transferring a computer program stored in the HDD 113 to the RAM 112 and executing it.

[0023] The image reading unit 211 makes the scanner 102 read an original placed on the scanner 102, converts a signal obtained by the reading into image data, and stores the image data in the HDD 113. The image printing unit 212 transfers the image data to the printer 101 and causes the printer 101 to print an image indicated by the image data. Specifically, the image printing unit 212 first reads out image data stored in the HDD 113, or receives image data from another device on the network 150 via the network I / F 114, and develops the read or received image data on the RAM 112. Next, the image printing unit 212 transfers the developed image data to the printer 101 and causes the printer 101 to print the image data.

[0024] The UI display unit 213 displays operation components such as buttons operated by the user, and UI (User Interface) components that display information such as processing progress. The input reception unit 214 receives operation input by the user to the UI components displayed by the UI display unit 213, and instructs each unit to execute processing corresponding to the operation input. The image transmission unit 215 transmits image data stored in the HDD 113 and the setting values ​​of the scan settings to an external device such as the image processing server 120 on the network 150 via the network I / F 114.

[0025] The image processing server 120 has, as its functional components, an image receiving unit 221, a data generating unit 222, a data transmitting unit 223, a correcting unit 224, a file generating unit 225, a file transmitting unit 226, and a character extracting unit 230. Each unit that the image processing server 120 has as its functional components is realized by the CPU 121 transferring a control program stored in the HDD 123 to the RAM 122 and executing it.

[0026] The image receiving unit 221 receives image data transmitted from the image transmitting unit 215 of the image forming apparatus 110. The image data received by the image receiving unit 221 is stored in the HDD 123. The data generating unit 222 generates data used for the operation of the extraction display unit 250 of the information processing terminal 130 described later. In this embodiment, the extraction display unit 250 is described as being realized by executing a Web application program (hereinafter referred to as a "Web application"). The Web application includes page description data described in a Web page description language such as HTML (HyperText Markup Language) and script data described in a script language such as JavaScript (registered trademark). The page description data of the Web application that realizes the extraction display unit 250 includes the image data received by the image receiving unit 221 and the extraction information extracted from the image data by the character extracting unit 230. Note that the page description data is not limited to HTML, and may be any data that can be interpreted by the information processing terminal 130, which is a client terminal, and operated as a Web application. Specifically, the page description data may be structured data in an XML (Extensible Markup Language) format or a JSON (JavaScript Object Notation) format.

[0027] Data transmission unit 223 transmits the data generated by data generation unit 222 to information processing terminal 130. Correction unit 224 receives a correction instruction based on a correction operation by a user using information processing terminal 130 from information processing terminal 130, and corrects or adds to the extracted information based on the correction instruction. For example, when an instruction to correct character string information is received as an instruction to correct the extracted information, correction unit 224 corrects the extracted information stored in HDD 123. Also, for example, when an instruction to perform character recognition is received as an instruction to correct the extracted information, correction unit 224 causes character extraction unit 230 to re-execute character extraction processing, and adds character information extracted by the character extraction processing to the extracted information.

[0028] The file generating unit 225 receives a file generation instruction transmitted from an instruction transmitting unit 242 of the information processing terminal 130 described later and generates a file. Specifically, for example, the file generating unit 225 acquires file format information from the setting value of the scan setting transmitted from the image transmitting unit 215, and generates a file from the image data based on the setting value. The file transmitting unit 226 transmits the file generated by the file generating unit 225 to the storage server 140 and causes the storage server 140 to store the file.

[0029] Character extraction unit 230 executes character extraction processing on image data received by image receiving unit 221, and acquires information indicating the extraction result. Character extraction unit 230 has area detection unit 231, object determination unit 232, area setting unit 233, and character recognition unit 234. Area detection unit 231 detects an area including characters (hereinafter referred to as a "character area") from image data. Specifically, for example, area detection unit 231 first extracts a group of black pixels and a group of white pixels from an image indicated by image data (hereinafter referred to as a "document image"). Next, area detection unit 231 detects the character area by classifying the document image into characteristic image areas such as characters, pictures or figures, tables, frames, or lines based on the shape, size, or state of the extracted groups.

[0030] The method of detecting a character region is not limited to the above-mentioned method, and known techniques can be used. For example, semantic segmentation using deep learning may be applied as a method of detecting a character region in the region detection unit 231. Specifically, in this case, the region detection unit 231 first labels each pixel of the document image as a pixel forming a character (hereinafter referred to as a "character pixel") or not, and then detects the character region by combining surrounding character pixels into one region.

[0031] Based on the height and width of the character area and its positional relationship with other character areas, the subject determination unit 232 determines a character area that is not a subject of character recognition (hereinafter referred to as a "non-subject character area"). The area setting unit 233 sets a character area (hereinafter referred to as a "subject character area") to be subjected to character recognition based on the determination result of the subject determination unit 232. The character recognition unit 234 performs character recognition on the subject character area set by the area setting unit 233, and stores the result of the character recognition in the HDD 123 as extraction information. Here, the extraction information includes information on the character string extracted by the character recognition (character code), information indicating the position of the character area corresponding to the extracted character string, information indicating the attributes of the character, language information, and the confidence level of the character recognition.

[0032] The information processing terminal 130 has, as its functional components, a data receiving unit 241, an instruction transmitting unit 242, an executing unit 243, and an extraction display unit 250. The extraction display unit 250 also has a UI display unit 251, an image display unit 252, an area display unit 253, an area selecting unit 254, a partial display unit 255, a character display unit 256, and an input accepting unit 257. Each unit that the information processing terminal 130 has as its functional components is realized by the CPU 131 transferring a control program stored in the HDD 133 to the RAM 132 and executing it.

[0033] The data receiving unit 241 receives data transmitted from the data transmitting unit 223, and stores the data in the HDD 303. The instruction transmitting unit 242 transmits an instruction corresponding to a user's operation input received by the input receiving unit 257 to the image processing server 120. Here, the instructions transmitted to the image processing server 120 include, for example, an instruction to generate a file, an instruction to correct extracted information, and an instruction to execute character extraction.

[0034] The execution unit 243 is, for example, a Web browser, and executes data received by the data receiving unit 241 as a Web application, thereby causing the CPU 131 to function as the extraction display unit 250. The extraction display unit 250 displays the rendering result of the page description data and script data of the Web application by the execution unit 243 as a GUI (Graphical User Interface) screen on the operation unit 136. Here, the Web application functioning as the extraction display unit 250 can confirm the image data received by the image receiving unit 221, and confirm and correct the extraction information that is the result of character recognition by the character recognition unit 234. Hereinafter, the extraction display unit 250 will be described as functioning by the execution of a Web application, but is not limited thereto, and the extraction display unit 250 may be realized by the execution of a control program stored in the HDD 133.

[0035] The functions of each of the units 251 to 257 in the extraction display unit 250 are realized by the rendering results of the page description data and script data of the Web application. The UI display unit 251 displays operation components such as buttons operated by the user, and UI components that display information such as the progress of processing. The image display unit 252 displays a document image indicated by image data received by the image receiving unit 221 and included in the data received by the data receiving unit 241. The area display unit 253 displays a character area included in the document image displayed by the image display unit 252. The area selection unit 254 selects a character area from one or more character areas displayed by the area display unit 253, based on an operational input by the user.

[0036] The partial display unit 255 cuts out an image (hereinafter referred to as a "partial image") corresponding to the character area selected by the area selection unit 254 from the document image, and displays the cut-out partial image. In this embodiment, the partial display unit 255 is described as being capable of simultaneously displaying a plurality of partial images. The character display unit 256 displays character information extracted from each character area by character recognition processing. The input acceptance unit 257 accepts an operation input by the user to the UI widgets displayed by the UI display unit 251, the image display unit 252, the area display unit 253, and the character display unit 256, and instructs each unit to execute a process corresponding to the operation input. The user can modify the character information displayed by the character display unit 256 through the operation input to the input acceptance unit 257.

[0037] <Processing flow of image processing system> FIG. 3 is a sequence diagram showing an example of a process flow of the image processing system 100 according to the first embodiment. Specifically, the sequence diagram shown in FIG. 3 shows a process flow in which image data obtained by scanning with the image forming apparatus 110 is converted into a file and transmitted to the storage server 140. In a normal state, a main screen on which buttons for using each function that can be provided are arranged is displayed on the operation unit 103 of the image forming apparatus 110. An application program that realizes a function for transmitting image data obtained by scanning to the image processing server 120 is installed in advance in the image forming apparatus 110. As a result, a button for executing the application program is displayed on the main screen displayed on the operation unit 103. When this button is pressed, a scan setting screen for setting scan, a transmission setting screen for transmitting image data obtained by scanning to the image processing server 120, a scan start button for starting scan, and the like are displayed on the operation unit 103. A user can perform various settings by performing operation input on the scan setting screen and the transmission setting screen.

[0038] When the scan start button is pressed, the process of the sequence diagram shown in Fig. 3 is started. Note that the image forming apparatus 110 may provide a screen that is different from the scan setting screen or the transmission setting screen and that allows settings other than the above-mentioned settings to be made before the start of scanning. For example, the image forming apparatus 110 provides a screen for setting what processing will be performed in the subsequent processing, such as a screen for setting the processing mode in the image processing server 120, such as a processing time priority mode or an accuracy priority mode. The user can set what processing will be performed in the subsequent processing by performing an operation input on the screen before pressing the scan start button. Note that in the following explanation, the symbol "S" means a step.

[0039] When a scan start button is pressed on the image forming apparatus 110, in S300, the image reading unit 211 of the image forming apparatus 110 scans the document placed on the scanner 102 in accordance with various scan settings set on the scan setting screen. Next, in S301, the image reading unit 211 of the image forming apparatus 110 generates document image data (image data) corresponding to the document based on a signal obtained by the scan executed in S300. Next, in S302, the image transmission unit 215 of the image forming apparatus 110 transmits the image data generated in S301 to the image processing server 120.

[0040] Next, in S303, the image receiving unit 221 of the image processing server 120 receives the image data transmitted in S302. Next, in S304, the character extraction unit 230 of the image processing server 120 executes character extraction processing on the image data received in S303, and acquires information indicating the result of the character extraction. Details of the character extraction processing will be described later with reference to FIG. 5. Next, in S305, the data generating unit 222 of the image processing server 120 generates data used for the operation of the extraction display unit 250 of the information processing terminal 130 from the information acquired in S304. Next, in S306, the data transmitting unit 223 of the image processing server 120 transmits the data generated in S305 to the information processing terminal 130.

[0041] Next, in S307, the data receiving unit 241 of the information processing terminal 130 receives the data transmitted in S306. Next, in S308, the extraction display unit 250 of the information processing terminal 130 executes the display process of the extracted information by rendering the data received in S307 by the execution unit 243. Details of the display process of the extracted information will be described later with reference to FIG. 7. Next, in S309, the instruction transmitting unit 242 of the information processing terminal 130 determines whether the instruction received from the user via the input receiving unit 257 is an instruction to perform character recognition in the non-target character region selected by the user. If it is determined in S309 that the instruction is an instruction to perform character recognition, in S310, the instruction transmitting unit 242 transmits information instructing the correction unit 224 of the image processing server 120 to perform character recognition in the selected non-target character region.

[0042] After S310, in S311, the correction unit 224 of the image processing server 120 receives the information indicating an instruction to execute character recognition in the selected non-target character region transmitted in S310, and extracts characters from the selected non-target character region based on the information. After S311, the image processing system 100 returns to S305, and the processes from S305 onwards are repeatedly executed.

[0043] If it is determined in S309 that the instruction is not an instruction to perform character recognition, then in S312, the instruction transmission unit 242 of the information processing terminal 130 determines whether the instruction received from the user via the input acceptance unit 257 is an instruction to create a file. If it is determined in S312 that the instruction is not an instruction to create a file, the information processing terminal 130 returns to S408, and the input acceptance unit 257 waits for a new instruction from the user. If it is determined in S312 that the instruction is an instruction to create a file, then in S313, the instruction transmission unit 242 of the information processing terminal 130 transmits information indicating an instruction to create a file to the image processing server 120. At this time, if the character corresponding to any character area has been corrected by the user via the input acceptance unit 257, the instruction transmission unit 242 also transmits information indicating an instruction to correct the extracted information based on the correction to the correction unit 224 of the image processing server 120.

[0044] After S313, in S314, the file generating unit 225 of the image processing server 120 receives information indicating a file generation instruction transmitted in S313, and generates a file corresponding to the image data received in S303. At this time, if information indicating a correction instruction for the extracted information is also received, the correction unit 224 of the image processing server 120 receives the information indicating the correction instruction, and corrects the extracted information before the file generation process by the file generating unit 225. After S314, in S315, the file transmitting unit 226 of the image processing server 120 transmits the file generated in S314 to the storage server 140. After S315, the CPU 141 of the storage server 140 receives the file transmitted in S315 in S316, and stores the file received in S316 in the HDD 143 and saves it in S317. After S317, the image processing system 100 ends the series of processes shown in the sequence diagram of FIG. 3.

[0045] <Character extraction process> The character extraction process in the character extraction unit 230 of the image processing server 120 will be described in detail with reference to Figs. 4 to 6. This process is performed to extract a necessary character string from a document image for the purpose of assigning a file name or transcribing a character string. At this time, it may be necessary to limit the processing time or processing load from the viewpoint of responsiveness to a user or the hardware resources of the image processing server 120. In particular, character recognition takes processing time, and the processing time generally increases in proportion to the number of characters to be recognized. For this reason, it is desirable to perform character recognition only on character strings to be assigned a file name or transcribed, and not on other character strings.

[0046] FIG. 4 is a diagram for explaining a document image 401 according to the first embodiment, and character strings to be extracted and character strings not to be extracted in the document image 401. Specifically, FIG. 4(a) is a diagram showing an example of the document image 401 according to the first embodiment, and is a diagram showing an example of the document image 401 when the document scanned by the image forming device 110 is an application form. The document image 401 includes, as image areas, an image area (hereinafter referred to as a "fill-in area") 402 corresponding to a fill-in field, a character area (hereinafter referred to as a "title area") 403 corresponding to the title of the document, and a character area (hereinafter referred to as a "description area") 410 corresponding to an explanation. The fill-in field area 402 also includes character areas (hereinafter referred to as "item name areas") 404 to 406 corresponding to item names, and character areas (hereinafter referred to as "written character areas") 407 to 409 corresponding to written characters that are item values.

[0047] Among these character regions, the character regions to be subjected to character recognition are the title region 403, the item name regions 404 to 406, and the written character regions 407 to 409. For example, in a system that uses extracted character strings as file names for image data, the document title or some of the written characters may be added as part of the file name. Also, in a system that automatically transcribes extracted character strings, in addition to the written characters, the item names may be character-recognized as keys for extracting predetermined items.

[0048] On the other hand, the character regions that are not the target of character recognition are the explanatory text regions 410. This is because explanatory text is not a character string that characterizes a document like a document title, nor is it a character string that differs from document to document like handwritten characters. Character strings such as explanatory text are generally often printed characters, and are characterized by small character sizes. For this reason, a method may be adopted in which character recognition is not performed on character regions with character strings of small character sizes. However, if character regions that are not subject to character recognition are determined based only on character size, character strings that should be extracted may not be extracted.

[0049] 4(b) to 4(d) are diagrams showing an example of visualization of the results of detecting a character region corresponding to a character string to be extracted. In FIG. 4(b) to 4(d), a region surrounded by a rectangle represents a character region that includes characters with a character size smaller than a predetermined threshold size, such as 8pt (point), among the detected character regions. FIG. 4(b) shows an example of a character string that includes a mixture of printed characters and handwritten characters, and is an example of a character string representing a date. In the example shown in FIG. 4(b), a character region corresponding to the character string of "Application date" which is an item name, a character region corresponding to the character string of "Gregorian calendar year", and a character region corresponding to the character string (characters) of "day" among "year", "month", and "day" are detected as character regions with small character sizes.

[0050] FIG. 4(c) shows an example of a character string made of handwritten characters, which is an example of a character string representing a date. In the example shown in FIG. 4(c), a character area corresponding to the character string (character) "-" on the right of two "-" (hyphens) is detected as a character area with a small character size. Character areas corresponding to characters expressed by symbols in a handwritten character string, not limited to "-" in a character string representing a date, tend to be easily detected as a character area with a small character size. FIG. 4(d) shows an example of a character string made of handwritten characters, which is an example of a character string composed of multiple alphabets. In the example shown in FIG. 4(d), a character area corresponding to a part of an alphabet written in lowercase is detected as a character area with a small character size.

[0051] As shown in Figs. 4(b) to (d) as an example, if it is determined whether or not a character region is to be subjected to character recognition based only on the character size, there are cases where it is not possible to extract a character string to be extracted. Therefore, it is necessary to determine a character region not to be subjected to character recognition based on conditions other than the character size in addition to the determination based on the character size. Specifically, the character extraction unit 230 determines a character region not to be subjected to character recognition, that is, a region not to be subjected to character recognition (non-target character region) by performing a determination based on the number of characters in the character region or the positional relationship between the character regions in addition to the determination based on the character size. For example, the character extraction unit 230 sets a character region such as the explanatory text region 410 that is composed of characters with small character sizes, spans multiple lines, and includes a character string with a large number of characters per line as a character region not to be subjected to character recognition. As a result, the number of characters to be recognized is reduced, and the processing time required for the character recognition process is suppressed. Furthermore, compared to the case where it is determined based only on the character size, the character string to be extracted is more appropriately extracted, and the extraction accuracy of the character string to be extracted is improved.

[0052] Fig. 5 is a flowchart showing an example of a processing flow in character extraction unit 230 according to the first embodiment, and is a flowchart showing an example of the flow of character extraction processing in S304 shown in Fig. 3. First, in S501, region detection unit 231 of character extraction unit 230 detects a character region in a document image represented by the image data received in S303. Character extraction unit 230 determines a character region (target character region) for character recognition from among the multiple character regions detected in S501.

[0053] Hereinafter, the area detection unit 231 will be described as detecting an area recognized as a character string that fits in one line as a character area, and outputting the detection result as data in which information indicating a circumscribing rectangle of each detected character area is listed in JSON format. For example, the circumscribing rectangle is represented by information indicating the x-coordinate and y-coordinate of the upper left vertex of the circumscribing rectangle, and the length (width) of the circumscribing rectangle in the main scanning direction and the length (height) of the circumscribing rectangle in the sub-scanning direction, when the x-axis is taken in the main scanning direction and the y-axis is taken in the sub-scanning direction in the document image. Note that the detection result is not limited to data listed in JSON format, and may be expressed in, for example, XML format. Also, the circumscribing rectangle may be expressed by the x-coordinate and y-coordinate of the upper left vertex and the x-coordinate and y-coordinate of the lower right vertex of the circumscribing rectangle. After S501, in S502, the target determination unit 232 of the character extraction unit 230 determines whether the character area detected in S501 is a target for character recognition.

[0054] FIG. 6 is a flowchart showing an example of a processing flow in object determination unit 232 according to the first embodiment, and is a flowchart showing an example of the flow of object determination processing in S502. First, in S601, object determination unit 232 selects an arbitrary character area from among the multiple character areas detected in S501. Hereinafter, the character area selected in S601 will be described as a "selected character area". Next, in S602, object determination unit 232 calculates the size (character size) of the characters included in the selected character area. Specifically, for example, object determination unit 232 calculates the character size based on the height of the selected character area, that is, the length in the sub-scanning direction of a circumscribing rectangle of the selected character area.

[0055] Next, in S603, object determination unit 232 determines whether the character size calculated in S602 is smaller than a predetermined size threshold value (hereinafter simply referred to as "threshold value") such as 8pt. If it is determined in S603 that the character size is smaller than the threshold value, object determination unit 232 sets the selected character region as a non-target candidate region in S604. As a result of the processing in S604, a character region containing only characters with small character sizes becomes a candidate for a character region not subject to character recognition. If it is determined in S603 that the character size is not smaller than the threshold value, that is, is equal to or larger than the threshold value, object determination unit 232 sets the selected character region as a target character region in S605.

[0056] After S604 or S605, in S606, the object determination unit 232 determines whether or not all the character regions detected in S501 have been selected in S601. If it is determined in S606 that all the character regions have not been selected, that is, that there is an unselected character region, the object determination unit 232 returns to the process of S601 and selects a character region that has not yet been selected in S601. Thereafter, the object determination unit 232 repeatedly executes the processes from S601 to S606 until it is determined in S606 that all the character regions have been selected. If it is determined in S606 that all the character regions have been selected, in S607, the object determination unit 232 selects an arbitrary non-target candidate region from the multiple non-target candidate regions set in S604. Hereinafter, the non-target candidate region selected in S607 will be described as a "selected candidate region".

[0057] After S607, in S608, the subject determination unit 232 determines whether or not another non-target candidate region exists within a predetermined range in the main scanning direction from the selected candidate region. If it is determined in S608 that another non-target character region exists, in S609, the subject determination unit 232 combines the selected candidate region with the other non-target character region to generate a new character region, and sets the generated character region as the non-target candidate region. In this case, the subject determination unit 232 sets the newly generated non-target candidate region as the selected candidate region in place of the selected candidate region, and deletes the two non-target candidate regions before the combination. After S609, the subject determination unit 232 returns to the process of S608, and repeatedly executes the processes of S608 and S609 until it is determined in S608 that no other non-target candidate region exists.

[0058] If it is determined in S608 that there are no other non-target candidate areas, then in S610, the target determination unit 232 determines whether the number of lines of character strings included in the selected candidate area is greater than a predetermined threshold number of lines (hereinafter, simply referred to as the "threshold"). Here, the number of lines of character strings included in the selected candidate area may be considered to be the same as the number of times the selected candidate area is combined with other non-target candidate areas in S609. If it is determined in S610 that the number of lines is greater than the threshold, then in S611, the target determination unit 232 sets the selected candidate area to a non-target character area. For example, if the threshold is set to 2, a selected candidate area containing three or more lines of character strings is set to a non-target character area.

[0059] If it is determined in S610 that the number of lines is not greater than the threshold, that is, that the number is equal to or less than the threshold, then in S612, the object determination unit 232 calculates the virtual number of characters for each line of the character string included in the selection candidate area. Here, the virtual number of characters is the virtual number of characters of the character string included in the character area. In this embodiment, since only the size of the circumscribing rectangle of the character area is known, it is not possible to obtain the accurate number of characters of the character string included in the character area. Therefore, the object determination unit 232 calculates, for example, the value obtained by dividing the length (width) of the circumscribing rectangle of the character area in the main scanning direction by the length (height) of the circumscribing rectangle of the character area in the sub-scanning direction, that is, the value obtained by dividing by the character size calculated in S602, as the virtual number of characters in the character area. Note that the method of calculating the virtual number of characters is not limited to the above-mentioned method, and a specific number of characters may be calculated or obtained using other methods.

[0060] After S612, in S613, the object determination unit 232 determines whether or not the maximum number of virtual characters per line of the character string included in the selection candidate area is greater than a predetermined threshold value of the number of characters (hereinafter, simply referred to as the "threshold value"). For example, when the character string included in the selection candidate area is one line, the virtual number of characters in that line is the maximum number of virtual characters. When the character string included in the selection candidate area is multiple lines, the virtual number of characters in the line with the largest number of virtual characters among the virtual numbers of all lines is the maximum number of virtual characters. When it is determined in S613 that the maximum number of virtual characters is greater than the threshold value, the object determination unit 232 executes the process of S611 to set the selection candidate area to a non-target character area. When it is determined in S613 that the maximum number of virtual characters is not greater than the threshold value, that is, is equal to or less than the threshold value, in S614, the object determination unit 232 sets the selection candidate area to a target character area.

[0061] After S611 or S614, in S615, the object determination unit 232 determines whether or not all of the non-target candidate regions set in S604 have been selected in S607. If it is determined in S615 that all of the non-target candidate regions have not been selected, that is, that there are non-target candidate regions that have not been selected, the object determination unit 232 returns to S607 and selects a non-target candidate region that has not yet been selected in S607. Thereafter, the object determination unit 232 repeatedly executes the processes from S607 to S615 until it is determined in S615 that all of the non-target candidate regions have been selected. If it is determined in S615 that all of the non-target candidate regions have been selected, the object determination unit 232 ends the process of the flowchart shown in FIG. 6, that is, the object determination process of character recognition in S502 shown in FIG. 5.

[0062] After S502, in S503, area setting unit 233 of character extraction unit 230 sets a character area (hereinafter referred to as a "recognition execution area") in which character recognition is performed in the document image based on the determination result of S502 set in S605, S611, or S614. Specifically, the character area set as the target character area in S605 or S614 is set as the recognition execution area. The process of S503 limits the area in which character recognition is performed, which leads to a reduction in the processing time for character recognition. Note that area setting unit 233 may change the method of setting the recognition execution area in accordance with the characteristics of an OCR (Optical Character Recognition) engine used when performing character recognition, which will be described later.

[0063] Next, in S504, the character recognition unit 234 of the character extraction unit 230 executes character recognition for the recognition execution area set in S503 using an OCR engine, and outputs the result of the character recognition as extraction information. The output extraction information is stored in the HDD 123. Here, the character recognition unit 234 does not need to include information on all recognized characters in the extraction information, and may select characters to be included in the extraction information from among the recognized characters by any method. For example, the character recognition unit 234 may determine whether the recognized characters are extracted from a specific document area in the document image and decide whether to include them in the extraction information, or may determine whether the recognized characters are specific characters and decide whether to include them in the extraction information.

[0064] In addition, the character recognition unit 234 may change the manner of inputting the information of the recognition execution area to the OCR engine according to the characteristics of the OCR engine to be used and the system in which the OCR engine is used. For example, data obtained by cutting out an area corresponding to the recognition execution area from the image data may be input to the OCR engine, or the range of the recognition execution area in the image data for character recognition may be selected and instructed to the OCR engine. In addition, when the recognition execution area is a combination of character areas in S609, the character recognition unit 234 may separate the recognition execution area to the state before the combination, and cause the OCR engine to perform character recognition on each of the separated character areas. After S504, the character extraction unit 230 ends the process of the flowchart shown in FIG. 5, that is, the character extraction process in S304 shown in FIG. 3.

[0065] The extracted information output in S304 is transmitted to the information processing terminal 130 and displayed on the operation unit 136 of the information processing terminal 130 in S308. A character string used for assigning a file name or transcribing a character string is selected by the user in S308 from the extracted information output in S304.

[0066] According to such a series of processes of the character extraction unit 230, it is possible to appropriately determine the character region to be the target of character recognition. In addition, by limiting the character region to be the target of character recognition by appropriately determining the character region to be the target of character recognition, it is possible to suppress the processing time of character recognition in the document image and improve the extraction accuracy of the character string to be extracted. It should be noted that the process of determining the character region to be the target of character recognition in this series of processes does not necessarily need to be executed, and may be executed only when, for example, the processing time is predicted to be longer than a predetermined time. Specifically, for example, when the total number of virtual characters of the character string included in the document image exceeds a threshold, a target determination process for character recognition may be performed. On the other hand, when the total number is equal to or less than the threshold, all character regions may be set as target character regions and character recognition may be performed on all character regions.

[0067] <Extracted information display process> The processing in the extraction display unit 250 of the information processing terminal 130 will be described with reference to Fig. 7. Fig. 7(a) is a diagram showing an example of a display screen 701 displayed on the operation unit 136 by the extraction display unit 250 according to the first embodiment. By performing an operation input while checking the display screen 701, the user can select, confirm, and correct information on a character string extracted by character recognition, which is included in the extraction information, and can also instruct the execution of character recognition on a character region that has been set as a non-target character region.

[0068] The UI display unit 251 of the extraction display unit 250 displays the display screen 701 on the operation unit 136. The document image is displayed in the image display area 704. The document image displayed in the image display area 704 is the document image to be processed. FIG. 7(b) is an enlarged view of the image display area 704 shown in FIG. 7(a). The area display unit 253 of the extraction display unit 250 superimposes an image such as a rectangle indicating the position of the character area included in the document image on the document image displayed in the image display area 704 based on information indicating the position of the extracted character string included in the extraction information. FIG. 7(b) shows a rectangle drawn by a dashed line as an example of an image indicating the position of the character area, and the rectangle corresponds to a circumscribing rectangle of the area of ​​the character string extracted from the document image. The page display area 702 displays the total number of pages of the document read by the image reading unit 211 and the page number of the document image displayed in the image display area 704. From the display contents of the page display area 702, the user can recognize which page of the document the document image displayed in the image display area 704 corresponds to.

[0069] The button 703 is a button for changing the document image displayed in the image display area 704. When the user presses the button 703 by operating the operation unit 136, a document image corresponding to a page previous to or following the page of the original corresponding to the document image displayed before the button 703 is displayed in the image display area 704, and the processing target is switched to the document image. If the button 703 is pressed in a state in which information on the character string included in the extracted information corresponding to the document image to be processed has been corrected, information indicating the content of the correction is temporarily stored in the HDD 133 or the like, and the document image displayed in the image display area 704 is changed. On the other hand, if the button 703 is pressed in a state in which information on the character string included in the extracted information has not been corrected, for example, the document image displayed in the image display area 704 is immediately changed.

[0070] When a user operates operation unit 136 to specify coordinates in a document image to be displayed in image display area 704, area selection unit 254 of extraction display unit 250 selects a character area corresponding to the specified coordinates from among the multiple character areas indicated by the extraction information. Partial display unit 255 of extraction display unit 250 cuts out an image corresponding to the character area selected by area selection unit 254 from the document image, and displays the cut-out image in partial display area 705 as a partial image.

[0071] The partial image displayed in the partial display area 705 is not limited to the one corresponding to the character area corresponding to the coordinates of the document image specified by the user. For example, the area selection unit 254 may display the partial images corresponding to all the character areas indicated by the extraction information. Also, for example, the data generation unit 222 may set in advance the character area corresponding to the partial image to be displayed in the partial display area 705, and the area selection unit 254 may select the character area based on the setting and display the partial image. Also, for example, the area selection unit 254 may store the coordinates of the character area previously selected by the user in the HDD 133 or the like, and the partial display unit 255 may display the partial image corresponding to the character area among all the character areas indicated by the extraction information based on the information of the coordinates.

[0072] The character display unit 256 displays character information extracted from a character area corresponding to a partial image in the character display area 706. The user can modify the character information displayed in the character display area 706 by operating the operation unit 136. The button 707 is a button for instructing a file generation process after completing confirmation and modification of the character information extracted from the document image to be processed. When the button 707 is pressed by the operation of the operation unit 136 by the user accepted by the input acceptance unit 257, the information processing terminal 130 executes the process of S313 shown in FIG. 3 to instruct the file generation unit 225 of the image processing server 120 to generate a file. Note that, if the user has modified the character information before the button 707 is pressed, the information processing terminal 130 instructs the file generation process and also instructs the modification unit 224 of the image processing server 120 to modify the character information. In this case, the image processing server 120 executes the file generation process after executing the modification process of the character information.

[0073] In FIG. 7B, the region surrounded by a dashed rectangle is a character region set as a target character region by the character extraction unit 230, and indicates a character region from which a character string is extracted by the character extraction unit 230 as described above. On the other hand, in this embodiment, the region 708 is a character region set as a non-target character region by the character extraction unit 230, and character recognition processing has not been performed on the region 708. When the region 708, which is a non-target character region, is selected by an operation by a user, the instruction sending unit 242 executes the process of S310 to send information to the correction unit 224 of the image processing server 120 instructing the execution of character extraction in the selected non-target character region. The correction unit 224 executes character recognition on the region 708, which is a non-target character region, based on the received information. At this time, the region on which character recognition is performed may be only the selected non-target character region, or may be all non-target character regions in the document image. It should be noted that the area enclosed by the dashed rectangle in FIG. 7(b) may be constantly displayed on the actual display screen 801, or may be displayed only when any coordinate within the area is selected.

[0074] In this manner, a series of processes by the extraction display unit 250 allows the selection, confirmation, and correction of extracted character strings, as well as an instruction to execute character recognition for character regions that have been set as non-target character regions. According to the image processing system 100 having the image processing server 120 configured as described above, the character regions to be the targets of character recognition can be appropriately determined, compared to the case where the character regions to be subjected to character recognition processing are determined based only on the character size. Furthermore, by limiting the character regions to be the targets of character recognition by appropriately determining the character regions to be the targets of character recognition, the processing time required for character recognition processing can be reduced, while improving the extraction accuracy of the character string to be extracted.

[0075] [Second embodiment] In the first embodiment, a form was described in which all character regions that meet the conditions are set as non-target character regions in the process of S611 in FIG. 6. With such a setting method, the processing time of the character extraction process according to the first embodiment can be minimized. In this embodiment, a form is described in which character recognition is performed on more character regions while completing the character extraction process within a certain time by setting an upper limit on the processing time of the character extraction process. This makes it possible to more appropriately determine the character regions to be the targets of character recognition while maintaining user operability, and further improve the extraction accuracy of the character string to be extracted. In the description of this embodiment, the description of the same configuration and processing as in the first embodiment will be omitted, and only the differences will be described.

[0076] Fig. 8 is a flowchart showing an example of a processing flow in the object determination unit 232 according to the second embodiment, and is a flowchart showing an example of the flow of object determination processing for character recognition in S502 in Fig. 5. In the flowchart shown in Fig. 8, S611 is changed to S804, and S801 to S803 and S805 are added, as compared with the flowchart in Fig. 6 showing the processing flow in the object determination unit 232 according to the first embodiment.

[0077] First, in S801, the object determination unit 232 calculates the processing time (hereinafter referred to as "first estimated processing time") estimated when character recognition processing is performed on all character regions detected in S501. Here, since the processing time of the character recognition processing is approximately proportional to the number of characters included in the character region, the first estimated processing time can be calculated based on the total number of virtual characters of the character string included in each character region detected in S501. Next, in S802, the object determination unit 232 determines whether the first estimated processing time calculated in S801 is equal to or less than the processing time allowed for character recognition processing (hereinafter referred to as "allowable time"). If it is determined in S802 that the first estimated processing time is equal to or less than the allowable time, in S803, the object determination unit 232 sets all character regions detected in S501 as object character regions. After S803, the object determination unit 232 ends the processing of the flowchart shown in FIG. 8, that is, the processing of S502.

[0078] If it is determined in S802 that the first expected processing time is not equal to or less than the permissible time, i.e., is longer than the permissible time, the object determination unit 232 executes the processes from S601 to S610 as appropriate. If it is determined in S610 that the number of lines is not greater than the threshold, i.e., is equal to or less than the threshold, the object determination unit 232 executes the processes of S612 and S613 as in the first embodiment. If it is determined in S613 that the maximum number of virtual characters is not greater than the threshold, i.e., is equal to or less than the threshold, the object determination unit 232 executes the process of S614 as in the first embodiment. If it is determined in S610 that the number of lines is greater than the threshold, or if it is determined in S613 that the maximum number of virtual characters is greater than the threshold, the object determination unit 232 calculates the priority of the selection candidate area in S804.

[0079] In the following description, the priority is set based on the total number of virtual characters in the string of characters included in the non-target candidate area, and a higher priority is set for the non-target candidate area with a larger total number of virtual characters. In addition, for non-target candidate areas with the same total number of virtual characters, the priority of the non-target candidate area with the larger number of lines is set higher, and if the number of lines is also the same, the priority of the non-target candidate area with the larger maximum number of virtual characters per line is set higher. Note that the method of setting the priority is not limited to the above-mentioned method, and for example, the priority may be set based on the total value of weighted values ​​for the total number of virtual characters, the number of lines, and the maximum number of virtual characters per line.

[0080] After S614 or S804, the subject determination unit 232 executes the process of S615. If it is determined in S615 that all non-target candidate regions have not been selected, that is, that there are non-target candidate regions that have not been selected, the subject determination unit 232 returns to the process of S607, as in the first embodiment. If it is determined in S615 that all non-target candidate regions have been selected, the subject determination unit 232 executes a subject setting process in S805 to set the non-target candidate regions as target character regions or non-target character regions based on the priority calculated in S804 and the allowable time.

[0081] FIG. 9 is a flowchart showing an example of a process flow in the target determination unit 232 according to the second embodiment, and is a flowchart showing an example of the flow of the target setting process in S805. First, in S901, the target determination unit 232 selects the character area with the highest priority calculated in S804 from among a plurality of character areas set as non-target candidate areas (S901). Hereinafter, the character area that is the non-target candidate area selected in S901 will be described as a "priority candidate area". Next, in S902, the target determination unit 232 calculates a processing time (hereinafter referred to as a "second estimated processing time") estimated when character recognition processing is performed on the priority candidate area selected in S901. Here, since the processing time of the character recognition processing is approximately proportional to the number of characters included in the character area, the second estimated processing time can be calculated based on the total number of virtual characters of the character string included in the priority candidate area.

[0082] Next, in S903, the subject determination unit 232 changes the setting of the priority candidate area to a non-target character area, and excludes the priority candidate area from the target of the character recognition process. Next, in S904, the subject determination unit 232 calculates the sum of the second estimated processing times calculated in S902 so far, and obtains a third estimated processing time. In other words, the third estimated processing time is the sum of the second estimated processing times corresponding to each non-target candidate area excluded from the target of the character recognition process in S903, among all the non-target candidate areas whose priorities were calculated in S804.

[0083] Next, in S905, the object determination unit 232 determines whether the third estimated processing time acquired in S904 is less than a value (hereinafter referred to as the "threshold value") obtained by subtracting the allowed time from the first estimated processing time. Here, the value obtained by subtracting the allowed time from the first estimated processing time is the time that is estimated to exceed the allowed time when character recognition processing is performed on all character regions detected in S501. For example, if the allowed time is 4 seconds and the first estimated processing time is 7 seconds, the threshold value is 3 seconds, which is the first estimated processing time of 7 seconds minus the allowed time of 4 seconds.

[0084] If it is determined in S905 that the third estimated processing time is less than the threshold, the process returns to S901 and repeats the processes from S901 to S905 until it is determined that the third estimated processing time is not less than the threshold, i.e., is equal to or greater than the threshold. This is because the processing time of the character recognition process still exceeds the allowable time because the third estimated processing time is not equal to or greater than the threshold. In this case, it is necessary to change the setting of the non-target candidate region with a higher priority to a non-target character region until the processing time of the character recognition process falls within the allowable time.

[0085] If it is determined in S905 that the third estimated processing time is not less than the threshold, i.e., is equal to or greater than the threshold, then in S906, the subject determination unit 232 changes the settings of all remaining non-target candidate regions to target character regions. Here, all remaining non-target candidate regions are all non-target candidate regions that have a lower priority than the non-target candidate regions that were last changed to non-target character regions in S903 and excluded from the targets of the character recognition process. After S906, the subject determination unit 232 ends the process of the flowchart shown in FIG. 9, i.e., the process of S802.

[0086] According to the image processing server 120 having the target determination unit 232 configured as above, the number of character regions to be character-recognized can be increased while maintaining the operability of the user by ensuring that the character recognition process is completed within the allowable time. Specifically, the target determination unit 232 changes the setting to the non-target character region in order from the non-target candidate region with the highest priority so that the character recognition process is completed within the allowable time. Therefore, the non-target candidate region with the lowest priority can be set as the target character region. As a result, according to the image processing system 100 having the image processing server 120, the character region to be the target of the character recognition can be more appropriately determined while maintaining the operability of the user, compared to the case where the character region to be subjected to the character recognition process is determined based only on the character size. As a result, according to the image processing system 100, the extraction accuracy of the character string to be extracted can be improved while suppressing the processing time required for the character recognition process. Note that the above-mentioned method of assigning priority is to give a higher priority to the non-target candidate region whose setting should be changed to the non-target character region, but the method of assigning priority is not limited to the above-mentioned method. For example, the priority of the non-target candidate region whose setting should be changed to the target character region may be given a higher priority. In this case, the process of S805, that is, the process of the flowchart shown in FIG. 9, may be appropriately changed according to the method of assigning priority.

[0087] [Third embodiment] When a character region includes a handwritten character, the character region corresponding to one handwritten character may be separated into multiple character regions, and the character may not be properly extracted as the character to be extracted. In this embodiment, a configuration that enables proper extraction of a character while suppressing the processing time of character recognition processing even when a character region corresponding to one character is separated into multiple character regions is described. In the description of this embodiment, the description of the same configuration and processing as in the first embodiment is omitted, and only the differences are described.

[0088] FIG. 10 is a diagram showing an example of a character region 1000 including a string of handwritten characters, in which a character region corresponding to some characters constituting the string is detected as being a plurality of character regions 1001 and 1002. Of the character regions 1001 to 1003, the character regions 1001 and 1003 are detected as character regions including only characters whose character size is equal to or larger than a predetermined threshold size, such as 8pt. The character region 1002 is detected as a character region including characters whose character size is smaller than the threshold. It is desirable that the character regions 1001 to 1003 are detected as one character region 1000.

[0089] In the first embodiment, character areas 1001 and 1003 are set as target character areas in S502, and when they are set as recognition execution areas in S503, character recognition processing is executed on each of character areas 1001 and 1003 in S504. Since the character (character string) included in character area 1001 has a character size equal to or larger than a threshold, character areas 1001 and 1002 are not combined in S502. Therefore, character area 1002 is also set as a target character area, and individual character recognition processing is executed thereon separately from character areas 1001 and 1003. Here, since character area 1001 includes characters with parts missing, the result of the character recognition processing may not be appropriate.

[0090] As a method for solving such a problem, it is possible to consider a method in which character recognition processing is not performed on the character area detected in S501, but rather, character recognition processing is performed on the character area detected using a separate detection method with higher accuracy. By using a detection method with higher accuracy, character areas 1001 to 1003 are detected as one character area 1000. For example, as a detection method with higher accuracy, a handwritten / printed character separation process is used to detect the character area. The handwritten / printed character separation process is a process in which pixels corresponding to handwritten characters (hereinafter referred to as "handwritten pixels") are identified, and an image composed of only handwritten pixels is separated from an image composed of pixels corresponding to printed characters and the background.

[0091] For example, semantic segmentation using deep learning can be applied to handwriting / printed character separation processing. For training of handwriting / printed character separation processing, multiple handwritten and printed character images, in which an image containing only handwritten characters (hereafter referred to as "handwritten character image") and an image containing only printed characters (hereafter referred to as "printed character image") are superimposed, are used as training data. In addition, pixel values ​​of a binary image obtained by binarizing the regions of handwritten character images in the handwritten and printed character mixed image become label values ​​indicating handwritten pixels, and data of this handwritten binary image becomes training data. Using these training data and training data, a DNN (Deep Neural Network) configured to perform semantic segmentation is trained. Through such training, a DNN model that performs handwriting / printed character separation processing is generated.

[0092] When handwriting / printed character separation processing is performed on a document image, the learned DNN model labels each pixel to indicate whether it is a handwritten pixel or a printed pixel. The pixels of the document image are then separated according to the label values ​​resulting from the labeling, and two images are generated: a handwritten character image containing only handwritten pixels, and a printed character image containing type and ruled lines. Then, character region detection processing is performed on each of the generated handwritten character image and printed character image. Specifically, for example, the following processing steps (i) to (iv) are executed as the character region detection processing. (i) First, line detection is performed on the image to remove ruled lines. (ii) Next, connected pixels that are connected to surrounding pixels are detected from the image after the ruled lines have been removed. (iii) Next, the connected pixels are thickened to make it easier for the connected pixels to be connected to the surrounding pixels. (iv) Next, a circumscribing rectangle corresponding to a set of pixels connected to each other after thickening is determined as a character region for one line.

[0093] In the above-mentioned processing steps, the parameters of each process, or the method of determining or detecting connected pixels, etc., may be changed for handwritten character images and printed character images according to their respective characteristics. For example, for printed character images, connection determination is performed based on characteristics such as the fact that consecutive characters are the same size, while for handwritten character images, connection determination is performed taking into account the characteristic that the size of each character varies. This allows appropriate character area detection to be performed for both handwritten character images and printed character images, improving the accuracy of character area detection.

[0094] The processing time of the handwriting / printed character separation process is shorter as the number of characters to be processed is smaller. Therefore, it is better to have a smaller total number of characters to be processed in the handwriting / printed character separation process. In addition, in processing using a DNN model, pixels corresponding to ruled lines in an image have a negative effect on the accuracy of the handwriting / printed character separation process. Therefore, an image corresponding to a character region is not extracted from the document image, but an image is generated in which the pixels of the character region in the document image set as a non-target character region in S611 are replaced with pixels of the background color of the document image (hereinafter referred to as "white pixels"). Hereinafter, this image will be described as a mask image. For example, the mask image is generated by replacing the pixels of the character region in the document image set as a non-target character region in S611 with white pixels using information indicating the position and size of the character region. Furthermore, the handwriting / printed character separation process is performed on the generated mask image. By configuring the character region set as a non-target character region in S611 with white pixels, the accuracy of the handwriting / printed character separation process for the document image can be improved while reducing the processing time.

[0095] FIG. 11 is a flowchart showing an example of a process flow in the region setting unit 233 according to the third embodiment, and is a flowchart showing an example of the flow of the recognition execution region setting process shown in S503 of FIG. 5. The process of the flowchart shown in FIG. 11 is started after the process of S502 is completed. First, in S1101, the region setting unit 233 generates a mask image in which pixels of the character region set as the non-target character region in S611 in the document image are replaced with white pixels. Next, in S1102, the region setting unit 233 detects a character region by performing handwritten / printed character separation processing on the mask image generated in S1101. Next, in S1103, the region setting unit 233 sets the character region detected in S1102 as the target character region in the recognition execution region. After S1103, the region setting unit 233 ends the process of the flowchart shown in FIG. 11, that is, the process of S503 shown in FIG. 5.

[0096] In the third embodiment, the character region detected in S1102 is set as the target character region, and therefore the character region set as the target character region in S605 or S614 is ignored without being subjected to character recognition. Therefore, for example, in the third embodiment, the character region may be set as at least a region different from the target character region, non-target character region, and candidate character region in S605 and S614.

[0097] The image processing server 120 configured as above can improve the accuracy of character recognition of a character string composed of handwritten characters in which a character area corresponding to one character is detected as being separated into multiple character areas, while suppressing the processing time. As a result, the image processing system 100 having the image processing server 120 configured as above can more appropriately determine a character area to be the target of character recognition, compared to a case in which a character area to be subjected to character recognition processing is determined based only on the character size. As a result, the image processing system 100 can improve the accuracy of extraction of a character string to be extracted, while suppressing the processing time required for character recognition processing.

[0098] [Modification 1 of the third embodiment] In the third embodiment, the above-mentioned processing steps (i) to (iv) are performed as the character region detection process for each of the generated handwritten character image and the printed character image. However, the above-mentioned processing steps (i) to (iv) do not necessarily need to be performed for the generated printed character image as the character region detection process. In this case, for example, the above-mentioned processing steps (i) to (iv) are performed for the generated handwritten character image as the character region detection process. On the other hand, for a character region including a character string of printed characters, the character region set as the target character region in S605 or S614 may be set as the recognition execution region. In this case, among the character regions set as the target character region in S605 or S614, a character region including at least a part of a character region detected from the generated handwritten character image may be ignored without performing character recognition process. According to the image processing server 120 configured in this way, the processing time can be further reduced compared to the image processing server 120 according to the third embodiment. On the other hand, for a character string composed of handwritten characters, the handwritten / printed character separation process can be performed to improve the character recognition accuracy.

[0099] [Modification 2 of the third embodiment] In the third embodiment, a mask image is generated in which pixels in a character area set as a non-target character area in a document image are replaced with white pixels in S611, and handwriting / printed character separation processing is performed on the generated mask image. However, the processing target of the handwriting / printed character separation processing is not limited to the mask image, and the handwriting / printed character separation processing may be performed on a document image before pixels in a character area set as a non-target character area are replaced with white pixels. In this case, the handwriting / printed character separation processing is first performed on the document image to generate a handwritten character image and a printed character image, and the above-mentioned processing steps (i) to (iv) are performed as a character area detection processing only on the generated handwritten character image. In this way, a character area including a character string composed of handwritten characters is extracted. On the other hand, for a character area including a character string composed of printed characters, a printed character image is used as the document image, and a character area including a character string composed of printed characters is extracted by the method described in the first embodiment, and a character area that satisfies a condition is set as a target character area. The image processing server 120 configured in this way can also improve character recognition accuracy while suppressing processing time.

[0100] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0101] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0102] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0103] <Configuration 1> A detection means for detecting a character region from a document image; a specifying means for specifying a target character region, which is a character region to be subjected to character recognition processing, from among the plurality of character regions detected by the detecting means, based on information on the height and width of the character region and on a positional relationship between the character region and other character regions present around the character region; character recognition means for performing the character recognition process on the target character region identified by the identification means; 13. An image processing device comprising:

[0104] <Configuration 2> an area setting means for setting the target character area specified by the specifying means as a recognition execution area; and the character recognition means performs the character recognition process on the character area set in the recognition execution area; 2. The image processing device according to claim 1,

[0105] <Configuration 3> the specifying means includes a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the specifying means specifies, as the target character region, the character region in which the estimated character size is equal to or larger than a predetermined threshold, and the character region in which the estimated character size is less than the threshold and in which there are less than a predetermined number of other character regions surrounding the character region and in which the estimated character size is less than the threshold; 3. The image processing device according to configuration 1 or 2,

[0106] <Configuration 4> the specifying means includes a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; and the specifying means further specifies, as the target character region, a character region in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold value; 4. The image processing device according to configuration 3,

[0107] <Component 5> The identification means is a processing estimation means for estimating a processing time required when the character recognition process is performed on all the character regions identified as the target character regions; a priority assigning means for assigning a priority to the character areas other than the character area specified for the character recognition process based on information on the height and width of the character area and a positional relationship between the character area and other character areas existing around the character area; having the specifying means specifies, as the target character region, a part or all of the plurality of character regions to which the priorities have been assigned, based on the priorities, within a range in which the processing time estimated by the processing estimation means falls within a predetermined processing time; 5. The image processing device according to any one of configurations 1 to 4,

[0108] <Component 6> A detection means for detecting a character region from a document image; a first specifying means for specifying, from among the plurality of character regions detected by the detecting means, a non-target character region which is a character region for which character recognition processing is not performed, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a generating means for generating a mask image in which the non-target character region identified by the identifying means is masked in the document image; a second specifying means for specifying a target character region to be subjected to character recognition processing by performing handwritten / printed character separation processing on the mask image; a character recognition unit that performs the character recognition process on the target character region identified by the second identification unit; 13. An image processing device comprising:

[0109] <Component 7> an area setting means for setting the target character area identified by the second identification means as a recognition execution area; and the character recognition means performs the character recognition process on the character area set in the recognition execution area; 7. The image processing device according to configuration 6,

[0110] <Component 8> the first specifying means includes: a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the first identification means identifies, as the non-target character region, the character region in which the estimated character size is less than a predetermined threshold and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold; 8. The image processing device according to configuration 6 or 7,

[0111] <Component 9> the first specifying means includes a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; and the first identification means identifies, as the non-target character region, a character region in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold value; 9. The image processing device according to configuration 8,

[0112] <Component 10> the second specifying means performs the handwritten / printed character separation process on the mask image to generate a handwritten character image including pixels corresponding to handwritten characters and a printed character image including pixels corresponding to printed characters, specifies a handwritten character region including the handwritten characters based on the handwritten character image, specifies a printed character region including the printed characters based on the printed character image, and specifies the specified handwritten character region and the specified printed character region as the target character region; 7. The image processing device according to configuration 6,

[0113] <Component 11> the second identifying means performs the handwritten / printed character separation process on the mask image to generate a handwritten character image including pixels corresponding to handwritten characters, identifies a handwritten character area including the handwritten characters based on the handwritten character image, identifies the character area, excluding the character area corresponding to the handwritten character area, from the character areas other than the non-target character area among the plurality of character areas detected by the detection means, as a printed character area including printed characters, and identifies the identified handwritten character area and printed character area as the target character area; 7. The image processing device according to configuration 6,

[0114] <Component 12> a first identification means for identifying a handwritten character area including handwritten characters by performing a handwritten character / printed character separation process on a document image; a generating means for generating a mask image in which the handwritten character area identified by the first identifying means is masked in the document image; a detection means for detecting a character region from the mask image; a second specifying means for specifying, from among the plurality of character regions detected by the detecting means, a character region to be subjected to character recognition processing based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition unit that performs the character recognition process on the handwritten character area identified by the first identification unit and the character area identified by the second identification unit; 13. An image processing device comprising:

[0115] <Component 13> an area setting means for setting the handwritten character area specified by the first specifying means and the character area specified by the second specifying means as a recognition execution area; and the character recognition means performs the character recognition process on the handwritten character area and the character area set in the recognition execution area; 13. The image processing device according to claim 12,

[0116] <Component 14> the second specifying means including: a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the second specifying means specifies, as the character regions to be subjected to the character recognition process, the character regions in which the estimated character size is equal to or larger than a predetermined threshold value and the character regions in which the estimated character size is less than the threshold value and in which there are less than a predetermined number of other character regions surrounding the character regions and in which the estimated character size is less than the threshold value; 14. The image processing device according to configuration 12 or 13,

[0117] <Component 15> the second specifying means including: a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; and the second specifying means further specifies, as the character region to be subjected to the character recognition process, a character region in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold value; 15. The image processing device according to claim 14,

[0118] <Component 16> the first identifying means performs the handwriting / printed character separation process on the document image to generate a handwritten character image including pixels corresponding to the handwritten characters, and identifies the handwritten character area based on the handwritten character image; 16. The image processing device according to any one of configurations 12 to 15,

[0119] <Method 1> a detection step of detecting a character region from the document image; a specifying step of specifying a target character region, which is a character region to be subjected to character recognition processing, from among the plurality of character regions detected by the detection step, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition step of performing the character recognition process on the target character region identified by the identification step; 13. An image processing method comprising:

[0120] <Method 2> a detection step of detecting a character region from the document image; a first specification step of specifying, from among the plurality of character regions detected by the detection step, a non-target character region, which is a character region for which character recognition processing is not performed, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a generating step of generating a mask image in which the non-target character region identified by the identifying step is masked in the document image; a second identification step of identifying a target character region to be subjected to character recognition processing by performing handwritten / printed character separation processing on the mask image; a character recognition step of performing the character recognition process on the target character region identified by the second identification step; 13. An image processing method comprising:

[0121] <Method 3> A first identification step of identifying a handwritten character area including handwritten characters by performing handwriting / printed character separation processing on a document image; a generating step of generating a mask image in which the handwritten character region identified by the first identifying step is masked in the document image; a detection step of detecting a character region from the mask image; a second identification step of identifying a character region to be subjected to character recognition processing from among the plurality of character regions detected by the detection step, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition step of performing character recognition processing on the handwritten character area identified in the first identification step and the character area identified in the second identification step; 13. An image processing method comprising:

[0122] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 16. [Explanation of symbols]

[0123] 120 Image Processing Server 231 Area detection unit 232 Subject Determination Section 233 Character recognition section

Claims

1. A detection means for detecting a character region from a document image; a specifying means for specifying a target character region, which is a character region to be subjected to character recognition processing, from among the plurality of character regions detected by the detecting means, based on information on the height and width of the character region and on a positional relationship between the character region and other character regions present around the character region; character recognition means for performing the character recognition process on the target character region identified by the identification means; 13. An image processing device comprising:

2. an area setting means for setting the target character area specified by the specifying means as a recognition execution area; Further comprising: the character recognition means performs the character recognition process on the character area set in the recognition execution area; 2. The image processing device according to claim 1,

3. the specifying means includes a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the specifying means specifies, as the target character region, the character region in which the estimated character size is equal to or larger than a predetermined threshold, and the character region in which the estimated character size is less than the threshold and in which there are less than a predetermined number of other character regions surrounding the character region and in which the estimated character size is less than the threshold; 2. The image processing device according to claim 1,

4. the specifying means includes a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; Further comprising: the specifying means further specifies, as the target character region, a character region in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold value; 4. The image processing device according to claim 3,

5. The identification means is a processing estimation means for estimating a processing time required when the character recognition process is performed on all the character regions identified as the target character regions; a priority assigning means for assigning a priority to the character areas other than the character area specified for the character recognition process based on information on the height and width of the character area and a positional relationship between the character area and other character areas existing around the character area; having the specifying means specifies, as the target character region, a part or all of the plurality of character regions to which the priorities have been assigned, based on the priorities, within a range in which the processing time estimated by the processing estimation means falls within a predetermined processing time; 2. The image processing device according to claim 1,

6. A detection means for detecting a character region from a document image; a first specifying means for specifying, from among the plurality of character regions detected by the detecting means, a non-target character region which is a character region for which character recognition processing is not performed, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a generating means for generating a mask image in which the non-target character region identified by the identifying means is masked in the document image; a second specifying means for specifying a target character region to be subjected to character recognition processing by performing handwritten / printed character separation processing on the mask image; a character recognition unit that performs the character recognition process on the target character region identified by the second identification unit; 13. An image processing device comprising:

7. an area setting means for setting the target character area specified by the second specifying means as a recognition execution area; Further comprising: the character recognition means performs the character recognition process on the character area set in the recognition execution area; 7. The image processing device according to claim 6,

8. the first specifying means including: a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the first specifying means specifies, as the non-target character region, a character region in which the estimated character size is less than a predetermined threshold and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold; 7. The image processing device according to claim 6,

9. the first specifying means including: a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; Further comprising: the first identification means identifies, as the non-target character region, a character region in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character regions surrounding the character region and in which the estimated character size is less than the threshold value; The image processing device according to claim 8 .

10. the second specifying means generates a handwritten character image including pixels corresponding to handwritten characters and a type image including pixels corresponding to type characters by performing the handwritten / type character separation process on the mask image, specifies a handwritten character region including the handwritten characters based on the handwritten character image, specifies a type region including the type characters based on the type image, and specifies the specified handwritten character region and the specified type region as the target character region; 7. The image processing device according to claim 6,

11. the second identification means performs the handwriting / printed character separation process on the mask image to generate a handwritten character image including pixels corresponding to handwritten characters, identifies a handwritten character area including the handwritten characters based on the handwritten character image, identifies the character area, excluding the character area corresponding to the handwritten character area, from the character areas other than the non-target character area among the plurality of character areas detected by the detection means, as a printed character area including printed characters, and identifies the identified handwritten character area and printed character area as the target character area; 7. The image processing device according to claim 6,

12. a first identification means for identifying a handwritten character area including handwritten characters by performing a handwritten / printed character separation process on a document image; a generating means for generating a mask image in which the handwritten character area identified by the first identifying means is masked in the document image; a detection means for detecting a character region from the mask image; a second specifying means for specifying, from among the plurality of character regions detected by the detecting means, a character region to be subjected to character recognition processing based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition unit that performs character recognition processing on the handwritten character area identified by the first identification unit and the character area identified by the second identification unit; 13. An image processing device comprising:

13. an area setting means for setting the handwritten character area specified by the first specifying means and the character area specified by the second specifying means as a recognition execution area; Further comprising: the character recognition means performs the character recognition process on the handwritten character area and the character area set in the recognition execution area; The image processing device according to claim 12 .

14. the second specifying means including: a character size estimating means for estimating a size of a character included in the character area based on information on a height and a width of the character area; having the second specifying means specifies, as the character regions to be subjected to the character recognition process, the character regions in which the estimated character size is equal to or larger than a predetermined threshold value and the character regions in which the estimated character size is less than the threshold value and in which there are less than a predetermined number of other character regions surrounding the character regions and in which the estimated character size is less than the threshold value; The image processing device according to claim 12 .

15. the second specifying means including: a character number estimating means for estimating the number of characters included in the character area based on information on the height and width of the character area; Further comprising: the second specifying means further specifies, as the character area to be subjected to the character recognition process, a character area in which the estimated character size is less than the threshold value and there are a predetermined number or more of other character areas surrounding the character area and in which the estimated character size is less than the threshold value; The image processing device according to claim 14 .

16. the first identifying means performs the handwriting / print separation process on the document image to generate a handwritten character image including pixels corresponding to the handwritten characters, and identifies the handwritten character area based on the handwritten character image; The image processing device according to claim 12 .

17. a detection step of detecting a character region from the document image; a specifying step of specifying a target character region, which is a character region to be subjected to character recognition processing, from among the plurality of character regions detected by the detection step, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition step of performing the character recognition process on the target character region identified by the identification step; 13. An image processing method comprising:

18. a detection step of detecting a character region from the document image; a first specification step of specifying, from among the plurality of character regions detected by the detection step, a non-target character region, which is a character region for which character recognition processing is not performed, based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a generating step of generating a mask image in which the non-target character region identified by the identifying step is masked in the document image; a second identification step of performing handwritten / printed character separation processing on the mask image to identify a target character region to be subjected to character recognition processing; a character recognition step of performing the character recognition process on the target character region identified by the second identification step; 13. An image processing method comprising:

19. a first identification step of identifying a handwritten character area including handwritten characters by performing a handwritten / printed character separation process on the document image; a generating step of generating a mask image in which the handwritten character region identified by the first identifying step is masked in the document image; a detection step of detecting a character region from the mask image; a second specifying step of specifying, from among the plurality of character regions detected by the detection step, a character region to be subjected to character recognition processing based on information on the height and width of the character region and a positional relationship between the character region and other character regions present around the character region; a character recognition step of performing character recognition processing on the handwritten character area identified in the first identification step and the character area identified in the second identification step; 13. An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 16.

Images