Image processing device, image processing method and program

Abstract

To provide an image processing apparatus.SOLUTION: An image processing apparatus 160 includes: a determination unit 167 which determines a type of a document based on a determination result of a character area and a non-character area in an input image; and a selection unit 166 which selects a model to be used preferentially from among a plurality of models 182, 184 based on the determined type of the document. The image processing apparatus 160 also includes: a reduced image generation unit 172 which reduces the input image, to generate a reduced image; and a cutout unit 174 which cuts out a part of the input image as a partial image. The image processing apparatus 160 further includes an output unit 178 which outputs a top-bottom determination result of the input image on the basis of the model selected by the selection unit 166 and one of the reduced image and the partial image corresponding to the model.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to image processing, and more particularly to an image processing device, an image processing method, and a program. [Background technology]

[0002] When scanning an original with a scanner, the original may be read upside down or in a different orientation from the original (for example, landscape orientation instead of portrait orientation). Conventionally, there has been known a technology called upside-down identification, upside-down correction, or original orientation detection (hereinafter referred to as upside-down identification) that automatically identifies the upside-down of an original scanned in such a state and corrects it to the correct upside-down orientation. Conventional technology that performs upside-down identification using optical character recognition (OCR) technology is known.

[0003] For example, Japanese Patent Application Laid-Open No. 2011-008549 (Patent Document 1) is known in relation to up-down identification technology. Patent Document 1 discloses a technology aimed at improving the speed of character recognition processing. The conventional technology of Patent Document 1 discloses an image processing device having an up-down determination unit that refers to Japanese dictionary data and English dictionary data, matches each character pattern shown in image data in each of four directions, up, down, left, and right, with a character pattern in the dictionary data, and determines the up-down direction of the image data based on the matching result, a character recognition unit that performs character recognition processing on the image data using the dictionary data, and a dictionary selection unit that selects dictionary data of a language according to the matching result as dictionary data to be used by the character recognition unit.

[0004] Although the above-mentioned conventional technology can perform top-down identification with high accuracy for documents that include characters, due to the nature of the technology to recognize characters, the accuracy is reduced for documents that do not include many characters, such as documents like photographs, and is therefore insufficient. Furthermore, there is room for improvement in terms of performing comprehensive and appropriate top-down identification for scanned documents that may include both characters and non-characters. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure has been made in consideration of the above points, and has an object to provide an image processing device capable of appropriately identifying the orientation of a document that may include characters and non-text images. [Means for solving the problem]

[0006] According to the present disclosure, in order to solve the above problems, an image processing device having the following characteristics is provided. The image processing device includes a determination unit that determines the type of document based on a determination result of character regions and non-character regions in an input image, and a selection unit that selects at least a model to be used preferentially from a plurality of models based on the determined type of document. The image processing device also includes a reduced image generation unit that generates a reduced image by reducing the input image, and a cut-out unit that cuts out a part of the input image as a partial image. The image processing device further includes an output unit that outputs an upside-down identification result of the input image based on the model selected by the selection unit and an image corresponding to the model from among the reduced image and the partial image. Effect of the Invention

[0007] With the above configuration, it is possible to appropriately identify the orientation of a document that may include characters and non-text images. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an embodiment of the hardware configuration of a multifunction peripheral as an image processing apparatus according to the present embodiment. [Diagram 2] FIG. 2 is a diagram showing an embodiment of a hardware configuration of a computer that can be used as an image processing device according to another embodiment. [Diagram 3] FIG. 3 is a block diagram showing a schematic configuration of an image forming apparatus including an image processing apparatus according to the present embodiment. [Figure 4] FIG. 4 is a functional block diagram showing details of an upside-down identification unit in an image forming apparatus including an image processing apparatus according to this embodiment. [Diagram 5] FIG. 5 is a flowchart showing the upside-down identification process executed by the image processing device according to this embodiment. [Figure 6] FIG. 6 is a diagram for explaining a determination as to whether the ratio of character pixels in image data is equal to or greater than a threshold value and a determination as to whether a non-character region in image data is flat in this embodiment. [Figure 7] FIG. 7 is a diagram for explaining the process of extracting a character region from image data and the process of reducing image data in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described, but the embodiment of the present invention is not limited to the embodiment described below. Note that in the embodiment described below, an image forming apparatus having an image reading function and an image forming function will be used as an example of an image processing apparatus.

[0010] 1 is a hardware configuration diagram of an MFP as an image forming apparatus including an image processing apparatus according to the present embodiment. As shown in FIG. 1, an MFP (Multifunction Peripheral / Product / Printer) 9 includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network I / F 950.

[0011] Of these, the controller 910 has a CPU (Central Processing Unit) 901 which is the main part of the computer, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907 which is a storage unit, an HDD (Hard Disc Drive) controller 908, and an HD (Hard Disk) 909 which is also a storage unit, and is configured such that the NB 903 and the ASIC 906 are connected by an AGP (Accelerated Graphics Port) bus 921.

[0012] Of these, the CPU 901 is a control unit that performs overall control of the MFP 9. The NB 903 is a bridge for connecting the CPU 901 with the MEM-P 902, the SB 904, and the AGP bus 921, and includes a memory controller that controls reading and writing to the MEM-P 902, a PCI (Peripheral Component Interconnect) master, and an AGP target.

[0013] The MEM-P 902 is composed of a ROM 902a which is a memory for storing programs and data for implementing the various functions of the controller 910, and a RAM 902b which is used for developing the programs and data, and as a drawing memory during memory printing, etc. The programs stored in the RAM 902b may be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disc - Read Only Memory), a CD-R (Compact Disc Recordable), or a DVD (Digital Versatile Disc).

[0014] The SB904 is a bridge for connecting the NB903 to a PCI device and a peripheral device. The ASIC906 is an IC (Integrated Circuit) for image processing purposes having hardware elements for image processing, and plays the role of a bridge connecting the AGP bus 921, the PCI bus 922, the HDD 908, and the MEM-C 907. The ASIC906 is composed of a PCI target and an AGP master, an arbiter (ARB) that is the core of the ASIC906, a memory controller that controls the MEM-C 907, a plurality of DMACs (Direct Memory Access Controllers) that rotate image data using hardware logic, and a PCI unit that transfers data between the scanner unit 931 and the printer unit 932 via the PCI bus 922. The ASIC906 may be connected to a Universal Serial Bus (USB) interface or an Institute of Electrical and Electronics Engineers 1394 (IEEE1394) interface.

[0015] The MEM-C907 is a local memory used as an image buffer for copying and a code buffer. The HD909 is a storage for storing image data, font data used during printing, and forms. The HD909 controls the reading and writing of data from and to the HD909 under the control of the CPU901. The AGP bus 921 is a bus interface for a graphics accelerator card proposed to speed up graphic processing, and by directly accessing the MEM-P902 at high throughput, the graphics accelerator card can be made faster.

[0016] Further, the short-range communication circuit 920 includes an antenna 920a. The short-range communication circuit 920 is a communication circuit such as NFC or Bluetooth (registered trademark).

[0017] Further, the engine control unit 930 is composed of a scanner unit 931 and a printer unit 932. Moreover, the operation panel 940 includes a panel display unit 940a such as a touch panel that displays current setting values ​​and a selection screen, etc., and receives input from an operator, and an operation panel 940b including a numeric keypad that receives setting values ​​for image formation conditions such as density setting conditions, and a start key that receives a copy start instruction. The controller 910 controls the entire MFP 9, and controls, for example, drawing, communication, and input from the operation panel 940. The scanner unit 931 or the printer unit 932 includes an image processing unit such as error diffusion and gamma conversion.

[0018] The MFP 9 can sequentially switch and select the document box function, copy function, printer function, and facsimile function using an application switching key on the operation panel 940. When the document box function is selected, the MFP 9 enters document box mode, when the copy function is selected, the MFP 9 enters copy mode, when the printer function is selected, the MFP 9 enters printer mode, and when the facsimile mode is selected, the MFP 9 enters facsimile mode.

[0019] The network I / F 950 is an interface for performing data communication using a communication network. The short-range communication circuit 920 and the network I / F 950 are electrically connected to the ASIC 906 via a PCI bus 922.

[0020] Fig. 2 is a hardware configuration diagram of a personal computer (PC) as an image processing device according to another embodiment. Fig. 2 is a hardware configuration diagram of a PC (server). Here, the hardware configuration of the server 5 will be described.

[0021] As shown in FIG. 2, the server 5 is constructed by a computer, and as shown in FIG. 2, includes a CPU 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HD 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0022] Among these, the CPU 501 controls the operation of the entire server 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 2.

[0023] Moreover, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0024] 2 can be configured as, for example, a cloud service, such as a device that transmits the upside-down identification result of a scanned image received from the MPF or the result of image correction based on the upside-down identification result to the MFP. Alternatively, with a configuration similar to that of the server 5, it can be used as a device on a desktop that obtains the upside-down identification result of a stored scanned image or the result of image correction based on the upside-down identification result.

[0025] Fig. 3 is a block diagram showing a schematic configuration of an image forming apparatus including an image processing apparatus according to this embodiment. The image forming apparatus 100 shown in Fig. 3 includes an image reading unit 110, a first image processing apparatus 120, an image forming unit 140, a storage unit 150, and a second image processing apparatus 160. The block diagram shown in Fig. 3 corresponds to an embodiment applied to an image forming apparatus having an image reading function and an image forming function as shown in Fig. 1.

[0026] The image reading unit 110 is a device that reads image data from the original 102. The image reading unit 110 corresponds to the scanner unit 931 shown in FIG. 1. The image reading unit 110 sends the read image data to the first image processing device 120. The image forming unit 140 transfers the image data output by the first image processing device 120 to a transfer member, and outputs the copy 104. The image forming unit 140 is a transfer printing unit, and corresponds to the printer unit 932 shown in FIG. 1. Below, a process flow for obtaining a copy image will be described with reference to FIG. 3.

[0027] The first image processing device 120 includes a gamma correction unit 122, a region detection unit 124, a printer correction unit 126, a color processing / UCR unit 128, and a data interface unit .

[0028] The gamma correction unit 122 performs one-dimensional conversion on each signal of the data read by the image reading unit 110 (for example, 8 bits for each RGB color after A / D conversion) in order to align the gradation balance for each color. For the sake of explanation, it is assumed that the converted signal is a density linear signal (RGB signal: the signal value indicating white is 0). The output of the gamma correction unit 122 is sent to the area detection unit 124 and the data interface unit 130.

[0029] The area detection unit 124 judges whether a pixel or pixel block of interest in the read image is a character area or a non-character area (i.e., a pixel or area corresponding to a photograph or a picture). The area detection unit 124 can also judge whether the color is chromatic or achromatic. Depending on the result of the color judgment, the color processing / UCR unit 128 performs color reproduction processing suitable for the document.

[0030] The data interface unit 130 is a storage device management interface for temporarily storing the determination result from the area detection unit 124 and the image data after gamma correction in the storage unit 150 .

[0031] Meanwhile, the image data after gamma correction and the detection result from the area detection unit 124 are sent from the data interface unit 130 to the color processing / UCR unit 128. The color processing / UCR unit 128 selects color processing or UCR processing based on the judgment result for each pixel or pixel block. The printer correction unit 126 receives the image signals of C (cyan), M (magenta), Y (yellow), and Bk (black) from the color processing / UCR unit 128, performs gamma correction processing and dither processing taking into account the printer characteristics, and sends the signals to the image forming unit 140.

[0032] The second image processing device 160 receives the gamma-corrected image data temporarily stored in the memory unit 150, performs upside-down identification on the image data, and uses the results of the upside-down identification to convert the image data into a specified file format such as PDF (Portable Document Format) F or Office Open XML Document format.

[0033] More specifically, the second image processing device 160 includes a character determination unit 162, an upside-down identification unit 164, and a file format conversion unit 190.

[0034] The character determination unit 162 applies character determination processing to the image data after gamma correction. The character determination unit 162 performs character determination processing similar to that performed by the area detection unit 124 of the first image processing device 120, determines character areas and non-character areas in the input image, and generates a determination result. More specifically, the character determination unit 162 determines whether each of a pixel of interest or a pixel block of interest in the read image is a character area or a non-character area. The image data after gamma correction and the result of applying the character determination processing (hereinafter referred to as the character determination processing result) are output to the top / bottom identification unit 164.

[0035] Top-and-bottom identification unit 164 identifies the document type from characteristics such as the ratio of character pixels and non-character areas based on the character determination processing result output from character determination unit 162, and applies preprocessing associated with the document type to the gamma-corrected image data. Top-and-bottom identification unit 164 then inputs the preprocessed image data to an inference model also associated with the identified document type, and performs top-and-bottom identification of the document. The top-and-bottom identification result obtained by top-and-bottom identification unit 164 is output to file format conversion unit 190.

[0036] The file format conversion unit 190 converts the file into a predetermined file format such as PDF or OOXML format using the top-bottom identification result output from the top-bottom identification unit 164. For example, the orientation of a series of documents can be aligned by rotating each page to the right or left according to the orientation of the document based on the top-bottom identification result.

[0037] In the embodiment described, the character determination process used in the top / bottom identification by the top / bottom identification unit 164 has been described as being calculated by the character determination unit 162 of the second image processing device 160. However, as described above, the area detection unit 124 on the first image processing device 120 side may also have a function for performing character determination. In this case, the second image processing device 160 may be configured not to have the character determination unit 162, but to obtain the result of the character determination process by the area detection unit 124 from the first image processing device 120.

[0038] In the embodiment shown in FIG. 3, the functional units are described as being provided separately in the first image processing device 120 and the second image processing device 160, but these may be configured together on a single device, or the functional units provided in the first image processing device 120 and the functional units provided in the second image processing device 160 may be further distributed into multiple devices, and this is not particularly limited.

[0039] 4 is a more detailed functional block diagram of the upside-down identification unit 164 in the image forming apparatus including the image processing device according to this embodiment. As shown in FIG. 4, the upside-down identification unit 164 includes an inference model selection unit 166, an upside-down identification preprocessing unit 168, and an upside-down identification inference unit 178.

[0040] Image data from storage unit 150 shown in FIG. 3 and the character determination process result for the image data from character determination unit 162 are input to top / bottom identification unit 164. Inference model selection unit 166 includes determination unit 167 that determines the type of document based on the character determination process results for character regions and non-character regions in the input image data. More specifically, determination unit 167 determines the type of document (document type such as photo document or document document) from the ratio (proportion) of the character region in the entire image and the image feature amount in the non-character region based on the character determination process result for the image data from character determination unit 162. Inference model selection unit 166 selects an inference model to be used for inference when identifying the top / bottom, which is previously associated with the determined type of document.

[0041] The number of types of document types and the number of types of inference models in this case are not particularly limited and can be any number equal to or greater than 2. The following description will be continued assuming that two inference models are prepared. Image data from storage unit 150, information identifying the inference model selected by inference model selection unit 166 (hereinafter referred to as inference model information), and the result of character determination processing on the image data by character determination unit 162 shown in FIG. 3 are input to upside-down identification preprocessing unit 168.

[0042] The top-bottom identification preprocessing unit 168 applies preprocessing based on the inference model information selected by the inference model selection unit 166. As an example of preprocessing, for the first inference model to be applied to documents with few characters such as photographs, reduction processing is applied to the entire document for the image data of each document from the storage unit 150. The reduced image of the entire document resulting from the reduction processing is the preprocessed image data when the first inference model is selected. The reduction processing can use any reduction algorithm such as nearest neighbor interpolation or bilinear interpolation, and is not limited to these. On the other hand, for the second inference model to be applied to documents with many characters (especially small characters) such as office documents and papers, a character area is calculated for the image data from the storage unit 150 based on the result of character determination processing on the image data by the character determination unit 162, and a rectangular area including characters is cut out. A partial image of the character area of ​​the document resulting from this cutting out is the preprocessed image data when the second inference model is selected.

[0043] The inference model information selected by the inference model selection unit 166 and the result of preprocessing applied based on the inference model information by the upside-down identification preprocessing unit 168 (preprocessed image data) are input to the upside-down identification inference unit 178.

[0044] The upside-down identification inference unit 178 switches to an inference model and inputs the result of preprocessing (preprocessed image data) output from the upside-down identification preprocessing unit 168, referring to the inference model information selected by the inference model selection unit 166, performs inference for upside-down identification, and outputs the upside-down identification result. The upside-down identification result indicates either north / east / west / south. The upside-down identification result output from the upside-down identification inference unit 178 is input to a file format conversion unit 190, which then executes processing according to the upside-down identification result.

[0045] 4 further illustrates a more detailed functional configuration of the top-bottom identification preprocessing unit 168. As shown in FIG. 4, the top-bottom identification preprocessing unit 168 includes a preprocessing switching unit 170, a reduced image generating unit 172, a character region extracting unit 174, and a preprocessing combining unit 176.

[0046] The image data input from the memory unit 150, the inference model information selected by the inference model selection unit 166, and the result of the character determination process on the image data by the character determination unit 162 are input to the upside-down identification preprocessing unit 168.

[0047] The pre-processing switching unit 170 receives as inputs the image data from the storage unit 150, the inference model information selected by the inference model selection unit 166, and the character determination processing result for the image data from the character determination unit 162. The pre-processing switching unit 170 refers to the inference model information, which is one of the inputs, and switches the pre-processing to be applied for each input unit image, for example, for each sheet of a document.

[0048] Specifically, when the first inference model is selected by the inference model selection unit 166 in Fig. 4, the preprocessing is a reduction process for the entire document. The image data input from the storage unit 150 and the inference model information selected by the inference model selection unit 166 are input to the reduced image generation unit 172. On the other hand, when the second inference model is selected by the inference model selection unit 166, the preprocessing is a process of calculating a character area and cutting out a rectangular area including characters. The image data input from the storage unit 150, the inference model information selected by the inference model selection unit 166, and the character determination process result for the image data by the character determination unit 162 are input to the character area cutout unit 174.

[0049] In a specific embodiment, the selection of whether to use the reduced image generating unit 172 or the character region extracting unit 174 in the top-bottom identification preprocessing unit 168 is made on an image data unit basis for each document, and this selection is exclusive within one document (one page). In other words, in this specific embodiment, both the reduced image generating unit 172 and the character region extracting unit 174 are not selected at the same time.

[0050] The reduced image generating unit 172 applies reduction processing to the image data from the storage unit 150. The reduction processing is not limited to a reduction algorithm, such as nearest neighbor interpolation or bilinear interpolation. In addition, when reducing an image, whether the aspect ratio of the image before and after reduction is maintained (same size) or ignored (variable magnification) is also not limited. The image data reduced by the reduced image generating unit 172 (preprocessed image data) and the inference model information selected by the inference model selecting unit 166 are input to the preprocessing combining unit 176.

[0051] The character area extraction unit 174 selects a character area (group of character pixels) at an arbitrary position based on the result of the character determination process on the image data by the character determination unit 162, and extracts (crops) an area of ​​the image data corresponding to the selected character area. The image data (preprocessed image data) extracted by the character area extraction unit 174 and the inference model information selected by the inference model selection unit 166 are input to the preprocessing combination unit 176.

[0052] It should be noted that reduced image generating unit 172 has been described as generating a reduced image by reducing the entire input image, and character area cutting unit 174 has been described as cutting out a part of the image without reduction. Although such a configuration is preferable, it is not necessarily limited to such a configuration. Insofar as reduced image generating unit 172 reduces an image of an area larger than the part cut out by character area cutting unit 174, it includes generating an image of a part of the input image rather than the entire image, and can be said to perform preprocessing with priority given to reduction. On the other hand, insofar as reduced image generating unit 172 cuts out an area smaller than the original image to be reduced, it does not exclude reduction, and can be said to perform preprocessing with priority given to cutting out a part of the image.

[0053] The preprocessing combination unit 176 assigns the inference model information selected by the inference model selection unit 166 to the preprocessed image data (reduced image data generated by the reduced image generation unit 172 or partial image data cut out by the character area cutout unit 174). The preprocessing combination unit 176 acts as an interface connecting the upside-down identification preprocessing unit 168 and the upside-down identification inference unit 178 in FIG.

[0054] 4 further illustrates a more detailed functional configuration of the top-bottom identification inference unit 178. As illustrated in FIG. 4, the top-bottom identification inference unit 178 includes an inference model switching unit 180, a first inference model 182, a second inference model 184, and a top-bottom identification combination unit 186.

[0055] The preprocessed image data (reduced image data generated by the reduced image generation unit 172 or partial image data cut out by the character area cutout unit 174) and the inference model information selected by the inference model selection unit 166 are input to the top-bottom identification inference unit 178.

[0056] The inference model switching unit 180 receives the preprocessed image data and the inference model information as input. The inference model switching unit 180 switches the inference model for inputting the preprocessed image data for each document by referring to the inference model information.

[0057] The first inference model 182 is an inference model to be applied to documents with few characters, such as photographs. In a specific embodiment, the first inference model 182 is an inference model trained on the assumption that an image obtained by reducing the entire scanned image is input. The training data for the first inference model 182 includes images with few characters, such as photographs, and ground truth labels indicating the top-bottom orientation of the images, and can be trained by an appropriate machine learning technique, such as supervised learning.

[0058] The second inference model 184 is an inference model that is applied to documents with many characters (especially small characters) such as office documents and papers. The second inference model 184 is an inference model that is trained on the premise that an image in which an area containing many characters in a scanned image is cut out at a specific size is input. The training data for the second inference model 184 includes images containing many characters and ground truth labels indicating the top-bottom orientation of the images, and can be trained by appropriate machine learning techniques such as.

[0059] The first inference model 182 and the second inference model 184 may be machine learning models as so-called classifiers. In a preferred embodiment, the first inference model 182 and the second inference model 184 may each be a neural network model trained by deep learning. Note that the first inference model 182 and the second inference model 184 are not limited to models based on deep learning. For example, the second inference model may be an existing technology that identifies the top and bottom of a document based on OCR.

[0060] The top-bottom identification combination unit 186 integrates the results of the top-bottom identification inference by the first inference model 182 and the second inference model 184, and inputs the results to the file format conversion unit 190. In other words, the top-bottom identification combination unit 186 serves as an interface that connects the top-bottom identification inference unit 178 and the file format conversion unit 190.

[0061] In the above description, the selection of the reduced image generating unit 172 and the character region extracting unit 174 in the top-bottom identification preprocessing unit 168 and the selection of the inference model to be used are described as being exclusive. However, in other embodiments, the selection of the inference model and the preprocessing may not be exclusive. For example, the preprocessing and inference of multiple inference models may be performed in parallel, the inference results may be obtained as primary results, and the final result may be determined based on the type of the identified manuscript and the multiple primary inference results. Specifically, the top-bottom identification may be performed using multiple inference models and multiple corresponding preprocessing, and the obtained top-bottom identification primary result may be weighted and evaluated according to the manuscript type. For example, since the classification result by a classifier usually has a degree of certainty that can be calculated, the reliability of the results obtained from multiple inference models may be weighted and evaluated according to the manuscript type, and the final top-bottom identification result may be obtained by adopting the top-bottom identification primary result with the highest reliability after weighting.

[0062] That is, in one or more embodiments, the inference model selection unit 166 selects at least a preferentially used inference model from among the multiple inference models based on the determined type of manuscript. In a specific embodiment, the inference model selection unit 166 selects an inference model to be used exclusively from among the multiple models based on the determined type of manuscript. In another specific embodiment, the inference model selection unit 166 selects an inference model to be used with the greatest weight from among the multiple models based on the determined type of manuscript.

[0063] Hereinafter, the top-bottom identification process according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the top-bottom identification process executed by the image processing device according to this embodiment. Note that the process shown in Fig. 5 is executed by a processor that implements the character determination unit 162 and the top-bottom identification unit 164 for each piece of image data of a document.

[0064] 5 starts at step S100 in response to image data being input from storage unit 150 to second image processing device 160. In step S101, the processor performs character determination processing on the image data using character determination unit 162. The character determination processing determines whether each pixel or pixel block is a character area or a non-character area, and a character determination processing result is obtained.

[0065] In step 102, the processor determines whether the ratio (proportion) of character pixels in the image data is equal to or greater than a preset threshold based on the result of the character determination process for the image data by the inference model selection unit 166. If it is determined in step 102 that the ratio of character pixels in the image data is less than the preset threshold (NO), the process branches to step S103.

[0066] In step S103, the processor further calculates image features of non-character regions in the image data using the inference model selection unit 166. In one or more embodiments, the calculated image features may be, but are not limited to, statistical features such as the variance of pixel values ​​in non-character regions of the image data. Here, the variance can be calculated by the difference between the squared mean of pixel values ​​in a predetermined range (here, all non-character regions) and the square of the mean, and the variance is an index value of the degree of pixel value dispersion. In step S104, the processor determines whether the non-character regions of the image data are flat using the inference model selection unit 166 based on the image features calculated in step S103. If the variance is smaller than a predetermined threshold, it is determined to be flat.

[0067] If it is determined in step S104 that the surface is flat (YES), the process branches to step S105. Referring back to step S102, if it is determined in step S102 that the ratio of character pixels in the image data is equal to or greater than a preset threshold (YES), the process also branches to step S105. On the other hand, if it is determined in step S104 that the non-character region of the image data is not flat (NO), the process branches to step S107. The branching method is summarized in Table 1 below. Note that in Table 1 below, if the determination in step S102 is YES, the document is further divided into different types of documents (flat-text document and text-photo document) depending on whether the determination in step S104 is YES or NO. However, regardless of the determination in step S104, from the viewpoint of determining the pre-processing and inference model to be used, these may be the same type of document, and this does not prevent them from being further classified into multiple document types linked to the same pre-processing and inference model.

[0068] [Table 1]

[0069] In step S105, the processor determines to use the second inference model 184 by the inference model selection unit 166, and adds inference model information to that effect to the image data. In step S106, the processor causes the character area extraction unit 174 to select a character area (group of character pixels) at an arbitrary position based on the result of the character determination process of the image data, based on the inference model information indicating the second inference model 184, and extracts (crops) the image data corresponding to the selected character area to generate partial image data. The size of the crop is the image size to be input to the second inference model 184.

[0070] In response to this, in step S107, the processor uses the inference model selection unit 166 to determine to use the first inference model 182, and adds inference model information to that effect to the image data. In step S108, the processor reduces the image data using the reduced image generation unit 172 based on the inference model information indicating the first inference model 182, and generates reduced image data of the entire image. For the reduction and magnification processing at this time, any reduction algorithm such as nearest neighbor interpolation or bilinear interpolation can be adopted. In addition, when reducing an image, there is no limitation as to whether the aspect ratio of the image is maintained before and after reduction, or ignored. The size after reduction becomes the image size to be input to the first inference model 182.

[0071] When the process of step S105 or S108 is completed, the process proceeds to step S109. In step S109, the processor causes the preprocessing combining unit 176 to assign information of an inference model to be used in the subsequent process of S110 to the character region of the partial image data cut out in the process of step S106 or the reduced image data generated in the process of S108 (referred to as preprocessed image data). More specifically, the processor assigns information of a first inference model 182 to the reduced image data generated in the process of S108, and assigns information of a second inference model 184 to the cut-out image generated in the process of S105, as models to be used during inference. When the process of S109 is completed, the process proceeds to S110.

[0072] In step S110, the processor performs inference by inputting the preprocessed image data to the corresponding inference model based on the model information used during inference given in step S109 by the top-bottom identification inference unit 178. Specifically, the reduced image generated in the process of S108 is input to the first inference model 182, and the partial image data of the character region cut out in the process of S106 is input to the second inference model 184. The inference result (top-bottom identification result) is either north / east / west / south. Note that in the embodiment described, the top-bottom direction is classified into one of four directions, and four directions is sufficient for top-bottom identification, but the number of classifications is not particularly limited.

[0073] When the process of step S110 ends, the processes of the character determination unit 162 and the top / bottom identification unit 164 end.

[0074] 6 is a diagram for explaining the determination in step S102 of whether the ratio of character pixels in the image data is equal to or greater than a threshold, and the determination in step S104 of whether the non-character region of the image data is flat. Note that, for the sake of convenience, a black border is added to the image in FIG. 6, but the target image does not necessarily have to have a black border.

[0075] First, a description will be given of an original with few characters, such as a photograph, to which the first inference model is applied. Here, it is assumed that an input image as shown in FIG. 6(A) is given. The result of performing character determination processing on the image data shown in FIG. 6(A) is a binary image as shown in FIG. 6(B). In FIG. 6(B), black pixels represent pixels belonging to a character region, and white pixels represent pixels belonging to a non-character region. For example, when processing by the second image processing device 160 is performed on an input image in which a photograph (or picture) such as that shown in FIG. 6(A) occupies the majority of the entire image, the character determination processing result of the binary image as shown in FIG. 6(B) is output in the character determination unit 162 (processing of S101).

[0076] When FIG. 6(B) is input to the top / bottom discrimination unit 164, the inference model selection unit 166 performs processing (S102 to S104). In step S102, the inference model selection unit 166 counts character pixels for the character determination processing result as shown in FIG. 6(B) and judges whether the ratio of the counted character pixel count to the number of pixels in the entire image is equal to or greater than a preset threshold. For convenience of explanation, the threshold is set to 50%. In the case of an input image as shown in FIG. 6(A), the photograph occupies the majority of the image, so the character determination processing result as shown in FIG. 6(B) has a small ratio of character pixels to the entire image, and the number of character pixels is also small (for explanation, 500 character pixels are used out of 10,000 pixels in the entire image). In the case of FIG. 6(A), the number of character pixels (500 pixels out of 10,000 pixels, so 5%) is less than the threshold (50%), so the processing in S102 results in a judgment result of NO.

[0077] If the process of step S102 gives a NO result, the processes of steps S103 and S104 are performed. In steps S103 and S104 of the inference model selection unit 166, it is determined whether or not the non-character region is flat for the character determination process result as shown in FIG. 6B. The flatness of the non-character region can be determined from the variance calculated from the non-character region of the input image shown in FIG. 6A, which corresponds to the non-character pixels of the binary image shown in FIG. 6B. In the case of the input image as shown in FIG. 6A, the region corresponding to the non-character pixels of FIG. 6B is a photographic part, so the non-character region is not flat, and the process of step S104 gives a NO result. As described above, if the process of step S102 gives a NO result and the process of step S104 gives a NO result, the first inference model is applied.

[0078] Next, we will explain about documents with many characters (especially small-sized characters), such as office documents and papers, to which the second inference model is applied. Here, it is assumed that an input image as shown in FIG. 6(C) is given. The result of performing character determination processing on the image data shown in FIG. 6(C) is a binary image as shown in FIG. 6(D). In the case of image data that includes a photograph (or picture) as part of the document as shown in FIG. 6(C), a binary image is obtained in which the parts of the image data that correspond to the document are black, as shown in FIG. 6(D).

[0079] For example, when processing an input image in which tiny characters such as those shown in FIG. 6(C) make up the majority of the image, the second image processing device 160 outputs a character determination processing result (binary image) such as that shown in FIG. 6(D) in the character determination unit 162 (processing of S101).

[0080] When the image shown in FIG. 6(D) is input to the top / bottom discrimination unit 164, the inference model selection unit 166 (processing of S102 to S104) is executed. In the process of S102 of the inference model selection unit 166, character pixels are counted for the character determination process result (binary image) as shown in FIG. 6(D), and it is determined whether the ratio of the counted character pixel count to the number of pixels in the entire image is equal to or greater than a preset threshold. For the sake of explanation, this threshold is set to 50%. In the input image shown in FIG. 6(C), minute characters occupy the majority of the entire image, so the character determination process result shown in FIG. 6(D) has a large ratio of character pixels to the entire image, and the number of character pixels is also large (for the sake of explanation, it is assumed that the number of character pixels is 7000 out of 10000 in the entire image).

[0081] In the case of the input image shown in FIG. 6(C), the number of character pixels (7000 pixels out of 10000 pixels, so the ratio is 70%) is greater than or equal to the threshold value (50%), so the result of the determination in S102 is YES. If the result of the determination in step S102 is YES, the processes in S103 and S104 are not performed. As described above, if the result of the determination in S102 is YES, the second inference model is applied.

[0082] Fig. 7 is a diagram explaining the process of extracting character regions from image data performed in the process of S106 and the process of reducing image data performed in the process of S108. Note that, as in Fig. 6, for the sake of convenience, the image is framed in black. The processes of steps S106 and S108 both correspond to pre-processing applied before inputting the data to the first inference model and the second inference model.

[0083] First, the pre-processing applied to the first inference model 182 will be described. When it is determined that the first inference model is to be applied in the processing up to the inference model selection unit 166 (up to S104), an input image as shown in FIG. 6(A) is input to the reduced image generation unit 172 (processing of S108). As the processing of S108 of the reduced image generation unit 172, a reduction process is applied to the input image as shown in FIG. 6(A). The reduction process at this time is not limited to a reduction / magnification algorithm to be adopted, such as nearest neighbor interpolation or bilinear interpolation. In addition, when reducing an image, whether the aspect ratio of the image is maintained before and after the magnification or ignored is also not limited. When the reduction process of S108 is applied to the input image as shown in FIG. 6(A), a reduced image as shown in FIG. 7(A) is obtained. The size after reduction is the image size to be input to the inference model. The reduced image in FIG. 7A is input to the top-bottom discrimination inference unit 178 via the preprocessing combination unit 176, and inference for top-bottom discrimination is performed in the first inference model 182.

[0084] Next, a description will be given of pre-processing applied to the second inference model 184. When it is determined in the processing up to the inference model selection unit 166 (up to S104) that the second inference model 184 is to be applied, an input image as shown in Fig. 6(C) and the character determination processing result of the image data shown in Fig. 6(B) as shown in Fig. 6(D) output by the character determination unit 162 (processing of S101) are input to the character area extraction unit 174 (processing of S106).

[0085] The processing of S106 in character area extraction unit 174 detects a character area contained in a rectangular area of ​​any size for the character determination processing result of the image data shown in Fig. 6(C), as shown in Fig. 6(D). Fig. 7(D) shows a rectangular area of ​​any size containing the character area detected for Fig. 6(D) with a black frame. The processing of step S106 further includes extracting an area of ​​the input image (shown in Fig. 7(C)) corresponding to the black frame shown in Fig. 7(D). The extracted character area extracted image (Fig. 7(B)) is input to upside-down identification inference unit 178 via preprocessing combination unit 176, and inference as upside-down identification is performed in second inference model 184.

[0086] According to the above-described embodiment, it is possible to provide an image processing device, an image processing method, and a program capable of appropriately identifying the orientation of a document that may include characters and non-character images.

[0087] As mentioned above, there are techniques for identifying top and bottom that use inference models such as deep learning neural network models. Techniques for identifying top and bottom using deep learning technology have an advantage over OCR-based techniques when it comes to documents such as photographs without text. However, due to memory consumption, it is common to reduce the size of the image before inputting it into the inference model. If an image is reduced uniformly without any pre-processing (in many cases, a scanned document image with a relatively high rectangular resolution is resized and reduced to a relatively small image with a square resolution that matches the input layer of the inference model), the smaller the characters written on the document, the more character information is lost during the reduction and resizing process, making it difficult to identify the top and bottom accurately.

[0088] According to the above-mentioned technology, multiple models are selected, and multiple preprocessing processes are prepared, including a preprocessing process that prioritizes reduction of the model and a preprocessing process that prioritizes cutting out a part of the image. Therefore, it is possible to switch the model to be used preferentially depending on the type of document. In addition, it is possible to achieve the same accuracy of top-bottom identification as other types of documents (such as photographs) even for documents containing many small characters. Compared to the conventional top-bottom identification method using ORC, it is expected that the accuracy of top-bottom identification for documents such as photographs will be improved, and it is also possible to suppress the decrease in accuracy for documents containing many small characters, which has been a concern as a trade-off for reducing memory consumption due to the adoption of deep learning technology. In other words, it is possible to achieve highly accurate top-bottom identification by appropriately switching the identification process depending on the type of document.

[0089] By determining the type of document for each document based on the ratio of text areas in the image and whether or not non-text areas are flat, and selecting a model, it becomes possible to prepare optimal pre-processing and inference models for each document, rather than applying a uniform inference model to all documents, which leads to an improvement in the overall accuracy of top-bottom identification. By preparing pre-processing and inference models for each document type, it becomes possible to perform highly accurate top-bottom identification for many types of documents. In addition, by making it possible to perform appropriate pre-processing for each inference model, it becomes possible to perform top-bottom identification with a high degree of freedom, which leads to an improvement in the overall accuracy of top-bottom identification.

[0090] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0091] In addition, the above functions can be realized by a computer-executable program written in legacy programming languages ​​such as assembler, C, C++, C#, Java (registered trademark), or object-oriented programming languages, and can be stored on a device-readable recording medium such as ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray Disc, SD card, or MO, or distributed via telecommunications lines.

[0092] So far, an image processing device, an image processing system, an image processing method, and a program according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and other modifications, changes, or deletions, etc., may be made within the scope of what a person skilled in the art can conceive. In any aspect, as long as the functions and effects of the present invention are achieved, it is within the scope of the present invention. [Explanation of symbols]

[0093] 100...image forming apparatus, 102...original, 110...image reading section, 120...first image processing device, 122...gamma correction section, 124...area detection section, 126...printer correction section, 128...color processing / UCR section, 130...data interface section, 140...image forming section, 150...storage section, 160...second image processing device, 162...character determination section, 164...top-bottom identification section, 166...inference model selection section, 168...top-bottom identification preprocessing section, 170...preprocessing switching section, 172...reduced image generation section, 174...character area extraction section, 176...preprocessing combination section, 178...top-bottom identification inference section, 180...inference model switching section, 182...first inference model, 184...second inference model, 186...top-bottom identification combination section, 190...file format conversion section [Prior art documents] [Patent documents]

[0094] [Patent Document 1] JP 2011-008549 A

Claims

1. a determination unit for determining a type of document based on a determination result of a character region and a non-character region in an input image; a selection unit that selects at least a model to be used preferentially from among a plurality of models based on the determined type of the document; a reduced image generating unit for generating a reduced image by reducing the input image; a cutout unit that cuts out a part of the input image as a partial image; an output unit that outputs an upside-down discrimination result of the input image based on the model selected by the selection unit and an image corresponding to the model among the reduced image and the partial image; 23. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the determining unit determines the type of the document based on at least a ratio of a text area in the input image.

3. The image processing apparatus according to claim 2 , wherein the determination section, when the ratio of character areas in the input image satisfies a predetermined condition, further determines the type of the document based on image features of non-character areas in the input image.

4. 4. The image processing device according to claim 1, wherein each of the plurality of models is an inference model that outputs a vertical direction for an input image.

5. The plurality of models are A first inference model trained to perform upside-down identification based on an input reduced image; A second inference model trained to perform upside-down discrimination based on a partial image corresponding to an input character region; and The image processing device according to any one of claims 1 to 4, comprising:

6. 6. The image processing device according to claim 1, wherein the selection by said selection section is switched for each input unit image.

7. 7. The image processing device according to claim 1, further comprising a character determination unit that determines character regions and non-character regions in the input image and generates the determination result.

8. 1. An image processing method for identifying the top and bottom of an input image, comprising: determining a type of document based on a result of determining whether an input image is a character region or a non-character region; selecting a model to be used at least preferentially from among a plurality of models based on the determined type of the document; outputting an upside-down discrimination result for the input image based on the selected model, a reduced image obtained by reducing the input image, and a corresponding image among partial images obtained by cutting out a part of the input image; An image processing method comprising:

9. Computer, a determination unit that determines a type of document based on a determination result of a character region and a non-character region in an input image; a selection unit that selects at least a model to be used preferentially from among a plurality of models based on the determined type of the document; a reduced image generating unit for generating a reduced image by reducing the input image; a cutout unit that cuts out a part of the input image as a partial image; and an output unit that outputs an upside-down discrimination result of the input image based on the model selected by the selection unit and the image corresponding to the model among the reduced image and the partial image; A program to function as a

Images