Image processing apparatus, image processing method, and program

Abstract

To reduce processing time of pre-processing on image data.SOLUTION: The present invention provides an image processing apparatus 20 that performs magnification varying processing for each of a plurality of magnification varying reference areas of image data, and the image processing apparatus has: a binarized image creation unit 15 that binarizes the image data; a blank processing unit 16 that generates flatness or non-flatness information for every block of the image data binarized by the binarized image creation unit, and removes a blank part on the basis of the flatness or non-flatness information; and a magnification-varied image creation unit 18 that determines a magnification varying method for every magnification varying reference area according to the flatness or non-flatness information for one or more blocks corresponding to the magnification varying reference areas, and creates a magnification-varied image of the image data by using the determined magnification varying method.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] In the field of image processing, AI (Artificial Intelligence) is used for various tasks. A function for making some inferences with AI is called a model, and the images input to the model are pre-processed suitable for the images and inferences so that favorable results can be obtained efficiently. For example, one of the pre-processes is a process of reducing and scaling (resizing) an image.

[0003] Even after reducing and scaling an image, a technique for maintaining the shape of the image elements of a binary image is known (see, for example, Patent Document 1). Patent Document 1 discloses a technique of applying scaling to a binary image, outputting it as a multi-valued image, and then binarizing it again under conditions suitable for the reduced image to generate a binary image.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technique has a problem that the processing time of the pre-processing for image data increases according to the image size. For example, the conventional technique binarizes the image data again after reduction. Further, since the conventional technique does not adopt a configuration for shortening the processing time of scaling, the processing time of the pre-processing depends on the image size (number of pixels) before scaling. Such an increase in the processing time of the pre-processing becomes a bottleneck in the overall processing time including inferences.

[0005] In view of the above problems, the present invention provides a technique for reducing the processing time of pre-processing for image data.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides an image processing apparatus that performs magnification processing for each of a plurality of magnification reference regions of image data, including a binary image generation unit that binarizes the image data, a margin processing unit that creates flat or non-flat information for each block of the image data binarized by the binary image generation unit and determines a margin portion based on the flat or non-flat information, and a magnification image generation unit that determines a magnification method for each magnification reference region according to the flat or non-flat information of one or more of the blocks corresponding to the magnification reference region and generates a magnified image of the image data using the determined magnification method. The image processing apparatus is characterized by having these components.

Effect of the Invention

[0007] The present invention can provide a technique for reducing the processing time of preprocessing for image data.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, an image processing apparatus and an image processing method performed by the image processing apparatus will be described with reference to the drawings.

[0010] <Regarding Terms> Binarization refers to the process of converting a multi-tone image into two colors, white and black (0, 1). If there are two colors, 0 and 1 may be assigned to colors other than white and black.

[0011] Magnification or reduction of image data refers to increasing or decreasing the size. In this embodiment, mainly reduction will be described. Note that when magnifying or reducing, it is preferable to maintain the aspect ratio of the image, but it may not be maintained.

[0012] A block is a rectangular area into which image data is divided in order to determine whether it is a margin part. The block does not necessarily have to be rectangular. A histogram is calculated from the image within the block, and whether it is a margin is determined based on the values of the histograms of white pixels and black pixels. In this embodiment, the block will be described using the term "histogram calculation block".

[0013] Flat means that the inside of the block can be regarded as almost only white pixels or only black pixels, and it is an example of little change in pixels. Non-flat means not flat. Non-flat means that it can be regarded that white pixels and black pixels exist within the block, and it is an example of a change in pixels.

[0014] The magnification reference area is an area in which pixel values are referred to during the magnification process, and is one area of the image to be reduced.

[0015] <Regarding preprocessing> In image processing using AI, it is often the case that the image size is reduced and input to the model for reasons of memory capacity and processing time (productivity). Therefore, the image processing apparatus described later reduces (resizes) the image as preprocessing for the image.

[0016] FIG. 1 is a diagram for explaining preprocessing of an image and inference by a model. FIG. 1 shows a process of reducing the image data 111 (input image) to a reduced image 112 and the model 113 identifying the top and bottom. The preprocessing in FIG. 1 is "margin removal + reduction scaling", and the inference is the identification of the top and bottom of the manuscript. If only reduction scaling is performed, in an image with a lot of margins, a lot of margin parts will remain in the image after reduction scaling, and important parts that are not margin parts will also decrease due to reduction, which may affect the subsequent processing.

[0017] As a countermeasure to avoid this inconvenience, it is known to perform margin removal before reduction scaling. Also, the image input to the preprocessing may be a multi-valued image or a binary image, but in some cases, it may be a binary image for reasons such as memory capacity and convenience for other uses. For example, in top and bottom identification that identifies the top and bottom of an image by AI, a binary image may be used as the input for preprocessing, and margin removal and reduction scaling may be performed as the preprocessing.

[0018] In a task of recognizing from the layout of the entire image such as top and bottom identification, it is necessary to scale the image size to an image size suitable for top and bottom identification so as to retain the layout information of the entire image. The image size input to the model is often very small in most cases, and the reduction rate is often extremely large compared to the size of the original input image. As an example, there is also a process of scaling an image size of A4 300 [dpi] = 2480 × 3508 pixels to 224 × 224.

[0019] Therefore, in the reduction scaling process, it is necessary to reliably reflect the information of the image before scaling in the information after scaling. However, depending on the scaling method, some of the pixels before scaling may deviate from the reference pixels, and some of the information of the image before scaling may be missing from the image after scaling. In particular, in an image containing characters, the loss due to information loss is likely to be significant. Even if the reduced image with information loss is input to the model, the information necessary for recognition is not included in the reduced image, and there is a risk that the model will output an incorrect recognition result (incorrect judgment of the top and bottom).

[0020] FIG. 2 is a diagram for explaining an example of information loss due to image reduction scaling. FIG. 2(a) is the original image, FIG. 2(b) is the image with information loss due to reduction scaling, and FIG. 2(c) is the image without information loss (with little information loss) due to reduction scaling. As shown in FIG. 2(b), when the magnification factor is extremely large, depending on the magnification method, some of the pixels before magnification deviate from the reference pixels. In particular, images containing characters and charts are easily affected by information loss, and depending on the layout of the characters, the shape of the characters in the original image may not be referenced. That is, in the reduced image of FIG. 2(b), the model is highly likely to fail in transposition identification.

[0021] To avoid such problems, an area averaging method (also called an integration method or an average pixel method) that references all pixels before reduction scaling is known. In the area averaging method, since all pixels of the image are referenced, the inconvenience of information loss as described above can be eliminated.

[0022] FIG. 3 is a diagram for explaining the area averaging method. The area averaging method is a method of dividing the image before reduction scaling into a plurality of regions according to the size of the image after reduction scaling, and setting the average of the pixels in the region as the pixel value corresponding to the pixel in the image after reduction scaling. The region of the original image is called a "magnification reference region". In FIG. 3, for convenience of explanation, the size of the image after reduction scaling is set to an extremely small size (3×3), so the original image is divided into 3×3 reduction scaling regions 115. The average of the pixel values in each reduction scaling region 115 becomes the pixel value in each region 116 after reduction scaling.

[0023] In the area averaging method, since all pixels of the original image are referenced, information loss is less likely to occur, but the processing time depends on the size (number of pixels) of the original image. The larger the size of the original image, the longer the processing time. This becomes a major bottleneck in the processing time of the entire inference.

[0024] The area averaging method has greater advantages in terms of image quality as the reduction magnification factor is smaller (i.e., the size of the image after magnification is larger). However, depending on the magnification reference area, it is not always necessary to calculate the pixel values by referring to all the pixels within the area. For example, when determining the magnification reference area according to the size of the image after reduction magnification as shown in FIG. 4, depending on the sizes of the original image and the image after magnification, the magnification reference area may consist only of a flat area such as the background.

[0025] FIG. 4 shows the image data set in a 3×3 magnification reference area. Note that FIG. 4 shows the image after performing blank area removal before reduction magnification. The reason why the outer edge of the magnification reference area does not coincide with the outer edge of the entire FIG. 4 is that, as a result of blank area removal, magnification is to be performed on the area (3×3 magnification reference area) obtained by removing the blank areas of the entire image.

[0026] The image in FIG. 4 has characters only at the top and bottom. Therefore, for the magnification reference areas R1 to R3 and R7 to R9, it is necessary to refer to (calculate the average of) all the pixels within the magnification reference areas R1 to R3 and R7 to R9 in order to prevent loss of information. On the other hand, since the magnification reference areas R4 to R6 are flat areas, referring to (calculating the average of) all the pixels within the magnification reference areas R4 to R6 does not make much sense. In a flat area such as the magnification reference areas R4 to R6, referring to all the pixels within the magnification reference area is excessive and may only increase the processing time unnecessarily.

[0027] <Outline of the preprocessing of the present embodiment> Therefore, in the present embodiment, the image processing apparatus realizes a "reduction magnification method with less loss of image information while considering the processing time" by using the analysis information calculated for removing the blank areas of the image during subsequent reduction magnification.

[0028] FIG. 5 is a diagram schematically explaining the outline of the reduction magnification method in the present embodiment. (1) The image processing apparatus performs blank area removal by binarizing the original image. (2) The image processing apparatus divides the image after blank area removal into histogram calculation blocks (the area surrounded by the dotted line), and creates a histogram for each histogram calculation block. (3) The image processing apparatus classifies the histogram calculation blocks into either "white", "black", or "black and white" based on the histograms of the number of "white" and "black" pixels. (4) The image processing apparatus changes the reduction method for each magnification reference area according to which histogram calculation block of "white", "black and white", or "black" the magnification reference areas R1 to R9 correspond to, and calculates the pixel values after magnification.

[0029] For example, when the magnification reference area includes a histogram calculation block that is only "white", the maximum frequency of 255 (when the density of one pixel is represented by 8 bits) is set as the pixel value after reduction magnification. (5) When the magnification reference area includes a histogram calculation block of "black and white" (non-flat case), the image processing apparatus calculates the pixel values of the magnification reference area by the area averaging method.

[0030] In this way, the preprocessing method of this embodiment can utilize the histogram calculated by blank area removal during subsequent reduction magnification. By determining whether the histogram calculation block is flat or not, and calculating the pixel values by the area averaging method only when it is not flat, it is possible to shorten the processing time of reduction magnification while retaining the information of the image before reduction magnification. That is, the blank area removal performed to remove the blank area of the image divides the image into blocks and determines whether each block is a blank area or not. By using the blank area information of each block calculated for blank area removal during subsequent reduction magnification, it is possible to shorten the processing time of reduction magnification while retaining the information of the original image. Therefore, it is possible to prevent the preprocessing from becoming a bottleneck in the overall processing time including inference.

[0031] <Configuration Example> FIG. 6 shows a configuration diagram of an example of an apparatus or system that performs preprocessing. The image processing apparatus 20 in FIG. 6(a) is an apparatus that combines a plurality of different functions, such as a multifunction printer or an MFP (Multifunction Peripheral), used by a user. The image processing apparatus 20 has at least a scanner function. A scanner is an apparatus or function that converts an image, document, etc. into a digital still image for communication or recording. In the present embodiment, the digital still image may be color or monochrome, but can generate a multi-valued image with a higher gradation than binary. The digital still image may be a snapshot of a video.

[0032] In addition to the scanner function, the image processing apparatus 20 may have a facsimile function, a print function, a copy function, etc. The image processing apparatus 20 may be called an image forming apparatus, a printing apparatus, a printer, or a scanner apparatus, etc.

[0033] The image processing apparatus 20 in FIG. 6(a) can, by itself, scan a document image to generate a multi-gradation input image, and perform binarization processing, margin removal processing, magnification / reduction processing, top-bottom estimation processing, etc. on the input image. The top-bottom estimation processing, etc. is performed as an example of preprocessing for OCR, but the top-bottom estimation processing, etc. may be performed regardless of OCR. For example, the image processing apparatus 20 attaches top-bottom information to the image data, and when an arbitrary PC, etc. displays it, the image data can be rotated based on the top-bottom information.

[0034] On the other hand, as shown in FIG. 6(b), different apparatuses may perform the generation of the input image and the top-bottom estimation processing, etc. of the present embodiment, respectively. FIG. 6(b) is an example of an image processing system 100 in which the information processing apparatus 40 performs the top-bottom estimation processing, etc. This image processing system 100 has an information processing apparatus 40 and an image processing apparatus 20. The information processing apparatus 40 and the image processing apparatus 20 are communicably connected by a LAN, Wi-Fi (registered trademark), or a USB cable, etc. within the facility.

[0035] When a user sets a document on the image processing apparatus 20 and executes scanning, the image processing apparatus 20 transmits a multi-tone input image to the information processing apparatus 40 via the network N. The information processing apparatus 40 can receive the multi-tone input image generated by the image processing apparatus 20 scanning the document and perform binarization processing, margin removal processing, magnification / reduction processing, top-bottom estimation processing, etc. on the input image.

[0036] Also, as shown in FIG. 6(c), the binarization processing may be executed as part of the workflow processing. A workflow is a series of processes that combine and execute a plurality of processes (for example, scanning, saving to the cloud, or sending an email, etc.). For example, there is a service in which the information processing system 60 performs predetermined processing on the image data generated by a device reading a document and then saves it to the cloud or sends an email.

[0037] FIG. 6(c) shows an image processing system 100 that executes a workflow. The image processing system 100 includes an information processing system 60 and an image processing apparatus 20. The information processing system 60 and the image processing apparatus 20 are communicably connected via a wide-area network N1 such as the Internet. The image processing apparatus 20 is arranged in a facility such as a company and is connected to the network N2 laid in the facility. The network N2 may be a LAN, Wi-Fi (registered trademark), wide-area Ethernet (registered trademark), or a mobile phone network such as 4G, 5G, 6G, etc.

[0038] The information processing system 60 may be realized by one or more computers. The information processing system 60 may be realized by cloud computing or may be realized by a single information processing apparatus. Cloud computing refers to a form in which resources on the network are used without awareness of specific hardware resources. The information processing system 60 may exist on the Internet or on-premises.

[0039] The image processing apparatus 20 and the information processing system 60 may execute a web application. A web application is an application that operates by the cooperation of a program in a programming language (e.g., JavaScript (registered trademark)) that operates on a web browser and a program on the web server side. On the other hand, an application that cannot be executed unless it is installed in the image processing apparatus 20 is called a native application. Regarding this embodiment as well, the application executed by the image processing apparatus 20 may be a web application or a native application.

[0040] The information processing system 60 generates screen information for the image processing apparatus 20 to display the screen of the web application. The screen information is a program described in HTML, XML, a scripting language, CSS (Cascading Style Sheet), etc. The structure of the web page is specified mainly by HTML, the operation of the web page is defined by the scripting language, and the style of the web page is specified by CSS.

[0041] In the form of FIG. 6(c), the image processing apparatus 20 generates an input image by a scanner function, performs binarization processing, margin removal processing, magnification / reduction processing, top / bottom estimation processing, etc., and transmits it to the information processing system 60 via the networks N1, N2. The information processing system 60 performs, for example, OCR processing and sends an email or saves it in the cloud.

[0042] Alternatively, the information processing system 60 may perform some or all of binarization processing, margin removal processing, magnification / reduction processing, top / bottom estimation processing, etc. The image processing apparatus 20 generates an input image by a scanner function and transmits the input image to the information processing system 60 via the networks N1, N2. The information processing system 60 performs binarization processing, margin removal processing, magnification / reduction processing, top / bottom estimation processing, etc. on the received input image and executes the subsequent workflow.

[0043] Further, the input images for which the information processing apparatus 40 in FIG. 6(b) and the information processing system 60 in FIG. 6(c) perform the estimation processing of up and down need not be those scanned by the image processing apparatus 20. For example, the image processing apparatus 20 may be a digital camera or a smartphone, and the information processing apparatus 40 or the information processing system 60 may perform the estimation processing of up and down on the input images generated by imaging documents or the like with these. Further, the information processing apparatus 40 or the information processing system 60 can perform the estimation processing of up and down on any image on the network.

[0044] In the following description, unless otherwise specified, it is assumed that the image processing apparatus 20 in FIG. 6(a) performs binarization processing.

[0045] <Hardware Configuration Example> With reference to FIGS. 7 and 8, the hardware configurations of the image processing apparatus, the information processing apparatus 40, and the information processing system 60 included in the image processing system according to the present embodiment will be described.

[0046] <<Configuration Example of Image Processing Apparatus>> FIG. 7 is a diagram showing an example of the hardware configuration of the image processing apparatus 20 according to the embodiment of the present invention. As shown in FIG. 7, the image processing apparatus 20 includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network I / F 950.

[0047] The controller 910 includes a CPU 901, which is the main part of a computer, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907, an HDD controller 908, and an HD 909, and is configured to connect between the NB 903 and the ASIC 906 via an AGP (Accelerated Graphics Port) bus 921.

[0048] Among these, the CPU 901 controls the entire image processing apparatus 20. The NB 903 is a bridge for connecting the CPU 901 to the MEM-P 902, SB 904, and the AGP bus 921, and has a memory controller for controlling read / write operations on the MEM-P 902, and a PCI (Peripheral Component Interconnect) master and an AGP target.

[0049] The MEM-P 902 consists of a ROM 902a which is a memory for storing programs and data for realizing the respective functions of the controller 910, and a RAM 902b which is used as a memory for developing programs and data, and for drawing during memory printing. Note that the programs stored in the RAM 902b may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD in an installable or executable file format.

[0050] The SB 904 is a bridge for connecting the NB 903 to PCI devices and peripheral devices. The ASIC 906 is an IC (Integrated Circuit) for image processing applications having hardware elements for image processing, and has the role of a bridge for connecting the AGP bus 921, PCI bus 922, HDD controller 908, and MEM-C 907 respectively. This ASIC 906 includes a PCI target and an AGP master, an arbiter (ARB) forming the core of the ASIC 906, a memory controller for controlling the MEM-C 907, a plurality of DMACs (Direct Memory Access Controllers) for performing operations such as rotation of image data by means of hardware logic, etc., and a PCI unit for performing data transfer via the PCI bus 922 between the scanner unit 931, printer unit 932, and fax unit 933. Note that a USB (Universal Serial Bus) interface or an IEEE1394 (Institute of Electrical and Electronics Engineers 1394) interface may be connected to the ASIC 906.

[0051] The short-range communication circuit 920 has a card reader 920a for reading user authentication information and the like stored in an IC card or the like.

[0052] The operation panel 940 has a touch panel 940a for receiving inputs from the user and a numeric keypad 940b. Also, the touch panel 940a displays a setting screen of the image processing apparatus 20 and the like.

[0053] <<Configuration Example of Information Processing Apparatus and Information Processing System>> The information processing apparatus 40 or the information processing system 60 is realized by, for example, a computer having the hardware configuration shown in FIG. 8. FIG. 8 is a hardware configuration diagram of an example of a computer. The computer 500 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD controller 505 (Hard Disk Drive), a display 506, an external device connection I / F 508 (Interface), a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0054] Among these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501 such as the IPL. The RAM 503 is used as the work area of the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as the cursor, menu, window, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. External devices in this case are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 8.

[0055] Also, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513 as an example of a removable recording medium. The optical drive 514 is a CD, DVD, Blu-Ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515 such as a flash memory.

[0056] <Functional Configuration of Image Processing Apparatus> Next, with reference to FIG. 9, the functions of the image processing apparatus 20 will be described. FIG. 9 is an example of a functional block diagram for explaining the functions of the image processing apparatus 20 by dividing them into blocks. In FIG. 9, as an example of the image processing apparatus 20, a schematic configuration of a digital color image forming apparatus is shown.

[0057] The image processing apparatus 20 includes a reading unit 1, an image processing unit 2, an image data storage unit 3, a printing unit 4, a setting reception unit 6, a preprocessing unit 11, and an inference unit 12. Each of these functional units included in the image processing apparatus 20 is a function or means realized by the CPU 901 executing instructions included in one or more programs installed in the image processing apparatus 20. Alternatively, each functional unit may be realized by an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a hard circuit module, etc.

[0058] The setting reception unit 6 receives the setting conditions at the time of scanning the document by the user. The user sets, as the condition settings, for example, the application to be used and any conditions within the application (file format, resolution, color / black and white, presence or absence of OCR application, storage destination, etc.). The setting conditions received by the setting reception unit 6 are also input to the image processing unit 2.

[0059] When the setting at the time of scanning by the user is completed and the process is started, the reading unit 1 reads the document. The reading unit 1 is a device that optically reads the document and generates image data. The reading unit 1 sends the image data read according to the setting conditions received by the setting reception unit 6 to the image processing unit 2.

[0060] The image processing unit 2 includes a gamma correction unit 21, a region detection unit 22, a data interface unit 23, a color processing / UCR unit 24, and a printer correction unit 25. At the time of generating a scanned image, the gamma correction unit 21 and the region detection unit 22 process the image data in order. At the time of generating a copy image, the gamma correction unit 21, the region detection unit 22, the data interface unit 23, the color processing / UCR unit 24, and the printer correction unit 25 process the image data in order.

[0061] The gamma correction unit 21 is a means for performing one-dimensional conversion on each signal of the image data (8 bits for each of the R, G, and B colors after A / D conversion) read by the reading unit 1 in order to equalize the tonal balance for each color. For the purpose of explanation here, the image data is converted into density linear signals (RGB signals: the signal value representing white is set to 0) after conversion, and the output of the gamma correction unit 21 is sent to the region detection unit 22 and directly to the data interface unit 23. The region detection unit 22 determines whether the pixel or pixel block of interest in the read image data is a character region or a non-character region (i.e., a pattern), and also determines whether it is a chromatic color or an achromatic color.

[0062] The data interface unit 23 is, for example, an interface for an HDD when temporarily storing the determination result from the region detection unit 22 and the image data after processing by the gamma correction unit 21 in the image data storage unit 3.

[0063] In the processing flow of the scanned image, the determination result from the region detection unit 22 and the image data processed by the gamma correction unit 21 are temporarily stored in the image data storage unit 3. The preprocessing unit 11 acquires the image data from the image data storage unit 3.

[0064] On the other hand, when generating a copy image, the gamma-corrected image data and the determination result from the region detection unit 22 are sent from the data interface unit 23 to the color processing / UCR unit 24. The color processing / UCR unit 24 is a means for selecting color processing and UCR processing based on the determination result for each pixel region or pixel block. Further, the printer correction unit 25 receives the C, M, Y, Bk image signals from the color processing / UCR unit 24, performs gamma correction processing and dither processing considering the printer characteristics, and sends them to the printing unit 4.

[0065] That is, in the processing flow of the copy image, the results of the character determination by the region detection unit 22 to determine whether the pixel or pixel block of interest in the read image data is a character region or a non-character region (i.e., a pattern), and the color determination to determine whether it is a chromatic color or an achromatic color are used. The color processing / UCR unit 24 performs color reproduction processing suitable for the original according to the results of the character determination and the color determination.

[0066] The printing unit 4 transfers the image data output by the image processing unit 2 to a medium such as paper. The printing unit 4 controls the transfer printing unit to output a copy image.

[0067] FIG. 10 is a functional block diagram for explaining the functions of the preprocessing unit 11 by dividing them into blocks. The preprocessing unit 11 performs predetermined preprocessing on the input image data. The image data output by the preprocessing unit 11 is input to the inference unit 12. The inference unit 12 performs inference by AI (such as sky-earth discrimination) on the image data output by the preprocessing unit 11. The inference unit 12 is a model that has learned the correspondence between image data and sky-earth in advance.

[0068] The preprocessing unit 11 includes a binary image generation unit 15, a margin processing unit 16, a division unit 17, and a scaled image generation unit 18. The binary image generation unit 15 converts the image data into a binary image when the image data is a multi-valued image. The binary image generation unit 15 may skip the process when the image data is not a multi-valued image, or may convert the binary image into a binary image. The margin processing unit 16 performs margin removal to remove the margin part at the edge of the image for the binary image. The division unit 17 divides the image data into a plurality of scaled reference regions. The size of the scaled reference region may be any of a fixed size, a size according to the image data, or a size according to the histogram calculation block. The scaled image generation unit 18 performs a reduction scaling process on the image data from which the margin has been removed. The scaled image (an example of a reduced scaled image) scaled by the scaled image generation unit 18 is input to the subsequent inference unit 12.

[0069] When the image data acquired from the image processing unit 2 is originally a binary image, the binary image generation unit 15 does not perform any particular processing. The binarization method may be any method such as the error diffusion method or the discriminant analysis method. The binary image created here can also be used for other processes such as OCR. The binary image (or the image generated by scanning) generated by the binary image generation unit 15 is input to the margin processing unit 16.

[0070] The margin processing unit 16 divides the binary image into histogram calculation blocks. The margin processing unit 16 calculates the histogram of the pixel values for each histogram calculation block, and determines whether the histogram calculation block is a "white", "black", or "black and white" block (see Fig. 12).

[0071] The margin processing unit 16 removes the margins existing at the image edges from the binary image input from the binary image generation unit 15. That is, the margin processing unit 16 regards a block group in which histogram calculation blocks are horizontally connected as a "row", and checks whether the row consists entirely of "white" histogram calculation blocks. Similarly, a block group in which histogram calculation blocks are vertically connected is regarded as a "column", and it is checked whether the column consists entirely of "white" histogram calculation blocks. The margin processing unit 16 looks at rows and columns from the inside of the image edges (top, bottom, left, and right), determines a row or column where a block group composed of "white" is connected as a margin, and removes the margin for one row or one column. The margin processing unit 16 starts processing from the image edge and ends the processing at the image edge at the timing when it faces a row or column having a histogram calculation block other than "white".

[0072] As a method for actually removing the block group determined to be the margin part, processing of directly removing the margin part from the binary image (the image data becomes smaller) may be performed. Alternatively, this method may be a method of sending the coordinates of the pixel region excluding the margin part (for example, the coordinates of the upper left vertex and the lower right vertex of the rectangular region excluding the margin part) as the result of the margin processing unit 16 to the subsequent processing without directly processing the binary image. In the following description, it is assumed that the latter method of "sending the coordinates of the pixel region excluding the margin part as the result of the margin processing unit 16 to the subsequent processing" is adopted for explanation.

[0073] The following analysis results are input to the division unit 17 and the scaled image generation unit 18 (the scaled image generation unit 18 may be input from the division unit 17). · The result of the margin processing unit 16 removing the margin (the coordinates of the pixel region excluding the margin part). ·Analysis results calculated during the process of performing margin removal (the histogram information calculated for each block by dividing the image into histogram calculation blocks, and the determination results of "white", "black", and "black and white"). The histogram information refers to the number of white pixels and black pixels. ·The binary image input from the binary image generation unit 15 (when the process of directly removing the margin part from the binary image is performed, the binary image obtained by removing the margin part from the binary image input from the binary image generation unit 15) The scaled image generation unit 18 uses the analysis results input by the margin processing unit 16 to determine an appropriate scaling method for each region and performs a reduction scaling process on the binary image. When the margin processing unit 16 performs the process of removing the margin from the binary image itself, the scaled image generation unit 18 performs the reduction scaling process of the binary image using only the analysis results (the histogram information calculated for each histogram calculation block and the determination results of "white", "black", and "black and white") calculated during the process of performing margin removal.

[0074] The scaled image generation unit 18 uses the existing area averaging method as the basic reduction scaling method. The scaled image generation unit 18 checks the analysis results (the histogram information calculated for each histogram calculation block and the determination results of "white", "black", and "black and white") calculated during the process of performing margin removal input by the margin processing unit 16 for each scaled reference region obtained by the division unit 17 dividing the image data.

[0075] The scaled image generation unit 18 refers to the determination results of "white", "black", and "black and white" of the histogram calculation block corresponding to the scaled reference region. Note that corresponding to the scaled reference region may be either the case where a part of the histogram calculation block overlaps with the scaled reference region or the case where the entire histogram calculation block is included in the scaled reference region (see Fig. 22).

[0076] When the magnification reference area consists only of histogram calculation blocks of "white" or "black", the magnification image generation unit 18 calculates the pixel values after reduction magnification with reference to the histogram information. When histogram calculation blocks of only "white" and only "black" are mixed in the magnification reference area, the pixel values after reduction magnification are calculated according to the ratio of each number. When the magnification reference area contains the determination result of "black and white", the magnification image generation unit 18 calculates the pixel values after magnification by the area averaging method. The magnification image generated by the magnification image generation unit 18 is input to the inference unit 12.

[0077] <Relationship between magnification reference area and histogram calculation block> FIG. 11 shows an example of division of image data into histogram calculation blocks. FIG. 11(a) shows the original image data, and FIG. 11(b) shows the image data divided into histogram calculation blocks. One frame surrounded by a dotted line is one histogram calculation block. In this way, the image data is divided into histogram calculation blocks of the same shape.

[0078] FIG. 12 shows an example of a histogram calculated by an arbitrary histogram calculation block. According to the histogram in FIG. 12(a), since there are only white pixels, the histogram calculation block is determined to be "white". According to the histogram in FIG. 12(b), since there are only black pixels, the histogram calculation block is determined to be "black". According to the histogram in FIG. 12(c), since white pixels and black pixels are mixed, the histogram calculation block is determined to be "black and white".

[0079] After dividing the binary image into histogram calculation blocks, the margin processing unit 16 calculates the histogram of pixel values for each histogram calculation block. For the sake of speed increase, it is preferable to perform the calculation of the histogram by sampling only a part of the pixels in the histogram calculation block.

[0080] Next, referring to FIG. 13, the relationship between the zoom reference region and the histogram calculation block will be described. FIGS. 13(a) and 13(b) both show the zoom reference region and the histogram calculation block set for the binary image. FIG. 13(a) shows the case where the zoom reference regions R1 to R9 are larger than the histogram calculation block, and FIG. 13(b) shows the case where the zoom reference regions R1 to R9 are smaller than the histogram calculation block. In FIG. 13, the dotted line indicates the histogram calculation block, and the solid line indicates the zoom reference regions R1 to R9.

[0081] When FIG. 13(a) is adopted, since a plurality of histogram calculation blocks overlap one zoom reference region R1 to R9, it is possible to more finely determine "white", "black", and "black and white" for each of the zoom reference regions R1 to R9. When FIG. 13(b) is adopted, since one histogram calculation block overlaps a plurality of zoom reference regions, the determination of "white", "black", and "black and white" for one histogram calculation block is likely to affect a plurality of zoom reference regions. However, since the number of times of calculating the histogram is reduced, the processing load (processing time) can be reduced.

[0082] The histogram calculation block may have a fixed number of blocks or a fixed size. Also, the size (i.e., the number as well) of the histogram calculation block may be dynamically changed according to the size of the binary image.

[0083] As described above, the zoom image generation unit 18 controls the zoom method and determines the pixel value after reduction zoom according to whether the histogram calculation block included in the zoom reference region contains only "white" and "black" or "black and white". An example of the classification method of "white", "black", or "black and white" is as follows. Note that the threshold value (e.g., 99%) is just an example. · Relative frequency of "white" in the histogram is 99% or more → "white" · Relative frequency of "black" in the histogram is 99% or more → "black" · When neither the determination condition of "white" nor "black" is satisfied → "black and white" For example, assume that the histogram calculation blocks A to C have the following histogram information. Histogram information of histogram calculation block A "Black (pixel value 0): 1%, white (pixel value 255): 99%", Histogram information of histogram calculation block B "Black (pixel value 0): 55%, white (pixel value 255): 45%", Histogram information of histogram calculation block C "Black (pixel value 0): 99%, white (pixel value 255): 1%" In this case, whether it is "white", "black", or "black and white" is determined as follows. Histogram calculation block A: "white" Histogram calculation block B: "black and white" Histogram calculation block C: "black" Figure 14 shows the classification results of "white", "black", and "black and white" for each histogram calculation block in the case of Figure 13(a). Note that a histogram calculation block of "white" or "black" may be referred to as "flat", and a histogram calculation block of "black and white" may be referred to as "non-flat". In Figure 14, flat histogram calculation blocks are indicated by hatching, and non-flat histogram calculation blocks are shown in solid color. For the magnification reference regions R1 to R9 in Figure 14, the method for determining flat and non-flat and the results will be described. · Magnification reference regions R1 to R3, R7 to R9 The magnification reference regions R1 to R3, R7 to R9 partially overlap with a histogram calculation block containing characters. A histogram calculation block containing characters is determined to be "black and white". Therefore, the pixel values of the magnification reference regions R1 to R3, R7 to R9 are determined by the area averaging method. · Magnification reference regions R4 to R6 There are no histogram calculation blocks containing characters in the zoom reference areas R4 to R6. Also, since the histogram calculation blocks in the zoom reference areas R4 to R6 consist only of white pixels, they are determined to be "white". Therefore, the pixel values in the zoom reference areas R4 to R6 are determined by the histogram information. Among the histogram information, the maximum frequency of 255 becomes the pixel value after zooming in the zoom reference areas R4 to R6. If the zoom reference areas R4 to R6 were composed only of "black" histogram calculation blocks, the maximum frequency of 0 would become the pixel value after zooming in the zoom reference areas R4 to R6. Details will be described with reference to FIG. 16.

[0084] FIG. 15 shows the classification results of "white", "black", and "black and white" for each histogram calculation block in the case of FIG. 13(b). In FIG. 15, the flat histogram calculation blocks are shown by hatched lines, and the non-flat histogram calculation blocks are shown without shading (in FIG. 15, there is only no shading). For the zoom reference areas R1 to R9 in FIG. 15, the method of determining flat and non-flat and the results will be described. BLK1 to BLK4 in FIG. 15 are histogram calculation blocks. · Zoom reference area R1 The zoom reference area R1 overlaps only with the histogram calculation block BLK1, and since the histogram calculation block BLK1 contains characters, it is determined to be "black and white". Therefore, the pixel value of the zoom reference area R1 is determined by the area averaging method. · Zoom reference area R2 The zoom reference area R2 overlaps with the histogram calculation blocks BLK1 and 2, and since the histogram calculation blocks BLK1 and 2 contain characters, it is determined to be "black and white". Therefore, the pixel value of the zoom reference area R2 is determined by the area averaging method. · Zoom reference area R3 The zoom reference area R3 overlaps with the histogram calculation block BLK2, and since the histogram calculation block BLK2 contains characters, it is determined to be "black and white". Therefore, the pixel value of the zoom reference area R3 is determined by the area averaging method. · Zoom reference area R4 The zoom reference area R4 overlaps with the histogram calculation blocks BLK1, 3, and since the histogram calculation blocks BLK1, 3 contain characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R4 are determined by the area averaging method. · Zoom reference area R5 The zoom reference area R5 overlaps with the histogram calculation blocks BLK1 to 4, and since the histogram calculation blocks BLK1 to 4 contain characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R5 are determined by the area averaging method. · Zoom reference area R6 The zoom reference area R6 overlaps with the histogram calculation blocks BLK2, 4, and since the histogram calculation blocks BLK2, 4 contain characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R6 are determined by the area averaging method. · Zoom reference area R7 The zoom reference area R7 overlaps with the histogram calculation block BLK3, and since the histogram calculation block BLK3 contains characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R7 are determined by the area averaging method. · Zoom reference area R8 The zoom reference area R8 overlaps with the histogram calculation blocks BLK3, 4, and since the histogram calculation blocks BLK3, 4 contain characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R8 are determined by the area averaging method. · Zoom reference area R9 The zoom reference area R9 overlaps with the histogram calculation block BLK4, and since the histogram calculation block BLK4 contains characters, it is determined as "black and white". Therefore, the pixel values of the zoom reference area R9 are determined by the area averaging method.

[0085] <Method for Determining Pixel Values of Zoom Reference Area> Figure 16 is a diagram for explaining the method for determining the pixel values of the zoom reference area. It is assumed that the margin removal process has been completed. Also, for convenience of explanation in Figure 16, one zoom reference area R11 to R14 each contains four histogram calculation blocks BLK1 to 4. · If the histogram calculation block included in the zoom reference area R11 contains even one "black and white" histogram calculation block, the zoom image generation unit 18 calculates the pixel value after reduction zooming by the area zooming method. · When all of the histogram calculation blocks included in the zoom reference area R12 are only a single "white", the zoom image generation unit 18 adopts 255, which is the maximum frequency, as the pixel value after zooming based on the histogram information. · When all of the histogram calculation blocks included in the zoom reference area R13 are only a single "black", the zoom image generation unit 18 adopts 0, which is the maximum frequency, as the pixel value after zooming from the histogram information. · When the histogram calculation blocks included in the zoom reference area R14 are a mixture of "white" and "black", the pixel value after zooming is determined according to the ratio of the number of "white" and "black" histogram calculation blocks.

[0086] (1) Since the zoom reference area R11 in FIG. 16 includes a "black and white" histogram calculation block, the pixel value after reduction zooming is calculated by the area averaging method. (2) Since the zoom reference area R12 in FIG. 16 only includes a "white" histogram calculation block, 255, which is the maximum frequency, becomes the pixel value after zooming according to the histogram information. (3) Since the zoom reference area R13 in FIG. 16 only includes a "black" histogram calculation block, 0, which is the maximum frequency, becomes the pixel value after zooming according to the histogram information. (4) Since the zoom reference area R14 in FIG. 16 has a mixture of "white" and "black" histogram calculation blocks, the pixel value after zooming is determined according to the ratio of the number of "white" and "black" histogram calculation blocks. The pixel value after reduction zooming is 3 / (3 + 1)×255 + 1 / (3 + 1)×0 ≈ 191.

[0087] <Flow of processing or operation> FIG. 17 is an example of a flowchart showing that the image processing apparatus 20 performs reduction zooming processing on a binary image as preprocessing.

[0088] In step S11, when the image data obtained by scanning the document is a multi-valued image, the binary image generation unit 15 performs binarization processing. If the image data obtained by scanning is originally a binary image, it may remain as a binary image. The binarization method may be an arbitrary method such as the error diffusion method or the discriminant analysis method. The created binary image can also be used for other processes such as OCR.

[0089] Next, in step S12, the margin processing unit 16 removes the margins existing at the image edges from the binary image. Note that, in order to identify the margin portions, the margin processing unit 16 divides the binary image into histogram calculation blocks and creates a histogram for each histogram calculation block. The details of margin removal will be described with reference to FIG. 18.

[0090] Next, in step S13, the scaled image generation unit 18 checks the determination results of "white", "black", and "white and black" determined from the histogram information calculated for each histogram calculation block for each scaled reference area.

[0091] Next, in step S14, the division unit 17 divides the image data into scaled reference areas. Based on the results confirmed in step S13, the scaled image generation unit 18 determines the method of scaling processing for each scaled reference area. The scaled image generation unit 18 determines whether the scaled reference area is composed of only "white", only "black", or a mixture of "white" and "black". If the determination in step S14 is Yes, the process proceeds to step S15; if No, the process proceeds to step S16.

[0092] In step S15, since the scaled reference area is composed of histogram calculation blocks of only "white" or only "black" (when not including the histogram calculation block of "white and black"), the scaled image generation unit 18 calculates the pixel values after reduction scaling by referring to the histograms of the histogram calculation blocks included in the scaled reference area. The details of the switching of the scaling method will be described with reference to FIG. 21.

[0093] In step S16, since the zoom reference area is configured to include the histogram calculation block of "black and white", the zoomed image generation unit 18 calculates the pixel values after zooming for the zoom reference area using the area averaging method.

[0094] Thus, the processing of the preprocessing unit 11 (including the binary image generation unit 15, the margin processing unit 16, and the zoomed image generation unit 18) is completed.

[0095] <Explanation of margin removal> FIG. 18 is a diagram for explaining the removal of margins by the margin processing unit 16. FIG. 18(a) shows the classification result of classifying the binary image into "white", "black", or "black and white". In FIG. 18(a), the histogram calculation block including characters is classified as "black and white". The margin processing unit 16 checks whether each of the upper, lower, left, and right ends of the classified histogram calculation block is a margin part one row and one column at a time in the inner direction.

[0096] First, the method of checking one row at a time will be explained. In the case of checking one row at a time, the margin processing unit 16 starts checking whether it is a margin part in the direction indicated by arrow 121 from the topmost block row of the binary image (it may also be checked from right to left). If the topmost block row of the binary image is entirely composed of histogram calculation blocks of "white", that row is determined to be a "margin" and becomes the margin removal target area.

[0097] After the check of the topmost block row is completed, the margin processing unit 16 checks the block row one below (that is, the second block row from the top) as indicated by arrow 122. As shown in FIG. 18(a), if the block row includes a histogram calculation block determined to be "black" or "black and white" other than "white", the row direction check from the upper end stops there, and the margin removal target area on the upper end side reaches the previous block row (in the example of FIG. 18(a), only the topmost block row).

[0098] If, hypothetically, the second block row were also composed only of "white", similar to the topmost block row, the margin processing unit 16 would recognize it as "margin" and add the second block row to the margin removal target area set when checking the topmost block row. The margin processing unit 16 then similarly checks the third block row next. This check is repeated until it reaches a block row containing a histogram calculation block determined to be "black" or "black and white" other than "white".

[0099] The margin processing unit 16 similarly determines whether it is a margin part in the direction indicated by arrow 123 also on the lower end side. In the example of FIG. 18(a), only the bottommost block row is determined to be the margin removal target area.

[0100] Next, a method of checking one column at a time will be described. In the case of column direction checking, the margin processing unit 16 checks whether it is a margin part in the same procedure from top to bottom for the block columns as indicated by arrow 124 in the column direction (alternatively, it may be checked from bottom to top). After finishing the check of the leftmost block column, the margin processing unit 16 checks the block column one to the right (i.e., the second block column from the left) as indicated by arrow 125. If the block column contains a histogram calculation block determined to be "black" or "black and white" other than "white", the column direction check from the left end will end there.

[0101] The margin processing unit 16 similarly determines whether it is a margin part in the direction indicated by arrow 126 also on the right end side. In the example of FIG. 18(a), only the rightmost block column is determined to be the margin removal target area. As described above, in the example of FIG. 18(a), the left and right block columns are determined to be the margin removal target areas.

[0102] FIG. 18(b) shows a binary image in which the margin removal target area has been determined after the checks from the upper, lower, left, and right ends are completed. The inside of the thick line 301 in FIG. 18(b) becomes the binary image after margin removal.

[0103] The margin processing unit 16 outputs the following information to the magnification image generation unit 18. "Binary image before margin removal" "The upper left and lower right coordinates indicating the image range after removing the margins shown by the thick frame in Fig. 18(b)" "The histogram information of each histogram calculation block and the determination results of 'white', 'black', and 'black and white'" Although it was explained that the result of margin removal is sent to the subsequent stage as coordinates due to memory and processing time limitations, if there are no memory and processing time constraints, for the binary image input from the binary image generation unit 15, the "binary image after corresponding margin removal processing" with the margin removal target area removed and the "histogram information calculated in each block and the determination results of 'white', 'black', and 'black and white'" may be output to the scaled image generation unit 18.

[0104] In addition, as a result of checking the margin parts from the upper, lower, left, and right ends to the inside, if all the histogram calculation blocks are determined to be "white", the input image data is determined to be "blank paper". In this case, the image processing apparatus 20 may end the processing for the image data (not perform the subsequent stage) and proceed to the control for processing the next image data. Alternatively, when all the histogram calculation blocks are determined to be "white", the image processing apparatus 20 may recognize it as "blank paper", leave one histogram calculation block in the inner central part, and perform control to process the other histogram calculation blocks as the margin removal target area.

[0105] In Fig. 18, the flow of a series of processes in the margin processing unit 16 was explained, but the binary image in Fig. 18(a) was a binary image composed only of "white" and "black and white". Therefore, Fig. 19 shows a binary image including histogram calculation blocks determined to be "white", "black", or "black and white".

[0106] Fig. 19(a) is a binary image, and Fig. 19(b) shows the binary image after the margin processing unit 16 removed the margins. Fig. 20 shows the binary image in Fig. 19(b) with symbols assigned to the histogram calculation blocks for explanation. The hatched areas are determined to be "white", the plain areas are "black and white", and the horizontal lines are determined to be "black". · Blocks b1 to b8 are determined to be histogram calculation blocks for "black". · Blocks b9 and b10 are determined to be histogram calculation blocks for "black". · Blocks b11 to b15 are determined to be "black and white" histogram calculation blocks. · Block b16 is determined to be a "black" histogram calculation block. · Blocks b17 and b18 are determined to be "black" histogram calculation blocks. · Blocks b19 to b21 are determined to be "black and white" histogram calculation blocks. · Blocks b22 to b24 are determined to be "black" histogram calculation blocks. · Blocks b25 to b32 are determined to be "black" histogram calculation blocks. · Blocks b33 to b40 are determined to be "black and white" histogram calculation blocks. · Blocks b41 and b42 are determined to be "white" histogram calculation blocks. · Blocks b43 to b46 are determined to be "black and white" histogram calculation blocks. · Blocks b47 and b48 are determined to be "white" histogram calculation blocks. · Blocks b49 to b56 are determined to be "white" histogram calculation blocks (since they are at the ends, they are regions to be subject to margin removal).

[0107] Thus, even if the histogram calculation block contains a black region, it is possible to determine "white", "black", or "black and white".

[0108] <<Example of Switching Magnification Methods>> Next, with reference to FIG. 21, the magnified image will be described using an image. FIG. 21(a) shows the magnification reference region and the histogram calculation block, and FIG. 21(b) shows the magnification method for each magnification reference region.

[0109] For each of the magnification reference regions R1 to R9 indicated by the red frames in FIG. 21(a), the magnification image generation unit 18 checks the "judgment results of 'white', 'black', and 'black and white'" among the "histogram information calculated for each block and the judgment results of 'white', 'black', and 'black and white'" created by the margin processing unit 16.

[0110] For each of the zoom reference regions R1 to R9, the zoomed image generation unit 18 determines, based on the "judgment results of 'white', 'black', and 'black and white'", a. only 'white' or only 'black' b. a region with a mixture of 'white' and 'black' c. a region containing 'black and white' and discriminates which one (Fig. 21(b)). · The zoom reference regions R1 to R3 and R7 to R8 are determined to be 'black and white' and are indicated by horizontal lines. · The zoom reference regions R4 to R6 are determined to be 'white' and are indicated by vertical lines.

[0111] As shown in Fig. 21(b), after the discrimination for each zoom reference region is completed, the zoomed image generation unit 18 performs reduction zoom while switching the zoom method for each zoom reference region. That is, the zoomed image generation unit 18 calculates the pixel values after reduction zoom.

[0112] a. In the zoom reference region of a., the pixel values after reduction zoom are calculated from the histogram information calculated by the margin processing unit 16. For example, the following description is given assuming the case where there are the following blocks The histogram of the histogram calculation block A is "black (pixel value 0): 1%, white (pixel value 255): 99%", The histogram of the histogram calculation block B is "black (pixel value 0): 55%, white (pixel value 255): 45%", The histogram of the histogram calculation block C is "black (pixel value 0): 99%, white (pixel value 255): 1%", Histogram calculation block A: 'white' Histogram calculation block B: 'black and white' Histogram calculation block C: 'black' When all the histogram calculation blocks included in the zoom reference area are single "white" like the histogram calculation block A, the zoom image generation unit 18 adopts 255 of the maximum frequency as the pixel value after zooming from the histogram information. When all the histogram calculation blocks included in the zoom reference area are single "black" like the histogram calculation block C, the zoom image generation unit 18 adopts 0 of the maximum frequency as the pixel value after zooming from the histogram information.

[0113] As in b., when the histogram calculation blocks corresponding to the zoom reference area are a mixture of "white" and "black" like the histogram calculation block A or the histogram calculation block C, the zoom image generation unit 18 adopts 128, which is the average of the maximum frequencies of each, as the pixel value after zooming if the number of "white" and "black" histogram calculation blocks is the same.

[0114] When the number of histogram calculation blocks of either "white" or "black" is large, the zoom image generation unit 18 calculates by multiplying the weight calculated from the occupancy rates of the "white" and "black" histogram calculation blocks in the zoom reference area by the maximum frequency of the histogram. For example, when the number of "white" histogram calculation blocks is N1 and the number of "black" histogram calculation blocks is N2, the zoom image generation unit 18 {N1 / (N1 + N2)} × 255 + {N2 / (N1 + N2)} × 0 as the pixel value.

[0115] Note that, as shown in FIG. 22, there may be a case where one histogram calculation block is not completely included in the zoom reference area and one histogram calculation block straddles the zoom reference area. FIG. 22 shows an enlarged view of the zoom reference area R4 in FIG. 14. The zoom reference area R4 overlaps a total of nine histogram calculation blocks b101 to b109. Among these, only the histogram calculation blocks b104 and b105 are entirely included in the zoom reference area R4. In such a case, the zoom image generation unit 18 can use only the histogram calculation blocks b104 and b105 for calculating the pixel values of the zoom process and not use the other histogram calculation blocks. By doing so, the image processing apparatus 20 can reduce the processing load of the zoom process.

[0116] Also, taking FIG. 22 as an example, regarding the determination of whether the blocks entirely included in the zoom reference area R4 are only "white", only "black", or "black and white", only b104 and b105 are determined to be flat or non-flat.

[0117] In the zoom reference area of c., the zoom image generation unit 18 calculates the pixel values after reduction by the area zoom method. The zoom image generation unit 18 may calculate the pixel values from all the pixels included in the zoom reference area, or may calculate the average by referring to every few pixels according to the sizes (i.e., the zoom ratios) of the images before and after zoom. The zoom image generation unit 18 switches the zoom method for each zoom reference area in this way to generate the zoomed image.

[0118] <Main effects> The image processing apparatus 20 of the present embodiment can utilize the histogram calculated by the margin removal during the subsequent reduction and magnification. The image processing apparatus 20 determines whether the histogram calculation block is flat or not, and calculates the pixel value by the area averaging method only when it is not flat, thereby shortening the processing time of the reduction and magnification while retaining the information of the image before the reduction and magnification. That is, the margin removal performed to remove the margins of the image divides the image into blocks and determines whether each block is a margin part or not. By using the margin information for each block calculated for the margin removal during the subsequent reduction and magnification, the processing time of the reduction and magnification can be shortened while retaining the information of the original image. Therefore, it is possible to suppress the preprocessing from becoming a bottleneck in the overall processing time including inference.

[0119] <Other application examples> As described above, the best mode for carrying out the present invention has been described using examples. However, the present invention is not limited to such examples, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0120] For example, in the present embodiment, the reduction image processing has been described, but a part or all of the present embodiment may be used for the enlargement image processing.

[0121] Configuration examples such as FIGS. 9 and 10 are divided according to the main functions in order to facilitate the understanding of the processing by the image processing apparatus 20. The present invention of the application is not limited by the method and name of the division of the processing units. The processing of the image processing apparatus 20 can be further divided into more processing units according to the processing content. Also, one processing unit can be divided so as to include more processes.

[0122] Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in this specification means a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a device such as a conventional circuit module.

[0123] <Appendix> [Appendix 1] An image processing apparatus that performs scaling processing for each of a plurality of scaling reference regions of image data, a binary image generation unit that binarizes the image data, a margin processing unit that creates flat or non-flat information for each block of the image data binarized by the binary image generation unit, and removes a margin portion based on the flat or non-flat information, a scaled image generation unit that determines a scaling method for each of the scaling reference regions according to the flat or non-flat information of one or more of the blocks corresponding to the scaling reference regions, and generates a scaled image of the image data by the determined scaling method, An image processing apparatus, characterized by comprising the same. [Appendix 2] The image processing apparatus according to Appendix 1, wherein the margin processing unit creates a histogram for each block of the binarized image data, and creates the flat or non-flat information for each block based on the histogram. [Appendix 3] When the block corresponding to the scaling reference region is flat, the scaled image generation unit calculates a pixel value after scaling based on the value of the histogram of the block corresponding to the scaling reference region, When the block corresponding to the zoom reference area is non-flat, the zoom image generation unit calculates the pixel value after zooming based on the average of the pixel values included in the zoom reference area. The image processing apparatus according to appended note 2, characterized in that. [Appended Note 4] When the white flat blocks and the black flat blocks are mixed in all the blocks corresponding to the zoom reference area, The zoom image generation unit calculates the pixel value after zooming based on the ratio of the number of white blocks or the number of black blocks to the number of all the blocks corresponding to the zoom reference area. The image processing apparatus according to appended note 2, characterized in that. [Appended Note 5] The zoom image generation unit determines the block that partially overlaps with the zoom reference area as the block corresponding to the zoom reference area. The image processing apparatus according to any one of appended notes 1 to 4. [Appended Note 6] The zoom image generation unit determines the block entirely included in the zoom reference area as the block corresponding to the zoom reference area. The image processing apparatus according to any one of appended notes 1 to 4. [Appended Note 7] The size of the zoom reference area is larger than the size of the block. The image processing apparatus according to any one of appended notes 1 to 6, characterized in that. [Appended Note 8] The size of the zoom reference area is smaller than the size of the block. The image processing apparatus according to any one of appended notes 1 to 6, characterized in that.

Explanation of Signs

[0124] 11 Preprocessing unit 15 Binary image generation unit 16 Margin processing unit 17 Zoom image generation unit 20 Image processing apparatus

Prior Art Documents

Patent Documents

[0125]

Patent Document 1

Claims

1. An image processing apparatus that performs scaling processing for each of a plurality of scaling reference regions of image data, comprising: a binary image generation unit that binarizes the image data; a margin processing unit that creates flat or non-flat information for each block of the image data binarized by the binary image generation unit, and removes a margin portion based on the flat or non-flat information; a scaled image generation unit that determines a scaling method for each scaling reference region according to the flat or non-flat information of one or more blocks corresponding to the scaling reference region, and generates a scaled image of the image data by the determined scaling method; An image processing apparatus characterized by comprising the above.

2. The image processing apparatus according to claim 1, wherein the margin processing unit creates a histogram for each block of the binarized image data, and creates the flat or non-flat information for each block based on the histogram.

3. When the block corresponding to the scaling reference region is flat, the scaled image generation unit calculates a pixel value after scaling based on the value of the histogram of the block corresponding to the scaling reference region. The image processing apparatus according to claim 2, wherein when the block corresponding to the scaling reference region is non-flat, the scaled image generation unit calculates a pixel value after scaling from the average of the pixel values included in the scaling reference region.

4. When both a white flat block and a black flat block are mixed in all the blocks corresponding to the scaling reference region, The image processing apparatus according to claim 2, wherein the scaled image generation unit calculates a pixel value after scaling based on the ratio of the number of white blocks or the number of black blocks to the number of all the blocks corresponding to the scaling reference region.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the scaled image generation unit determines a block that overlaps at least partially with the scaling reference region as a block corresponding to the scaling reference region.

6. The image processing apparatus according to any one of claims 1 to 4, wherein the scaled image generation unit determines a block that is entirely included in the scaling reference region as a block corresponding to the scaling reference region.

7. The image processing apparatus according to claim 1, wherein the size of the scaling reference region is larger than the size of the block.

8. The image processing apparatus according to claim 1, wherein the size of the zoom reference area is smaller than the size of the block.

9. An image processing method in which an image processing apparatus performs zoom processing for each of a plurality of zoom reference areas of image data, a binary image generation unit performs a process of binarizing the image data, a margin processing unit creates flat or non-flat information for each block of the image data binarized by the binary image generation unit, and performs a process of removing a margin portion based on the flat or non-flat information, a zoomed image generation unit determines a zoom method for each zoom reference area according to the flat or non-flat information of one or more of the blocks corresponding to the zoom reference area, and generates a zoomed image of the image data by the determined zoom method, An image processing method for performing the above.

10. An image processing apparatus that performs zoom processing for each of a plurality of zoom reference areas of image data, a binary image generation unit that binarizes the image data, a margin processing unit that creates flat or non-flat information for each block of the image data binarized by the binary image generation unit, and removes a margin portion based on the flat or non-flat information, a zoomed image generation unit that determines a zoom method for each zoom reference area according to the flat or non-flat information of one or more of the blocks corresponding to the zoom reference area, and generates a zoomed image of the image data by the determined zoom method, A program for causing the apparatus to function as such.

Images