Image processing apparatus and image processing method
Patent Information
- Application Number
- JP2021178183
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-10-29
AI Technical Summary
Existing image capture methods with auxiliary illumination can degrade image quality, leading to inaccurate extraction of embedded information from printed documents.
An image processing apparatus that acquires and processes both illuminated and non-illuminated image data to extract embedded information, using masks to modulate pixel values and enhance patterns, and employs methods to determine authenticity and detect tampering.
Improves the accuracy of information extraction and authenticity verification by enhancing pattern detection and reducing image quality degradation, allowing for reliable identification of original documents.
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for extracting embedded information from a printed manuscript.
Background Art
[0002] When printing an electronic document on paper, a technique of embedding management information is known. In Patent Document 1, an example is described in which auxiliary illumination is irradiated onto an embedded portion of information in a printed manuscript, and the printed manuscript or the embedded information is accurately captured within the focal length.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, when photographing by irradiating auxiliary illumination as in Patent Document 1, the image quality of the captured image may deteriorate depending on the printing method of the printed manuscript. In such a case, there is a risk that the embedded information cannot be extracted.
[0005] Therefore, an object of the present invention is to improve the extraction accuracy of information.
Means for Solving the Problems
[0006] An image processing apparatus according to one aspect of the present invention is an image processing apparatus for extracting embedded information from a print original, comprising: a first acquisition means for acquiring the printing method of the print original; a second acquisition means capable of acquiring first image data by reading the print original while irradiating it with light; a third acquisition means capable of acquiring second image data by reading the print original without irradiating it with light; and an extraction means for extracting the embedded information using at least one of the first image data and the second image data according to the printing method acquired by the first acquisition means. [Effects of the Invention]
[0007] According to the present invention, the accuracy of information extraction can be improved. [Brief explanation of the drawing]
[0008] [Figure 1] This diagram shows the system configuration. [Figure 2] This flowchart shows the process for printing an original document. [Figure 3] This figure shows an example of a page image. [Figure 4] This figure shows an example of a mask. [Figure 5] This figure shows the pattern formed by the mask. [Figure 6] This figure shows an example of a mask. [Figure 7] This figure shows the pattern formed by the mask. [Figure 8] This is a flowchart showing the process of copying an original printed document. [Figure 9] This figure shows the spatial frequency characteristics of the pattern used for embedding. [Figure 10] This diagram shows the details of the tampering check process. [Figure 11] This is a diagram illustrating pattern enhancement masks. [Figure 12] This diagram illustrates a variation of the embedding technique. [Figure 13] It is a diagram showing the system configuration of a mobile terminal. [Figure 14] It is a flowchart showing the flow of article ID information extraction. [Figure 15] It is a diagram for explaining the dark part. [Figure 16] It is a flowchart showing the flow of article ID information extraction. [Figure 17] It is a flowchart showing the flow of article ID information extraction. [Figure 18] It is a flowchart showing the flow of article ID information extraction. [Figure 19] It is a flowchart showing the flow of article ID information extraction. [Figure 20] It is a flowchart showing the flow of determining the printing method. [Figure 21] It is a schematic diagram of an inkjet type and an electrophotographic type MFP.
Embodiments for Carrying Out the Invention
[0009] Hereinafter, embodiments for implementing the technology of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of the features described in the embodiments are essential for the solution of the invention.
[0010] In the following embodiments, we will describe an example in which document ID information is used as embedded information. Furthermore, we will describe an example of guaranteeing the authenticity of the original document as a specific usage form. The terms used in this context are defined below. In this specification, "original document" means an electronic document (digital document) that is registered and managed together with information indicating that its contents are authentic, and "printed original document" means a printed document printed using the data of the electronic document. Therefore, for example, an official document such as the aforementioned "copy of resident registration" issued by a government office falls under the category of "printed original document." Also, "copy of original document" means a printed document obtained by copying the "printed original document" using an image processing device equipped with a copying function. Therefore, for example, a copy of the aforementioned "copy of resident registration" falls under the category of "copy of original document." It should be noted that the embodiments described below are not limited to forms that guarantee the authenticity of the original document, but can be applied in general to multiplexing processes that embed predetermined information on paper in a form that is difficult for the user to see.
[0011] <<Embodiment 1>> <System Configuration> Figure 1 is a block diagram showing the configuration of a printing system according to this embodiment. As shown in Figure 1, this printing system includes MFP (Multi Function Printer) 40, MFP 60 as printing devices, and PC 20 as a host device. MFP 40 has multiple functions such as printer function and scanner function, and also has a copy function that works in conjunction with both functions. MFP 40 may also have functions for saving and transmitting image data to be printed, and functions for sending and receiving faxes. The hardware configurations of MFP 40 and PC 20 will be described below.
[0012] Figure 21 shows the external view of each MFP. Figure 21(a) shows a schematic diagram of the inkjet type MFP40, and Figure 21(b) shows a schematic diagram of the electrophotographic type MFP60. The MFP40 uses an inkjet printing method and has a scanner unit 2102 and a print unit 2103. The print unit 2103 has a recording head 2104 and an ink tank unit 2105. The recording head 2104 ejects ink according to the print data. The ink tank unit 2105 stores the ink supplied to the recording head 2104. The stored inks are cyan, magenta, yellow, and black. Depending on the model, spot colors and other inks may be stored separately. The MFP60 uses an electrophotographic printing method and has a scanner unit 2112 and a print unit 2113. Inside the print unit 2113 are a photosensitive drum 2114 and a transfer roller 2115. The photosensitive drum 2114 holds inks for each color, corresponding to cyan, magenta, yellow, and black. A latent image is formed on the photosensitive drum according to the print data. The transfer roller 2115 transfers the formed latent image to the transported paper, and the printing process is carried out.
[0013] The MFP unit 40, which uses an inkjet printing method, is mainly composed of the following elements. The CPU 41 is an arithmetic processing unit that comprehensively controls the entire MFP 40 and executes, for example, the copying process described later, according to the programs stored in the ROM 43 and RAM 42. RAM 42 is volatile storage that temporarily holds programs and data. ROM 43 is non-volatile storage that holds table data and programs used in the various processes described later. The data transfer interface (DATA TRANSFER I / F) 44 controls the transmission and reception of data between the MFP 60 or PC 20. The print controller 45 controls the heating operation of a heater mounted on a recording head (not shown) based on the print data and ejects ink. The image processing accelerator 46 is an arithmetic processing unit that can perform image processing at a higher speed than the CPU 41. Note that the image processing accelerator 46 is not necessarily required, and depending on the printer specifications, the creation of the above table parameters and image processing may be performed solely by the CPU 41. The scanner controller 47 controls the illumination of LEDs mounted on a scanner unit (not shown), acquires light intensity information from the scanner unit, and controls writing to RAM 42. This allows it to read images of documents placed on a document glass (not shown). The motor controller 48 controls multiple motor units (not shown) to move the recording head relative to the recording paper and move the scanner unit relative to the document.
[0014] Furthermore, the electrophotographic MFP unit 60 has the same configuration as the inkjet MFP unit 40, but the printing device is different. The print controller 65 develops and transfers the print data to the drum surface according to the print data based on control parameters from a predetermined address in RAM 62, and then performs the printing process. Note that the configurations of MFP 40 and MFP 60 are not limited to those shown in the figures, and for example, they may be equipped with a network interface that connects to an external network and communicates with other external PCs, etc.
[0015] The PC20 is mainly composed of the following elements. The CPU21 is a processing unit that comprehensively controls the entire PC20 and executes tasks such as printing the original document (described later) according to the programs stored in the HDD23 and RAM22. RAM22 is volatile storage that temporarily holds programs and data. The HDD23 is non-volatile storage that also holds programs and data. The data transfer interface (DATA TRANSFER I / F)24 controls the transmission and reception of data between the PC20 and the MFP40 or MFP60. This data transmission and reception connection can be wired, such as USB, IEEE1394, or LAN, or wireless, such as Bluetooth or WiFi. The keyboard and mouse interface (KEYBOARD MOUSE I / F)25 is an interface that controls HID (Human Interface Device) such as a keyboard and mouse. The display interface (DISPLAY I / F)26 controls the display on a display not shown. The network interface (Network I / F) 27 connects PC 20 to an external network and communicates with one or more external PCs to perform tasks such as requesting document ID verification, requesting results, and requesting document data.
[0016] <Printing process for the original document> Next, we will explain the process of printing the original electronic document, referring to the flowchart in Figure 2. The series of processes shown in the flowchart in Figure 2 begins when the user selects the electronic document to be printed via the UI of a designated printing application installed on PC20 and instructs it to print, including embedding information indicating its authenticity. In the following explanation, the symbol "S" represents a step.
[0017] In S201, data of the electronic document designated for printing is acquired. In this embodiment, various original electronic documents are stored and managed on an external PC (not shown). PC20 sends a request to the external PC to acquire a specific electronic document via the network interface 27, and receives and acquires data for printing the designated electronic document, specifically PDL data, from the external PC. Here, PDL stands for Page Description Language, and is composed of a set of drawing commands on a page-by-page basis. The types of drawing commands are defined for each PDL specification, and in this embodiment, the following three types are mainly used as examples. Text drawing command: (X1, Y1, color, font information, string information) Box drawing command: (X1, Y1, X2, Y2, color, fill shape) Image drawing command: (X1, Y1, X2, Y2, image file information) In addition to the three types mentioned above, there are also DOT drawing commands for drawing points, LINE drawing commands for drawing lines, and CIRCLE drawing commands for drawing arcs, and these drawing commands are used depending on the application. Commonly used PDLs include PDF (Portable Document Format) proposed by Adobe, XPS proposed by Microsoft, and HP-GL / 2 proposed by HP.
[0018] Figure 3 shows an example of an image representing one page of an electronic document (hereinafter referred to as "page image"). The page image 300 shown in Figure 3 is assumed to have a width of 600 pixels and a height of 900 pixels. The PDL corresponding to page image 300 is shown below. <PAGE=001> <text>50,50, 550, 100, BLACK, STD-18, "ABCDEFGHIJKLMNOPQR”< / text> <text>50,100, 550, 150, BLACK, STD-18, "abcdefghijklmnopqrstuv”< / text> <text>50,150, 550, 200, BLACK, STD-18, "1234567890123456789”< / text> <box> 50, 300, 200, 450, GRAY, STRIPE< / box> 250, 300, 550, 800, “PORTRAIT.jpg”
[0019] The first line of the above PDL<PAGE=001> This is a tag that indicates the page number. Typically, PDLs are designed to describe multiple pages, and tags indicating page breaks are written within the PDL. In the example above, it indicates that everything up to line 9 is page 1. If there is a second page,<PAGE=002> The following will be written. The second line <text> From the third line< / text> This is the TEXT drawing command. In the TEXT drawing command, the first two coordinates indicate the coordinates of the top left of the drawing area (X1, Y1), and the following two coordinates indicate the coordinates of the bottom right of the drawing area (X2, Y2). Next, it is written that the color is "BLACK (black: R=0, G=0, B=0)", the font is "STD (standard)", the font size is 18 points, and the string to be drawn is "ABCDEFGHIJKLMNOPQR". Line 4 <text> From the 5th line< / text> up to and the 6th line <text> From the 7th line< / text> This is also a TEXT drawing command. The strings to be drawn correspond to "abcdefghijklmnopqrstuv" and "1234567890123456789". Line 8 <box> from< / box>The first line is a BOX drawing command, where the first two coordinates indicate the top-left coordinates (X1, Y1) which are the starting point of the drawing, and the following two coordinates indicate the bottom-right coordinates (X2, Y2) which are the ending point of the drawing. Next, the color is specified as "GRAY (R=128, G=128, B=128)" and the fill shape is specified as "STRIPE (striped pattern)". Line 9 is an IMAGE drawing command. The first two coordinates indicate the top-left coordinates (X1, Y1) of the drawing area, and the following two coordinates indicate the bottom-right coordinates (X2, Y2) of the drawing area. It also states that the file name of the image in the specified drawing area is "PORTRAIT.jpg". Note that ".jpg" is the file extension and indicates that it is a JPEG file, a commonly used image compression format. Finally, line 10 indicates that the drawing of the page has finished. Note that the document data actually obtained is often a single file with font data and image files added to the PDL data. This is because, when font data and image files are managed separately, drawing commands alone are insufficient to form the character and image portions, resulting in insufficient information to create the image to be printed. In this step, document data containing such PDL data is obtained by accessing an external PC.
[0020] In S202, information indicating the authenticity of the electronic document to be printed is acquired. In this embodiment, a request to acquire information indicating the authenticity of the electronic document is sent to the external PC mentioned above via the network interface 27, and the information is received and acquired. Here, document ID information is used as the information indicating the authenticity of the electronic document. Document ID information is a predetermined number of bits (32 bits in this embodiment) of information calculated using a hash function based on the document data (data that integrates PDL data, font data, and image files). If even one byte of the digital data constituting the electronic document is changed, different document ID information will be calculated, so a unique document ID will be associated with each electronic document. Note that the risk of tampering with document ID information may be reduced by having multiple external PCs cooperate in storing and managing the electronic document file and document ID information, and by adopting a system configuration such as blockchain. Also, if PC 20 has sufficient resources and can manage document ID information within PC 20, it is possible to perform the verification process internally.
[0021] In S203, rendering is performed based on the document data acquired in S201. This rendering process generates bitmap image data composed of color information for each pixel by executing each drawing command included in the PDL data. For example, in the case of page image 300 in Figure 3 above, a 600x900 pixel bitmap image will be generated. In this embodiment, each pixel of the bitmap image has 256 gradations with 8 bits each for RGB.
[0022] In S204, the process of embedding the document ID information obtained in S202 into the Bitmap image generated in S203 is executed. This process of embedding document ID information into a Bitmap image is called "multiplexing" or "multiplex encoding." When attempting to copy a printed copy obtained by printing a Bitmap image that has undergone this multiplexing process, it becomes possible to extract (decode) the document ID information from a scanned image of the printed copy to determine whether the copy to be copied is the "original printed copy."
[0023] In the first place, handling information with an information processing device like the PC20 means handling binary data. Binary data is information represented by combinations of "0"s and "1"s, and when these "0"s or "1"s are linked together in sequence, they take on a specific meaning. For example, when handling the English word "hello" as binary data, using Shift JIS, one of the character encodings, as an example, the letter "h" corresponds to the binary data "01101000". Similarly, "e" corresponds to the binary data "01100101", "l" to "01101100", and "o" to "01101111". In other words, the string "hello" can be represented as the binary data "0110100001100101011011000110110001101111". Conversely, if we can obtain the binary data "0110100001100101011011000110110001101111", we can obtain the character information representing the English word "hello". Based on this idea, it can be seen that multiplexing can be achieved by embedding predetermined data within the image so that it can be determined as either "0" or "1".
[0024] <Multiplexing> Figures 4(a) and 4(b) show examples of 8x8 pixel masks used to generate "0" and "1". Figure 4(a) is a mask for generating "0", and Figure 4(b) is a mask for generating "1". By adding the values in the mask to the pixel values of the bitmap image, a periodic pattern can be given to each 8x8 pixel region in the image. As described above, the bitmap image in this embodiment is represented by 8 bits of 1 color, and one of the values from 0 to 255 is assigned to each pixel. Values outside the range of 0 to 255 cannot be used as image data, so if the addition result is less than 0 or 256 or greater, it is common practice to replace the value of the addition result with 0 or 255 to bring it within the valid range. In the masks in Figures 4(a) and 4(b), the pixel values are changed to "-10" or "0", but if all the pixel values in the bitmap image corresponding to the mask region are "0", then all the values in that region will be "0". Note that although the case of 8 bits of 1 color is explained here, it is not limited to this. When dealing with digital images, regardless of the number of bits used for representation, there is an effective range, and changes that fall outside that range cannot be applied.
[0025] Figures 5(a) and 5(b) visually illustrate the patterns formed on a bitmap image by the masks shown in Figures 4(a) and 4(b). In the masks in Figures 4(a) and 4(b), the position of "-10" is represented by solid black, and the position of "0" is represented by hatching. As can be seen from Figures 5(a) and 5(b), in the bitmap image after applying the mask, a diagonal line sloping downwards to the left appears as the "pattern representing 0," and a diagonal line sloping downwards to the right appears as the "pattern representing 1."
[0026] The following is pseudocode for alternately applying the masks in Figure 4(a) and Figure 4(b) to the entire bitmap image. 01: int i, j, k, l; 02: int width = 600, height=900; 03: unsigned char *data = image data; 04: int **maskA = mask data; 05: bool isMaskA = true; 06: for(j = 0; j < height; j+=8){ 07: for(i = 0; i < width; i+=8){ 08: for(k = 0; k < 8; k++){ 09: for(l = 0; l < 8; l++){ 10: if(isMaskA == true){ 11: data[(i+k)+(j+l)*width] += maskA[k][l]; 12:} 13:} 14:} 15:} 16:}
[0027] In this embodiment, the pattern data is embedded only for the B value among the RGB values of each pixel in the bitmap image. This is because, when recording with the four CMYK inks on the white area of paper, the Y ink has lower visibility compared to the CMK inks. When multiplexing, it is preferable to ensure that the embedded pattern does not affect the original image as much as possible, that is, to make it as invisible as possible. The three primary colors of light, RGB, and the three primary colors of pigment, CMY, are complementary colors to each other, with R and C, G and M, and B and Y having opposite hues. This means that modulating the B value among the RGB values will significantly change the amount of Y ink used. Therefore, in this embodiment, the modulation of only the B value among the RGB values is controlled to increase the variation in Y ink density.
[0028] In the case of the page image 300 shown in Figure 3, the white area is sufficiently large, and the area excluding the TEXT drawing area 301, BOX drawing area 302, and IMAGE drawing area 303 is the white area. In this case, the masks shown in Figures 4(a) and (b) may not properly embed the pattern data in each drawing area other than the white area. For example, for a drawing area that is solid black (R=0, G=0, B=0), the result after embedding will also be solid black (R=0, G=0, B=0). Therefore, when embedding pattern data in areas other than the white area, especially in the image drawing area, it is preferable to apply a mask like the ones shown in Figures 6(a) and (b). The masks in Figures 6(a) and (b) are masks that give each pixel of the Bitmap image a change of "-10", "0", and "+10". In the case of the masks in Figures 6(a) and (b), even if all the pixel values of the Bitmap image corresponding to the masked area are "0", the pixel values of that area will ultimately be "0" or "+10". Therefore, pattern data can be embedded for all pixels in the Bitmap image. Figures 7(a) and (b) visually show what kind of patterns are given to the Bitmap image by the masks in Figures 6(a) and (b). In the masks in Figures 6(a) and (b), the "-10" position is represented by solid black, the "0" position by hatching, and the "+10" position by solid white. As can be seen from Figures 7(a) and (b), in the Bitmap image after applying the mask, multiple diagonal lines sloping downwards to the left with different densities will appear as the "pattern representing 0", and multiple diagonal lines sloping downwards to the right with different densities will appear as the "pattern representing 1". Note that for TEXT drawing areas and BOX drawing areas, there is usually a high possibility that there will be many white areas, so it is preferable to apply the masks in Figures 4(a) and (b). However, even in text and box drawing areas, the masks shown in Figures 6(a) and (b) may be more suitable depending on the specified color, font thickness, etc. Also, even in image drawing areas, they may be very close to the white area of the paper. For this reason, the type of mask to apply may be determined by obtaining a density histogram for each drawing area.For example, if the maximum RGB value is greater than the threshold, the masks in Figures 4(a) and (b) are applied; otherwise, the masks in Figures 6(a) and (b) are applied. Alternatively, the input color space is L. * a * b * In this case, L represents luminance. * You can also switch the mask applied by comparing the value with a threshold. This makes it possible to achieve reliable embedding.
[0029] Let's return to the explanation of the flowchart in Figure 2.
[0030] In S205, a print image is generated based on the multiplexed Bitmap image (hereinafter referred to as the "multiplexed image") generated in S204. A known method can be used to generate this print image. In this embodiment, each pixel of the multiplexed image is subjected to the following processes: color conversion, color separation, output characteristic conversion, and quantization. Each process is briefly described below.
[0031] (Color conversion process) The color conversion process converts the pixel values (RGB values) of the multiplexed image so that they can be reproduced appropriately by the MFP40. Generally, the color values specified in the PDL drawing command are set to be displayed appropriately on the display, and if these color values are output directly to the printer, the output will have a different color tone than what is seen on the display. Therefore, a color conversion process is performed to absorb the difference in color between the two. This color conversion process uses a 3D lookup table (LUT) that associates a combination of input pixel values (Rin, Gin, Bin) in the multiplexed image with a combination of appropriate output pixel values (Rout, Gout, Bout). Currently, the input values Rin, Gin, and Bin each have 256 gradations. Therefore, it is ideal to prepare a table (Table1
[0256]
[0256]
[0256] [3]) with a total of 16,777,216 pairs of output values (Rout, Gout, Bout) as shown below. Rout = Table1[Rin][Gin][Bin][0] Gout = Table1[Rin][Gin][Bin][1] Bout = Table1[Rin][Gin][Bin][2] However, known techniques to reduce the table size may be used, such as reducing the number of LUT grids from 256 grids to, for example, 16 grids, and interpolating the table values between grids to determine the output value.
[0032] (Color separation process) The color separation process converts the output values Rout, Gout, and Bout from the color conversion process into the output values of each ink color (in this case, the four CMYK colors) to be recorded on paper using an inkjet method. There are various ways to implement this color separation process. In this embodiment, a three-dimensional lookup table (Table2
[0256]
[0256]
[0256] [4]) is used, which associates the above-mentioned combination of output pixel values (Rout, Gout, Bout) with the following combination of ink color pixel values (C, M, Y, K). C = Table2[Rout][Gout][Bout][0] M = Table2[Rout][Gout][Bout][1] Y = Table2[Rout][Gout][Bout][2] K = Table2[Rout][Gout][Bout][3] In this case, it is preferable that only the Y value among the CMYK values corresponding to the modulation of the white area in the multiplexing process of S204 is greater than 0. More specifically, for the CMYK values corresponding to the area where the pixel values R=255, G=255, B=255 have been converted to pixel values R=255, G=255, B=245, it is preferable that the Y value is greater than 0 and the CMK value is smaller than the Y value and close to 0. The reason for this is, as explained in S204, to reduce the visibility of the embedded pattern. As with the color conversion process described above, known techniques for reducing the table size may also be used.
[0033] (Output characteristic conversion process) The output characteristic conversion process converts the density of each CMYK ink color into a recording dot ratio. Specifically, for example, the density of 256 gradations for each color is converted into a dot ratio of 1024 gradations for each color. This output characteristic conversion process uses a one-dimensional lookup table (Table3[4]
[0256] ) in which the following recording dot ratios (Cout, Mout, Yout, Kout) corresponding to the density of each ink color are set. Cout = Table3[0][C] Mout = Table3[1][M] Yout = Table3[2][Y] Kout = Table3[3][K] As with the color conversion and color separation processes described above, known techniques for reducing table size may be used.
[0034] (Quantization process) Quantization is the process of converting the recording dot ratios (Cout, Mout, Yout, Kout) for each ink color described above into the following quantization values (Cdot, Mdot, Ydot, Kdot) that represent the on or off state of the recording dots in each pixel. Cdot = Halftone[Cout][x][y] Mdot = Halftone[Mout][x][y] Ydot = Halftone[Yout][x][y] Kdot = Halftone[Kout][x][y] The above values represent the quantized values in the dithering method. By comparing these values with thresholds in the dither matrix corresponding to each pixel position, values representing the on or off state of each ink color recording dot can be obtained. Here, the occurrence probabilities for each recording dot are Cout / 1023, Mout / 1023, Yout / 1023, and Kout / 1023. Note that the quantization method is not limited to dithering; other methods such as error diffusion can also be used.
[0035] By executing each of the processes described above in order, a print image is generated from the multiplexed image. Return to the explanation of the flowchart in Figure 2.
[0036] In S206, the print image data generated in S205 is sent to the MFP40 or another printer (not shown) along with the print command, and the print process is executed at the destination printer.
[0037] The above describes the process until the original printed document is generated. In this embodiment, the B value of the RGB values was modulated in the multiplexing process (S204), but modulation may also be performed on the CMYK values. In this case, the pixel values of the white area of the paper will be Y=0, M=0, C=0, K=0, so a positive value must be used for modulation. In the case of the masks illustrated in Figures 4 and 6 above, the sign of the modulation value in the mask should be reversed, that is, "-10" should be changed to "+10" and "+10" to "-10". When modulation is performed on the CMYK values in this way, the controllability when limiting the ink applied to the white area of the paper to only Y ink is increased. On the other hand, when modulation is performed on the RGB values, the controllability of suppressing hue fluctuations when embedding in the image drawing area is increased. Therefore, it is preferable to select a suitable modulation method according to the characteristics of the recording process such as electrophotography and inkjet, and the ratio of white area, text, and image area in the page image to be printed.
[0038] <Copying process for the original printed document> Next, we will explain the process of copying the "original printed document," which is the result of the printing process of the original document described above, using the MFP40, with reference to the flowchart in Figure 8. The series of processes shown in the flowchart in Figure 8 are started when the user places the original printed document on the unillustrated document tray via the MFP40's UI (user interface) and issues a copy command. In the following explanation, the symbol "S" represents a step.
[0039] First, in the S811, the printed document to be copied, which is set in the MFP40, is read by the built-in scanner unit (not shown). In this document reading process, LED light is shone onto the document placed on the platen, and the reflected light is converted into an analog electrical signal by an image sensor such as a CCD facing each pixel.
[0040] In the next step, S812, the analog electrical signal obtained in S811 is digitized to acquire a bitmap image in the RGB color space. At this time, image processing such as MTF correction, input correction, shading correction, and color conversion is performed on each pixel of the bitmap image. A brief explanation of each process follows.
[0041] ≪MTF Correction Processing≫ MTF (Modulation Transfer Function) correction processing is a correction process related to resolution in the scanning performance of the scanner unit. When an image is optically read by the scanner unit, the image may become blurred due to deviations from the focus position or the performance limitations of the lens itself, so some degree of restoration is performed by filtering. However, if the enhancement processing is strong enough to completely restore the image, image defects such as blown-out highlights, image noise, and emphasis on dust pixels will become more noticeable, so it is necessary to design the filter strength to balance image quality improvement with the detrimental effects. The following is an example of an edge enhancement filter that multiplies the central part of the image by 5 and multiplies the pixel values on the top, bottom, left, and right by -1. R'[x][y] = R[x][y]×5-R[x-1][y]-R[x+1][y]-R[x][y-1]-R[x][y+1] G'[x][y] = G[x][y]×5-G[x-1][y]-G[x+1][y]-G[x][y-1]-G[x][y+1] B'[x][y] = B[x][y]×5-B[x-1][y]-B[x+1][y]-B[x][y-1]-B[x][y+1]
[0042] (Input correction processing) Input correction processing is a process that converts the output value of a CCD (image sensor), which is originally the amount of photons, into a brightness that matches the sensitivity of the human eye. For example, this converts the R'G'B' signals of 4096 gradations for each color into color intensity values (R", G", B) of 1024 gradations for each color. For this conversion, the following one-dimensional lookup table (Table 4[4]
[4096] ) is used, which sets the optimal recording dot ratio for the density of each ink color. R” = Table4[0][R'] G” = Table4[1][G'] B” = Table4[2][B'] However, known techniques for reducing the table size may be used, such as reducing the number of LUT grids from 4096 grids to, for example, 256 grids, and interpolating table values between grids to determine the output value.
[0043] (Shading correction processing) Shading correction processing is a process that reduces color and density unevenness caused by differences in reading sensitivity at each pixel position due to manufacturing and assembly variations in components such as lenses, LEDs, and CCDs that make up the scanner device. For example, the 1024-level R"G"B" signals for each color are converted into 256-level color intensity values (R"', G"', B"') for each color. For this conversion, the following one-dimensional lookup table (Table 5[x][3]
[1024] ) is used, which defines the density adjustment value for each pixel position in the X direction (the direction in which the scanner lens is positioned). R"' = Table5[x][0][R"] G"' = Table5[x][1][G"] B"' = Table5[x][2][B"] As with the input correction process described above, known techniques for reducing the table size may also be used.
[0044] (Color conversion process) The 256-level color intensity values (R"', G"', B"') for each color calculated through the processes up to this point are, contrary to printing, values specific to the scanner unit. The color conversion process here is the process of converting the values specific to the scanner unit into RGB values (Rout, Gout, Bout values) suitable for display on the display. Currently, the input values R"', G"', and B"' each have 256 levels. Therefore, for this conversion, the following three-dimensional lookup table (Table6
[0256]
[0256]
[0256] [3]), which has a total of 16,777,216 output value pairs (256 x 256 x 256), is used. Rout = Table1[R”'][G”'][B”'][0] Gout = Table1[R”'][G”'][B”'][1] Bout = Table1[R”'][G”'][B”'][2] As with the input correction and shading correction processes described above, known techniques for reducing the table size may be used. In this way, a Bitmap image is obtained.
[0045] In the subsequent S813, a process is executed to extract document ID information from the Bitmap image acquired in S812. Specifically, it is determined whether the aforementioned patterns representing "0" or "1" are formed within the Bitmap image in predetermined areas (in this case, 8x8 pixels), and a process is performed to extract numerical information represented by "0" and "1". By repeatedly applying this to the entire Bitmap image, the document ID information embedded by the multiplexing process during the printing of the original document is decoded. Information embedded by the multiplexing process is sometimes called "embedded information" or "multiplexed information". Here, the decoded information of embedded information will be explained in detail.
[0046] (Decryption of embedded information) First, the location where embedded information is embedded within the acquired Bitmap image is detected. The embedding location can be detected by analyzing the spatial frequency characteristics of each unit region (in this case, an 8x8 pixel region). Figure 9 shows the spatial frequency characteristics of the pattern used for embedding. The horizontal axis represents the horizontal frequency, and the vertical axis represents the vertical frequency, with higher frequencies occurring as you move away from the origin. In this embodiment, two types of patterns corresponding to "0" and "1" are embedded in the image (see Figures 5 and 7 above). At that time, a value of "10" is subtracted from the B component of each RGB color component (in the case of the mask in Figure 4; in the case of the mask in Figure 6, addition and subtraction are performed). As a result, for example, the downward-sloping pattern formed in the image by the mask in Figure 4(a) generates a large power spectrum at line A02. Similarly, the downward-sloping pattern formed in the image by the mask in Figure 4(b) generates a large power spectrum at line A01. Therefore, by detecting this power spectrum, data of "0" or "1" can be extracted. Furthermore, it is possible to enhance the power spectrum by performing edge detection as a preprocessing step before power spectrum detection.
[0047] In the frequency analysis-based data extraction described above, accurate extraction of the analysis area from the image data is necessary, so a process to correct for coordinate position shifts is also performed. For example, first, the extraction of a unit area from the bitmap image and frequency analysis are repeated vertically and horizontally, shifting by one pixel at a time. If the image size is 600 pixels wide and 900 pixels high, this is repeated a total of 64 times. The point where the spectrum is strongest is then used as the reference position for extraction. By extracting embedded information based on this reference position, the embedded sequence of "0" or "1" can be obtained with high accuracy.
[0048] In this embodiment, as explained in S204, the embedded information to be embedded is treated as text data, and its character code is converted to a numerical value using "Shift JIS". In this case, in Shift JIS 1-byte code (half-width characters), for example, the alphabet "h" corresponds to the binary data "01101000", "e" to "01100101", "l" to "01101100", and "o" to "01101111". Therefore, if the extracted sequence of embedded information is "0110100001100101011011000110110001101111", the English word string "hello" is obtained. In this way, the document ID information embedded as embedded information is extracted.
[0049] Let's return to the explanation of the flowchart in Figure 8.
[0050] In the following step S814, it is determined whether the extraction of document ID information in S813 was successful. If the extraction of document ID information was successful, the process proceeds to S815; otherwise, it proceeds to S820. There are two possible cases in which the extraction fails. One is that the document ID information was not embedded in the printed material to be copied in the first place (Possibility 1). The other is that the document ID information was embedded in the printed material to be copied, but the printed material was dirty or had handwritten text added to it later, making it impossible to correctly detect the predetermined pattern representing the embedded information (Possibility 2). In the case of Possibility 1, the process simply proceeds to S820. On the other hand, in the case of Possibility 2, the user may be notified by displaying a message such as "You are trying to copy a genuine document (original printed material) with a document ID embedded in it." This allows the user to know that they were trying to copy a printed material that may not be genuine, and gives them an opportunity to choose to cancel the copying process. For example, if the result obtained from the extraction process is between 1 and 31 bits of the total 32-bit document ID information, it can be determined to be possibility 2, and such a notification can be issued. It should be noted that it is not entirely impossible for only one pattern similar to the predetermined pattern to be included in the image by chance. Considering such rare cases, it is preferable to determine it as possibility 1 if less than 16 bits (half of 32 bits) are extracted, and as possibility 2 if 16 bits or more and 31 bits are extracted.
[0051] In the subsequent S815, the document ID information that was successfully extracted is verified. In this embodiment, similar to the document ID acquisition process in S202, the MFP40 accesses an external PC to request verification and obtains the verification result. Specifically, a verification request to confirm whether the extracted document ID information is officially registered and valid is sent via a network interface (not shown), and the verification result is received and obtained from the external PC. If the MFP40 has sufficient resources and can manage the document ID information within the MFP40, it is also possible to perform the verification process internally.
[0052] Next, in S816, the process is divided based on whether the document ID information extracted in S813 is officially registered and valid, as a result of the above verification. If the document ID is valid, the process proceeds to S817; if the document ID is invalid, the process proceeds to S820. Here, if the document ID is invalid, the user may be notified by displaying a message such as "You are attempting to copy a non-genuine document with an invalid document ID." This gives the user an opportunity to choose to cancel the copying process, etc.
[0053] In S817, a check is performed to determine whether the printed material to be copied has been tampered with (tampering check). The general flow of this tampering check is as follows: First, the MFP40 sends a document data acquisition request, along with document ID information, to an external PC via the network I / F17. Then, the MFP40 receives and acquires the document data (PDL data) associated with the document ID information from the external PC and performs rendering processing on the document data. This completes the preprocessing. Next, the Bitmap image obtained from the rendering process is compared with the Bitmap image acquired in S812 to determine whether or not it has been tampered with. Here, the tampering check process will be explained in detail according to the separate flowchart shown in Figure 10. In the explanation of the flow in Figure 10, the Bitmap image obtained from the rendering process will be referred to as the "rendered image," and the Bitmap image obtained from the original document reading process will be referred to as the "scanned image."
[0054] <Details of the tampering check process> In S1001, the counter that counts the pixels determined to have been tampered with in S1010 (described later) is initialized. Specifically, the counter's count (number of tampered pixels) is set to "0".
[0055] In the next step, S1002, normalization processing is performed on the scanned image acquired in S812. This normalization processing is done to match the dynamic range between the scanned image and the rendered image. For example, the brightest part of the scanned image is generally the color of the paper of the printed material being scanned, and in principle, it has a value with some density. On the other hand, the brightest part of the rendered image is a pixel with a pixel value of R=255, G=255, B=255. Also, the darkest part of the scanned image is generally the black of the ink or toner, and in principle, it has a value with brightness due to some reflected light. On the other hand, the brightest part of the rendered image is a pixel with a pixel value of R=0, G=0, B=0. Thus, there is inherently a difference between the brightest and darkest colors of the two. Furthermore, if the printed material is output in color, there is also a difference in the hue of the colors between the two. For example, even for the most vivid red that can be printed, the saturation in the scanned image is lower compared to the pixel value that shows the most vivid red in the rendered image (R=255, G=0, B=0). Therefore, each pixel value (RGB value) of the scanned image is normalized using the following equations (1) to (3) to obtain new pixel values (Rnorm, Gnorm, Bnorm). Rnorm = (R - Rd) / (Rw-Rd)×255 ...Equation (1) Gnorm = (G - Gd) / (Gw-Gd)×255...Equation (2) Bnorm = (B - Bd) / (Bw-Bd)×255...Equation (3) By normalizing using the above equations (1) to (3), the RGB values of the brightest color (white) in the scanned image become R=255, G=255, B=255, and the RGB values of the darkest color (black) become R=0, G=0, B=0.
[0056] In the next step, S1003, filtering is performed on the scanned image after normalization. This is done to enhance edges more strongly in order to make it easier to compare with the rendered image.
[0057] In the subsequent S1004, a process is performed to remove the image patterns representing embedded information from the filtered scanned image. The printed image generated in S205 in the flow shown in Figure 2 above differs from the original document image (the image of the original electronic document) due to the multiplexing process (S203). Therefore, this process is performed to remove this difference as much as possible. Specifically, the pattern data embedded by the multiplexing process is subtracted from the scanned image data. This brings the scanned image closer to the state before the multiplexing process was applied. In this embodiment, this is achieved by adding a value obtained by multiplying each value in the mask shown in Figures 4 and 6 above by "-1" to each pixel of the scanned image.
[0058] Next, in S1005, a comparison process is performed between the scanned image from which the embedded information has been removed and the rendered image. Specifically, the rendered image and the scanned image after the embedded information has been removed are compared pixel by pixel, and the difference values (ΔR, ΔG, ΔB) expressed by the following equations (4) to (6) are calculated. ΔR = |R[x][y] of the rendered image - R[x][y] of the scanned image| ... Equation (4) ΔG = |G[x][y] of the rendered image - G[x][y] of the scanned image| ... Equation (5) ΔB = |B[x][y] of the rendered image - B[x][y] of the scanned image| ... Equation (6)
[0059] Next, in S1006, it is determined whether the difference values (ΔR, ΔG, ΔB) obtained in S1005 exceed the threshold. In this embodiment, the values are compared with pre-defined thresholds (Rth, Gth, Bth) for each RGB channel. If any of ΔR, ΔG, or ΔB exceeds the corresponding threshold, the process proceeds to S1007; otherwise, it proceeds to S1008. The thresholds are determined considering the characteristics of the scanner unit and printing device, and in this embodiment, for example, values such as Rth=Gth=Bth=64 can be set.
[0060] In S1007, the counter indicating the number of altered pixels is incremented (+1). After the counter is incremented, the process proceeds to S1008. In S1008, it is determined whether the comparison of pixel values has been completed for all pixels. If there are unprocessed pixels, the process returns to S1005 and continues; if the comparison of pixel values has been completed for all pixels, the process proceeds to S1009.
[0061] In S1009, it is determined whether the counter value indicating the number of altered pixels exceeds a threshold. In this case, the threshold should be set to a value such as 3% of the total number of pixels, taking into account cases where dust is introduced during the acquisition of the scanned image. If the count value exceeds the threshold, the process proceeds to S1010; otherwise, it proceeds to S1011.
[0062] In S1010, the tampering check is determined to be unaltered, and the process is exited. On the other hand, in S1011, the tampering check is determined to be unaltered, and the process is exited.
[0063] The above describes the content of the tamper-proofing process. In this embodiment, the comparison is performed at the pixel level, but it is not limited to this, and the process may be performed on a larger scale, for example, by comparing the average value at multiple pixel blocks. Once the tamper-proofing process is complete, the process proceeds to S818 in Figure 8.
[0064] Let's return to the explanation of the flowchart in Figure 8.
[0065] In S818, the bitmap image obtained in S812 is sorted for processing according to the result of the tampering check. If no tampering is found, the process proceeds to S819; if tampering is found, the process proceeds to S820. If the tampering check results in tampering, the user may be notified through a message such as "You are attempting to copy a tampered document." This gives the user an opportunity to choose to cancel the copying process, etc.
[0066] Next, in S819, information indicating that it is a copy of the "original printed document" is added to the Bitmap image obtained in S812. By applying an embedding method that is more visible than embedding document ID information (for example, modulating the RGB values), it becomes easy to recognize that it is a copy and not the "original printed document." In addition, the user may be notified through a message indicating that "you are trying to copy the original printed document" during this addition process. This gives the user an opportunity to choose to cancel the copying operation, etc.
[0067] <Highlight the embedded pattern in the scanned image> Bitmaps acquired with S812 suffer from degradation of the embedded pattern due to printing and scanning. Figure 11(a) simply illustrates the pattern degradation. Mask 1100 is the extracted pattern. Region 1101 is a part of the embedded pattern. Region 1102 is also a part of the embedded pattern, and compared to region 1101, the difference between the pixel values and those outside the embedded pattern is smaller, indicating weaker embedding strength.
[0068] If the embedded pattern of the scanned image is like mask 1100, it can be determined that the extracted embedded pattern is weak. Similarly, if error correction is performed when extracting document ID information in S813 and the pattern is corrected, it can also be determined that the pattern is weak. When the embedded pattern is weak, the embedded pattern in the scanned image is enhanced. Mask 1103 in Figure 11(b) is an enhancement mask created based on pixels with weak embedded patterns, based on Figure 4(a).
[0069] Region 1104 sets the intensity to be added to pixels with weak embedding patterns. The intensity may be set from the difference between regions 1101 and 1102, or it may be set uniformly from a predetermined value. Multiplexing processing is performed on the extracted text ID information in the same way as in S204. At this time, for regions where the embedding pattern is judged to be weak, an enhancement mask is added to the mask that creates the variation shown in Figure 4, and multiplexing processing is performed. In addition, the embedded pattern can be enhanced by generating a pattern based on the created enhancement mask and enhancing the scanned image based on that pattern.
[0070] <Reduce the embedded patterns in scanned images and superimpose new embedded patterns.> Alternatively, the embedded pattern in the scanned image may be reduced, and the multiplexing process may be performed again. One method for reducing the embedded pattern is to apply a smoothing filter of a predetermined size to the scanned image. The filter size may be set based on the size of the embedded pattern (8x8 in this case). Alternatively, an inverse filter may be calculated from the embedded pattern and used. If the pattern embedding area is a white area, the pixel values may be changed to the white area by removing the background or other methods. If the embedded pattern is attached to a specific color plane, the above reduction process may be performed only on that specific color.
[0071] For images with reduced embedding patterns, a multiplexing process is performed to embed the extracted text ID information, similar to S204.
[0072] In the next step, S820, a print image is generated based on the Bitmap image acquired in S812 or the Bitmap image that has undergone additional processing in S819. This process corresponds to S205 in the flow chart of Figure 2 mentioned above, and there are no particular differences, so the explanation is omitted. In the following step, S821, printing is performed using the print image generated in S820.
[0073] The above describes the processing flow when copying an original printed document. Alternatively, instead of displaying messages or other notification processes in S814, S816, S818, and S819, control may be implemented to automatically cancel the copying process at each point. This eliminates the need to repeatedly confirm the user's intentions.
[0074] Furthermore, the information added in S819 indicating that it is a copy can be said to be "information indicating that the content is genuine and a copy," guaranteeing that the original has not been tampered with. Therefore, it may be possible to add string information that more clearly expresses that the printed material being copied is an authentic document, such as "copy of the original printed material." Moreover, even if tampering is detected in S818, it may be possible to add text such as "illegible (tampered) copy" before proceeding to the generation of the printed image (S820).
[0075] Alternatively, the authentic document ID information extracted from the "original printed document" to be copied may be updated with document ID information indicating that it is a copy, embedded again in a Bitmap image, and then printed out by S820. This makes it possible to notify the user if they attempt to copy the printed document with the updated document ID information embedded, which is output by S820. In this case, the document ID information indicating that it is a copy can be acquired and stored together with the document ID information obtained during the printing process of the original (S202) in a format similar to reservation ID information. This makes it possible to notify the user that it is a copy without having to perform verification on an external PC.
[0076] As described above, according to this embodiment, when a "printed original document" is copied, information indicating that it is a copy is added as highly visible information, so that the user can easily recognize that it is a "copy of the original document". Furthermore, when a user attempts to make another copy of the "copy of the original document" obtained through the above copying process, the copier can detect additional information indicating that it is a copy and notify the user of that fact.
[0077] <Example 1> In the above-described embodiment, an example was explained in which document ID information is multiplexed and embedded in 8x8 pixel units. In this method, a predetermined pattern is formed in the high-frequency region of the image, making it possible to embed information in a way that is difficult for the user to see. However, the method of embedding information is not limited to this; for example, document ID information may be converted into a two-dimensional code and embedded in a way that is difficult to see.
[0078] Figure 12(a) shows an example of a two-dimensional code, and Figure 12(b) shows the corresponding actual print pattern. In the example in Figure 12(b), the pattern forms one dot for every 8x8 pixel area. The dot corresponding to the black pixel 1201 in the two-dimensional code in Figure 12(a) corresponds to one dot 1202 in the print pattern shown in Figure 12(b). In other words, no dots in the print pattern are formed at the positions corresponding to the white pixels in the two-dimensional code in Figure 12(a). The specific processing flow to achieve this is as follows: In the multiplexing process (S204) in the flow of Figure 2 described above, the document ID information is first converted into a two-dimensional code, and then converted into pattern data of spaced-out dots and embedded in the rendering image in predetermined area units. Then, in the next print image generation process (S205), the spaced-out dots are formed using only Y ink, making them even less visible.
[0079] Furthermore, in the process of copying printed materials to which document ID information is embedded using this modified method, the document ID information can be extracted by extracting a two-dimensional code pattern formed of 8x8 pixel yellow dots from the read bitmap image and then decoding it.
[0080] Alternatively, as an embedding method other than two-dimensional codes, for example, threshold modulation may be performed during the quantization process for multiplexing.
[0081] <Modification 2> Up to this point, the explanation has assumed that document ID information indicating the authenticity of the original document is embedded in a manner that makes it as difficult as possible for the user to see during the printing process. However, it is also acceptable to embed document ID information in a manner that is easily visible to the user (for example, modulating the R and G values, which have large variations in the density of each CMK ink).
[0082] In this case, the copying process may involve printing either the rendered image generated during the preprocessing of the tamper-evident check process (S817) or printing the Bitmap image (S1004) from which the embedded data has been removed. Printing using either of these methods will produce a printed document without embedded document ID information. In this case, from the user's perspective, since no visible pattern data is recorded in the white areas of the printed document obtained by copying, it is easy to recognize that it is not the "original printed document." Furthermore, since the copied printed document does not contain embedded data, if it is to be copied again, it will always be determined as No in S814.
[0083] The method described above involves extracting document ID information from printed documents using a scanner, but information extraction can also be performed using a mobile device such as a smartphone or tablet. With a mobile device, it is possible to photograph the printed document using a camera and then extract the document ID information from the captured image within the mobile device.
[0084] The following describes the process of extracting document ID information by taking a picture of a printed document using the camera function of a mobile device and extracting document ID information from the acquired image. When extracting document ID information from an inkjet printed document, brightening the document during shooting improves the accuracy of extraction from dark areas. Dark areas are regions in the printed document where the color density is high, and this particularly includes low-luminosity areas such as black.
[0085] On the other hand, when photographing a document printed on plain paper using the electrophotographic method, the dark areas may be strongly reflected by the toner wax when the flash is ON, causing overexposure. Therefore, when extracting document ID information, the image quality of the captured image may deteriorate due to overexposure, reducing the extraction accuracy. To avoid this, this embodiment describes a method in which the printed document is photographed with the flash ON and OFF in the electrophotographic method, and the image from which document ID information is extracted is switched depending on whether or not the printed document contains dark areas.
[0086] Figure 13 is a block diagram showing the system configuration of a mobile terminal. The CPU 1301 executes processing according to the program stored in the ROM 1302 or RAM 1303. RAM 1303 is volatile storage that temporarily holds programs or data. ROM 1302 is non-volatile storage that also holds programs or data. The NETORK I / F (Network Interface) 1304 connects the mobile terminal 130 to an external network and communicates with one or more external PCs to perform tasks such as requesting document ID matching, requesting results, and requesting document data. The DISPLAY I / F (Display Interface) inputs user operations via display and touch panel operation. The SENSOR 1306 is a sensor that measures the distance to the subject and the ambient brightness. The CAMERA 1307 adjusts the exposure and other settings based on the terminal and shooting conditions measured by the SENSOR 1306 to photograph the subject. FLASH1308 automatically emits a flash when the light intensity measured by SENSOR1306 is low, during shooting with CAMERA1307. Alternatively, the flash can be forcibly switched ON or OFF through user settings or application settings. This concludes the explanation of the mobile device's system configuration.
[0087] Figure 14 shows the flow for extracting document ID information from an electrophotographically printed document. Figure 14 illustrates how the document reading and document ID information extraction process described in S811 to S813 of Figure 8 is performed using the CAMERA 1307 of the mobile terminal 130. In reality, the mobile terminal 130 performs the processes from S811 to S821 in Figure 8, but the processes from S814 onward are the same as when using the PC 20, so the explanation is omitted in this embodiment. The processing in each step of Figure 14 is performed by the CPU 1301 of the mobile terminal 130, which loads the program code stored in the ROM 1302 into the RAM 1303 and executes it.
[0088] In S1401, the mobile terminal 130 turns on the flash and takes a picture of the document at the user's command. Alternatively, the flash can be switched on or off by the mobile terminal 130 notifying the user that it will turn the flash on or off, and the user granting permission. The mobile terminal 130 may also be configured to automatically switch the flash on or off after a single command from the user. Next, in S1402, the mobile terminal 130 digitizes the analog electrical signal acquired in S1401 and obtains a bitmap image.
[0089] Next, in S1403, the mobile terminal 130 turns off the flash at the user's command and takes a picture of the document. In S1404, the mobile terminal 130 digitizes the analog electrical signal acquired in S1403 and obtains a bitmap image. In S1405, the mobile terminal 130 determines whether or not the document contains dark areas based on the acquired bitmap image. If there are no dark areas, the process proceeds to S1406; if there are dark areas, the process proceeds to S1407.
[0090] Figure 15 illustrates the dark areas. Document 1501 shows a document containing dark areas and includes a text area 1502, a rectangular drawing area 1503, and an image area 1504. The rectangular drawing area 1503 is the dark area and is a solid black rectangular area. On the other hand, document 1505 shows a document without dark areas and includes a text area 1506.
[0091] In S1405, the determination of whether or not a dark area is present is made by calculating the average brightness from the RGB values of N×N pixels based on the bitmap image obtained by photographing the document. If the average brightness is less than a threshold, the N×N pixels are considered a dark area. The threshold is, for example, 64 / 255, which is equivalent to 1 / 4 of the brightness range. This is performed on the rectangular drawing area 1503 and the image area 1504. For the sake of explanation, if even one dark area exists, it is determined that a dark area is present.
[0092] Returning to the explanation of Figure 14, in S1406, where it was determined that there were no dark areas, the mobile terminal 130 sets the Bitmap image captured and acquired in S1402 with the flash ON as the image for extracting text ID information.
[0093] On the other hand, in S1407, when it is determined that there is a dark area, the mobile terminal 130 sets the Bitmap image captured with the flash OFF in S1404 as the image for extracting text ID information. In S1408, the mobile terminal 130 extracts text ID information from the set Bitmap image. Subsequent processing is performed by the mobile terminal 130 in the same way as in S814 to S821 in Figure 8.
[0094] As described above, this embodiment improves the accuracy of information extraction. Specifically, the printed document is photographed with the flash ON and OFF, and the image from which the document ID information is extracted is switched depending on whether or not the document contains dark areas. This makes it possible to extract information with high accuracy even when the document printed using the electrophotographic method contains dark areas.
[0095] In this embodiment, the example of turning the flash ON and OFF was used, but multiple shots or scans may be taken with different light intensity settings. Alternatively, instead of a mobile device, a scanner may be used to perform multiple scans with different light intensity settings, and the images from which the document ID information is extracted may be switched using the same process.
[0096] <<Embodiment 2>> Scanning printed documents using mobile devices can be performed not only on documents printed using electrophotography but also on documents printed using, for example, inkjet technology. However, because the characteristics of printed documents differ depending on the printing method, using the same scanning method for both electrophotographic and inkjet documents may result in image quality degradation and a decrease in extraction accuracy. Therefore, to maintain high extraction accuracy, it is necessary for the mobile device performing the scanning to process the document appropriately according to its printing method.
[0097] For example, with documents printed on plain paper using an inkjet method, turning on the flash can improve the accuracy of extracting text ID information from dark areas. On the other hand, with documents printed using an electrophotographic method, turning on the flash can cause overexposure in dark areas, potentially reducing extraction accuracy. In other words, with the inkjet method, shooting with the flash on results in higher extraction accuracy compared to shooting with the flash off, so it is preferable to shoot with the flash on. On the other hand, with the electrophotographic method, shooting with the flash on reduces extraction accuracy in dark areas, so it is preferable to shoot with the flash off. In this embodiment, a method for switching the control during image acquisition by specifying the printing method used to print the document will be described.
[0098] Figure 16 shows a flowchart illustrating the extraction of document ID information from an original document printed using electrophotographic or inkjet methods, using a mobile terminal 130. The flowchart in Figure 16, like that in Figure 14, performs the document reading and document ID information extraction process described in steps S811-S813 of Figure 8, but using the CAMERA 1307 of the mobile terminal 130. While the mobile terminal 130 actually performs the processes from S811-S821 in Figure 8, the processes from S814 onward are the same as those performed using the PC 20, and therefore are omitted from this embodiment. The processing in each step of Figure 16 is performed by the CPU 1301 of the mobile terminal 130, which loads the program code stored in the ROM 1302 into the RAM 1303 and executes it.
[0099] In S1601, the user inputs information about the printing method of the document to be photographed via the touch panel on the display 1305 of the mobile terminal 130. In this embodiment, it is assumed that either electrophotographic or inkjet printing is input.
[0100] In S1602, the mobile terminal 130 determines whether to use the flash function when scanning the document based on the printing method of the document entered by the user in S1601. If the printing method entered by the user is inkjet, the scan will be performed with the flash function, and the process proceeds to S1603. If the document is printed using the electrophotographic method, the scan will be performed without the flash function, i.e., the process proceeds to 1604.
[0101] In S1603, the user controls the mobile terminal 130 to turn on the flash and take a picture of the document. In S1604, the user controls the mobile terminal 130 to turn off the flash and take a picture of the document.
[0102] In step S1605, the mobile terminal 130 digitizes the analog electrical signal of the print original captured in the above step and obtains a bitmap image.
[0103] In step 1606, the mobile terminal 130 extracts text ID information from the Bitmap image acquired in step S1605.
[0104] The subsequent processes are the same as those shown in S814-S821 of Figure 8, so their explanation will be omitted.
[0105] As described above, according to this embodiment, by allowing the user to specify the printing method of the original document and switching the shooting control when acquiring the image from which the document ID is extracted, it is possible to avoid image quality degradation such as overexposure in dark areas when scanning a printed document. As a result, a decrease in extraction accuracy can be suppressed. In this embodiment, a method using flash ON / OFF with a mobile terminal is described, but images for extracting document ID information may also be acquired by scanning with different light intensity settings using a scanner.
[0106] <<Embodiment 3>> The following describes a method for extracting document ID information from a printed document using a mobile device, when the printed document contains a mixture of dark and highlighted areas. Highlighted areas refer to regions within the printed document where the brightness value exceeds a certain level.
[0107] When a document is printed using an inkjet method, the extraction accuracy is higher when both the dark and highlight areas are photographed with the flash ON during scanning. On the other hand, when a document is printed using an electrophotographic method, the extraction accuracy is higher when the dark areas are photographed with the flash OFF and the highlight areas with the flash ON during scanning. Therefore, for documents printed using an inkjet method, extraction is performed from the image photographed with the flash ON, and for documents printed using an electrophotographic method, extraction is performed using both images photographed with the flash ON and OFF. Specifically, reading is performed from images photographed with the flash ON and OFF, and an image is created for extracting document ID information by combining the dark and highlight areas. Then, the document ID information is extracted from the created image. This embodiment will be explained in detail below.
[0108] Figure 17 shows a flow that switches the method of extracting document ID information based on the printing method specified by the user. Note that, similar to Figure 14, the flow in Figure 17 is a process that uses the CAMERA 1307 of the mobile terminal 130 to perform the document reading and document ID information extraction process described in S811-S813 of Figure 8. In practice, the mobile terminal 130 performs the processes from S811-S821 in Figure 8, but the processes from S814 onwards are the same as those when using the PC 20, so their explanation is omitted in this embodiment.
[0109] In S1701, the user inputs information about the printing method of the document to be photographed via the touch panel on the display 1305 of the mobile terminal 130. In this embodiment, it is assumed that either electrophotographic or inkjet printing is input.
[0110] In S1702, the user controls the mobile terminal 130 to turn on the flash and photograph the original document. In 1703, the mobile terminal 130 digitizes the analog electrical signal of the print original document photographed in S1702 and obtains a bitmap image.
[0111] In S1704, the mobile terminal 130 switches the method of extracting document ID information based on the printing method of the document entered by the user in S1701. If the printing method entered by the user is inkjet, proceed to S1705; if the document was printed using electrophotography, proceed to 1706.
[0112] In S1705, the mobile terminal 130 extracts document ID information from the Bitmap image of the document, which was captured with the flash ON in S1703. The extraction method in this step is referred to as "Method 1".
[0113] In S1706, the user controls the mobile terminal 130 to turn off the flash and take a picture of the document.
[0114] In S1707, the mobile terminal 130 digitizes the analog electrical signal of the print original captured in S1706 and acquires a bitmap image. For the sake of simplicity, the bitmaps captured with the flash ON and OFF are assumed to be the same size, and no positional shifts occur during capture.
[0115] In S1708, the mobile terminal 130 extracts document ID information using two images: the Bitmap image of the document taken with the flash ON, acquired in S1703, and the Bitmap image of the document taken with the flash OFF, acquired in S1707. The extraction method in this step is referred to as "Method 2". Subsequent processing is the same as that shown in S814-821 of Figure 8. The above is a description of the flow for switching the method of extracting document ID information based on the printing method specified by the user.
[0116] Here, we will explain in detail the second method, which involves comparing images with the flash ON and OFF at the pixel level, creating a composite image for extracting the text ID based on that comparison, and then extracting the text ID from the composite image.
[0117] Figure 18 shows the workflow in which the mobile terminal 130 compares flash ON and flash OFF images to create a composite image for extraction.
[0118] In S1801, the mobile terminal 130 calculates the average brightness of the entire image for both the flash ON and flash OFF images and stores the difference (Dth). In S1802, the mobile terminal 130 initializes the coordinate (x, y) values for comparing the flash ON and flash OFF images pixel by pixel and specifies this as the initial coordinate. In S1803, the mobile terminal 130 calculates the difference D(x, y) between the brightness value of the flash ON image and the brightness value of the flash OFF image at the coordinate specified in S1802 (for example, (0,0)).
[0119] In S1804, the mobile terminal 130 determines whether the difference value D(x, y) calculated in S1803 is greater than the average luminance difference Dth calculated in S1801. If the determination is YES (i.e., D(x, y)>Dth), the process proceeds to S1805. If the determination is NO (i.e., D(x, y)<Dth), the process proceeds to S1806.
[0120] In S1805, the mobile terminal 130 sets the pixel value of the flash OFF as the pixel value of the composite image at the specified coordinates. That is, since the difference in luminance between flash ON and OFF at the specified coordinates is greater than the average luminance difference, it is determined that white jumping occurs at the specified coordinate position, and at this coordinate, a flash OFF image with less image quality degradation is used for the composite image.
[0121] In S1806, the mobile terminal 130 sets the pixel value of the flash ON as the pixel value of the composite image at the specified coordinates. That is, since the difference in luminance between flash ON and OFF is equal to or less than the average luminance difference, it is determined that white jumping does not occur at the specified coordinate position, and at this coordinate, a flash ON image with high extraction accuracy is used for the composite image.
[0122] In S1807, the mobile terminal 130 determines whether the generation of the composite image has ended for all pixels. If YES, the process proceeds to S1809. If NO, the process proceeds to S1808. In S1808, the position of the specified coordinates for generating the composite image is updated. After the update, the process returns to S1803 and the same processing is repeated.
[0123] In S1809, the mobile terminal 130 performs text ID information extraction processing using the composite image created by comparing the luminances of all pixels of the printed document.
[0124] The above is an explanation of a method for comparing the images of flash ON and OFF for each pixel, creating a composite image for text ID extraction based on this comparison, and extracting the text ID from the composite image. By determining whether the difference at the specified coordinates is greater than the average luminance, local image quality degradation factors including white jumping can be excluded, and an improvement in extraction accuracy can be expected.
[0125] Here, we explained how to switch between flash ON and OFF images when creating a composite image, but it is also possible to determine whether each pixel is a dark area or not and switch between flash ON and OFF images accordingly. By doing so, it is possible to limit the use of flash OFF images to dark areas and increase the area where flash ON images are used, which is expected to further improve extraction accuracy.
[0126] As described above, according to this embodiment, when a printed document contains both dark and highlighted areas, the method of extracting document ID information can be switched according to the printing method specified by the user. In particular, when extracting document IDs from a document printed using the electrophotographic method, by detecting overexposure and combining flash ON and OFF images for extraction, the area of the flash OFF image can be limited. As a result, a decrease in extraction accuracy can be suppressed. Although this description describes a method using flash ON and OFF with a mobile device, it is also possible to obtain images from which document ID information can be extracted by scanning with different light intensity settings using a scanner.
[0127] <<Embodiment 4>> In this embodiment, we will describe a method for determining the printing method from the scanned print document, rather than through user manual settings. The method to be determined will be either inkjet or electrophotographic.
[0128] Figure 19 shows a flow in which images are taken with the mobile terminal with the flash ON and OFF, the printing method is determined using the images, and the method for extracting document ID information is switched. Note that the flow in Figure 19 is the same as in Figure 14, where the document reading and document ID information extraction process described in S811 to S813 of Figure 8 is performed using the CAMERA 1307 of the mobile terminal 130. In reality, the mobile terminal 130 performs the processes from S811 to S821 in Figure 8, but the processes from S814 onwards are the same as the processes when using the PC 20, so the explanation is omitted in this embodiment.
[0129] In S1901, the user controls the mobile terminal 130 to turn on the flash and photograph the original document. In S1902, the mobile terminal 130 digitizes the analog signal of the print original document photographed in S1901 and obtains a bitmap image with the flash on.
[0130] In S1903, the user controls the mobile terminal 130 to turn off the flash and photograph the document. In S1904, the mobile terminal 130 digitizes the analog signal of the print document photographed in S1903 and obtains a bitmap image with the flash off. For the sake of simplicity, it is assumed that the bitmaps taken with the flash on and off are the same size, and that no positional shifts occur during shooting.
[0131] In S1905, the mobile terminal 130 uses the flash ON / OFF images acquired in S1902 and S1904 to determine the printing method of the document. The determination method will be explained in detail below with reference to Figure 20.
[0132] Figure 20 shows the flow for determining the printing method using images with the flash ON and OFF. The printing method is determined by the fact that when an electrophotographic document is scanned with the flash ON, overexposure occurs. If overexposure occurs, it is determined to be an electrophotographic document; if no overexposure occurs, it is determined to be an inkjet document.
[0133] In S2001, the mobile terminal 130 calculates the average brightness of the entire image for both the flash ON and flash OFF images and stores the difference (Dth). In S2002, the mobile terminal 130 resets the counter for counting the number of pixels that have been determined to be overexposed. It also initializes the coordinate (x, y) values for comparing the flash ON and flash OFF images pixel by pixel and sets these as the initial coordinates.
[0134] In S2003, the mobile terminal 130 calculates the difference D(x, y) between the luminance value of the flash ON image and the luminance value of the flash OFF image at the specified coordinates. In S2004, the mobile terminal 130 determines whether the difference value D(x, y) calculated in S2003 is greater than the average luminance difference Dth calculated in S2001. If the determination is YES (i.e., D(x, y) > Dth), it proceeds to S2005. If the determination is NO (i.e., D(x, y) < Dth), it proceeds to S2006.
[0135] In S2005, the mobile terminal 130 increments the count by one. This is because the difference between flash ON and OFF is greater than the average luminance difference, so it is determined that white bleeding occurs at the specified coordinate position. In S2006, the mobile terminal 130 determines whether the generation of the composite image has ended for all pixels. If YES, it proceeds to S2008; if NO, it proceeds to S2007.
[0136] In S2007, the mobile terminal 130 updates the position of the specified coordinates for which the white bleeding determination is made. After the update, it returns to S2003 and repeats the same process. In S2008, the mobile terminal 130 determines whether the number of white bleeding pixels is equal to or greater than a predetermined threshold TH using the result of counting white bleeding for all pixels. The predetermined threshold TH is set relatively from the number of pixels in the dark part of the printed manuscript. For the dark part of the printed manuscript, the shadow point is set from the luminance histogram of the flash OFF image, and the number of pixels with luminance values below the shadow point is counted. A predetermined ratio may be multiplied by the obtained number of pixels in the dark part to provide a margin. If the determination result is YES (i.e., white bleeding counter > TH), it proceeds to S2009; if NO (i.e., white bleeding counter < TH), it proceeds to S2010.
[0137] In S2009, since there is a high possibility that white bleeding has occurred, the mobile terminal 130 determines that the printed manuscript was printed by an electrophotographic method and returns to the flow of FIG. 19.
[0138] In S2010, since the possibility of overexposure is low, it is determined that the print document was printed using an inkjet method, and the process returns to the flow shown in Figure 19. This concludes the explanation of the flow for determining the printing method using images with and without flash.
[0139] Returning to the explanation of Figure 19, in S1906, the mobile terminal 130 switches processing based on the printing method of the document determined in S1905. If the printing method determined in S1905 is inkjet, the process proceeds to S1907; if it is electrophotographic, the process proceeds to S1908.
[0140] In S1907, the mobile terminal 130 extracts document ID information using "Method 1" as described in Embodiment 3. On the other hand, in S1908, the mobile terminal 130 extracts document ID information using "Method 2" as described in Embodiment 3.
[0141] The above describes the flow for taking images with the flash ON / OFF, determining the printing method using those images, and switching the method for extracting document ID information. According to this embodiment, the mobile terminal 130, rather than the user, determines the printing method of the print document when reading the document ID information, thus saving the user time. Furthermore, since the determination is made mechanically using the flowchart in Figure 20 rather than by user judgment, the accuracy of document ID information extraction can be improved.
[0142] In this embodiment, an example is shown where the printing method of the original document is determined by comparing images with the flash ON and OFF, but this is not limited to this. The screen rulings included in the captured image may be analyzed using pattern analysis, and if a screen is confirmed, it may be determined to be an electrophotographic method. Alternatively, the inkjet and electrophotographic methods may be determined from the contrast ratio of white paper and black text that has been determined in advance. Furthermore, even in the case of an inkjet method, if the plain paper is pre-coated with clear ink, it may be determined to be an electrophotographic method using the method described above, and extraction processing may be performed based on that. However, since this may cause overexposure when the flash is ON due to the coating, in that case, extraction processing using an image captured with the flash OFF, similar to the electrophotographic method, is preferable.
[0143] <<Other Embodiments>> In the embodiments described above, the reading of the printed document and the extraction of document ID information were performed using a mobile terminal, but the invention is not limited to this. For example, a device equipped with a shooting function or a copying function, such as a digital camera or copier, may be used. Furthermore, regarding the flash function used when reading with the mobile terminal, other methods of irradiating the printed document with light may be used.
[0144] Furthermore, in the embodiments described above, a rendered image based on PDL was used as the image data to which the embedded information used for printing was embedded, but this is not the only option. For example, the PC20 may embed the embedded information into RGB image data acquired from an external source via USB, IEEE1394, LAN, etc. Also, as mentioned above, the embedded information is not limited to a form that guarantees the original document; it is acceptable as long as the predetermined information is embedded on the paper in a form that makes it difficult for the user to see.
[0145] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
Claims
1. An image processing device that extracts embedded information from a print document, comprising: a first acquisition means for acquiring information indicating that the print document is printed in a first printing method or a second printing method; Based on the information, if the print document is printed by a first printing method, first image data is obtained by reading the print document while irradiating the print document with light; a second acquisition means for acquiring second image data read without irradiating the print document with light when the print document is printed by a second printing method; an extracting means for extracting the embedded information using at least one of the first image data and the second image data; An image processing device comprising:
2. 2. The image processing apparatus according to claim 1, wherein the printing method is an inkjet method or an electrophotographic method.
3. The image processing device according to claim 1 or 2, characterized in that the extraction means extracts the embedded information using the first image data when the printing method indicated by the information acquired by the first acquisition means is an inkjet method, and extracts the embedded information using the second image data when the printing method indicated by the information acquired by the first acquisition means is an electrophotographic method.
4. The image processing device according to claim 1 or 2, characterized in that the extraction means extracts the embedded information using both the first image data and the second image data when the printing method indicated by the information acquired by the first acquisition means is an electrophotographic method.
5. The image processing device according to claim 4, characterized in that the extraction means synthesizes the first image data and the second image data based on a comparison between the luminance difference of each pixel of the first image data and the second image data and the average luminance difference of the entire image, and extracts the embedded information from the synthesized image data.
6. 6. The image processing apparatus according to claim 1, wherein the first acquisition unit acquires the information by accepting an instruction from a user.
7. the first acquisition unit acquires the information by determining a printing method of the print document based on the first image data and the second image data; 3. The image processing apparatus according to claim 1, wherein the extraction means extracts the embedded information by using image data corresponding to the printing method.
8. The image processing device according to claim 1 , further comprising a photographing function.
9. 9. The image processing device according to claim 1, wherein the irradiation of the light is controlled by turning a flash on or off.
10. 10. The image processing apparatus according to claim 1, wherein the second acquisition unit reads the print document by capturing an image a plurality of times with different light amounts of the irradiated light.
11. 8. The image processing apparatus according to claim 1, wherein the second acquisition unit reads the print document by scanning the print document a plurality of times with different light amount settings.
12. 12. The image processing apparatus according to claim 1, wherein the embedded information is information indicating authenticity of the print document.
13. A method for controlling an image processing device that extracts embedded information from a print document, comprising the steps of: a first acquisition step of acquiring information indicating that the print document is printed by a first printing method or a second printing method; Based on the information, if the print document is printed by a first printing method, first image data is obtained by reading the print document while irradiating the print document with light; a second acquisition step of acquiring second image data read without irradiating the print document with light when the print document is printed by a second printing method; an extraction step of extracting the embedded information using at least one of the first image data and the second image data; 13. A method for controlling an image processing device, comprising:
14. The method for controlling an image processing device according to claim 13, wherein the printing method is an inkjet method or an electrophotographic method.
15. A control method for an image processing device as described in Claim 13 or 14, characterized in that the extraction step extracts the embedded information using the first image data when the information acquired by the first acquisition step is of an inkjet type, and extracts the embedded information using the second image data when the information acquired by the first acquisition step is of an electrophotographic type.
16. A control method for an image processing device as described in Claim 13 or 14, characterized in that the extraction step extracts the embedded information using both the first image data and the second image data when the information acquired by the first acquisition step is of an electrophotographic type.
17. The control method of an image processing device described in Claim 16, characterized in that the extraction step synthesizes the first image data and the second image data based on a comparison of the luminance difference of each pixel of the first image data and the second image data with the average luminance difference of the entire image, and extracts the embedded information from the synthesized image data.
18. A control method for an image processing device as described in any one of claims 13 to 17, characterized in that the first acquisition step acquires the information by accepting an instruction from a user.
19. The first obtaining step obtains the information by determining a printing method of the print document based on the first image data and the second image data, 15. The method according to claim 13, wherein the extraction step extracts the embedded information by using image data corresponding to the printing method.
20. The method for controlling an image processing device according to any one of claims 13 to 19, characterized in that the image processing device has a photographing function.
