Image processing device and image processing method

Abstract

To improve the accuracy of extracting information.SOLUTION: An image processing apparatus extracts embedded information from a print document, and comprises: first acquisition means that acquires a printing method for the print document; second acquisition means that can acquire first image data by performing reading while irradiating the print document with light; third acquisition means that can acquire second image data by performing reading without irradiating the print document with light; and extraction means that extracts the embedded information by using at least one of the first image data and the second image data according to the printing method acquired by the first acquisition means.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to a technique for extracting embedded information from a print document. [Background technology]

[0002] There is a known technology for embedding management information when printing an electronic document onto paper. Patent Document 1 describes an example in which auxiliary lighting is applied to the portion of the printed manuscript where the information is embedded, thereby accurately capturing the printed manuscript or the embedded information within the focal distance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 00 / 70585 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when capturing an image using auxiliary lighting as in Patent Document 1, the image quality of the captured image may be degraded depending on the printing method of the print document, and in such cases, there is a risk that the embedded information cannot be extracted.

[0005] Therefore, an object of the present invention is to improve the accuracy of information extraction. [Means for solving the problem]

[0006] An image processing apparatus according to one aspect of the present invention is an image processing apparatus for extracting embedded information from a print document. the printing document, is the first printing method or the second Printing method Information indicating that it was printed by a first acquisition means for acquiring the If the printing manuscript is printed by a first printing method based on the information, The printed document is read while being irradiated with light. was Acquire the first image data death , When the printing manuscript is printed by a second printing method, The printed document is read without irradiating it with light. was Acquire the second image data R No. two Acquisition method and 、 and an extracting unit that extracts the embedded information using at least one of the first image data and the second image data. [Effects of the Invention]

[0007] According to the present invention, the accuracy of information extraction can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Figure 2] 10 is a flowchart showing a flow when printing an original print document. [Figure 3] FIG. 10 is a diagram illustrating an example of a page image. [Figure 4] FIG. 10 is a diagram illustrating an example of a mask. [Figure 5] FIG. 2 is a diagram showing a pattern formed by a mask. [Figure 6] FIG. 10 is a diagram illustrating an example of a mask. [Figure 7] FIG. 2 is a diagram showing a pattern formed by a mask. [Figure 8] 10 is a flowchart showing the flow of copying an original printout. [Figure 9] FIG. 10 is a diagram showing the spatial frequency characteristics of a pattern used for embedding. [Figure 10] FIG. 10 is a diagram showing details of the falsification check process. [Figure 11] FIG. 10 is a diagram illustrating a pattern emphasis mask. [Figure 12] FIG. 10 is a diagram illustrating a modified example of the embedding method. [Figure 13] FIG. 2 is a diagram illustrating a system configuration of a mobile terminal. [Figure 14] 10 is a flowchart showing the flow of extracting text ID information. [Figure 15]FIG. 10 is a diagram illustrating a dark portion. [Figure 16] 10 is a flowchart showing the flow of extracting text ID information. [Figure 17] 10 is a flowchart showing the flow of extracting text ID information. [Figure 18] 10 is a flowchart showing the flow of extracting text ID information. [Figure 19] 10 is a flowchart showing the flow of extracting text ID information. [Figure 20] 10 is a flowchart showing a flow of determining a printing method. [Figure 21] 1 is a schematic diagram of an inkjet type and an electrophotographic type MFP. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments for carrying out the technology of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] In the following embodiments, an example in which document ID information is used as embedded information will be described. As a specific example of use, an example in which the original is guaranteed will be described. The following definitions apply to these terms. In this specification, the term "original" refers to an electronic document (digital document) registered and managed together with information indicating the authenticity of its contents, and the term "original printout" refers to a printout printed using the data of the electronic document. Therefore, for example, a public document such as the aforementioned "copy of resident registration card" issued by a government office corresponds to the "original printout." Furthermore, the term "copy of original" refers to a printout obtained by copying the "original printout" using an image processing device with a copying function. Therefore, for example, a copy of the aforementioned "copy of resident registration card" corresponds to the "copy of original." The following embodiments are not limited to those that guarantee the original, but can be generally applied to multiplexing processes that embed predetermined information on paper in a form that is difficult for users to see.

[0011] <<Embodiment 1>> <System configuration> FIG. 1 is a block diagram showing the configuration of a printing system according to this embodiment. As shown in FIG. 1, this printing system has an MFP (Multi Function Printer) 40 and an MFP 60 as printing devices, and a PC 20 as a host device. The MFP 40 has multiple functions, such as a printer function and a scanner function, and also has a copy function that combines these functions. The MFP 40 may also have a function to save and transmit image data to be printed and a function to send and receive faxes. The hardware configurations of the MFP 40 and the PC 20 will be described below.

[0012] Figure 21 shows the external appearance of each MFP. Figure 21(a) shows a schematic diagram of the inkjet MFP 40, and Figure 21(b) shows a schematic diagram of the electrophotographic MFP 60. The MFP 40 uses an inkjet printing method and has a scanner unit 2102 and a printer unit 2103. The printer unit 2103 has a recording head 2104 and an ink tank unit 2105. The recording head 2104 ejects ink according to print data. The ink tank unit 2105 stores the inks to be supplied to the recording head 2104. The stored inks are cyan, magenta, yellow, and black. Some models may also store spot color inks separately. The MFP 60 uses an electrophotographic printing method and has a scanner unit 2112 and a printer unit 2113. The printer unit 2113 contains a photosensitive drum 2114 and a transfer roller 2115. The photosensitive drum 2114 holds ink for each color: cyan, magenta, yellow, and black. A latent image is formed on the photosensitive drum according to the print data. A transfer roller 2115 transfers the formed latent image onto the transported paper, and the printing process is carried out.

[0013] The MFP main body 40, which uses an inkjet printing method, is mainly composed of the following elements: The CPU 41 is an arithmetic processing unit that controls the entire MFP 40 and performs, for example, the copy process (described below) in accordance with programs stored in the ROM 43 and RAM 42. The RAM 42 is a volatile storage device that temporarily stores programs and data. The ROM 43 is a non-volatile storage device that stores table data and programs used in various processes (described below). The data transfer interface (DATA TRANSFER I / F) 44 controls the sending and receiving of data between the MFP 60 and the PC 20. The print controller (PRINT Controller) 45 controls the heating operation of a heater mounted on a print head (not shown) based on print data to eject ink. The image processing accelerator (IMAC) 46 is an arithmetic processing unit that can perform image processing faster than the CPU 41. Note that the image processing accelerator 46 is not necessarily required; the table parameter creation process and image processing described above may be performed solely by the CPU 41, depending on the printer specifications. A scanner controller 47 controls the emission of an LED mounted on a scanner unit (not shown), acquires light intensity information from the scanner unit, and controls writing to RAM 42. In this way, an image of a document set on a document table (not shown) is read. A motor controller 48 controls a plurality of motor units (not shown) to move the recording head relative to the recording paper and to move the scanner unit relative to the document.

[0014] Furthermore, MFP main body 60, which uses an electrophotographic printing method, has the same configuration as MFP main body 40, which uses an inkjet printing method, but the printing device portion is different. Print controller 65 performs printing processing by developing and transferring data onto the drum surface according to control parameters from a predetermined address in RAM 62. Note that the configurations of MFP 40 and MFP 60 are not limited to those shown in the drawings, and may also include, for example, a network interface that connects to an external network and communicates with other external PCs, etc.

[0015] The PC 20 is primarily composed of the following elements: The CPU 21 is an arithmetic processing unit that controls the entire PC 20 and executes, for example, the printing of an original document (described later) in accordance with programs stored in the HDD 23 and RAM 22. The RAM 22 is volatile storage and temporarily stores programs and data. The HDD 23 is nonvolatile storage and also stores programs and data. The data transfer interface (DATA TRANSFER I / F) 24 controls the transmission and reception of data between the MFP 40 and the MFP 60. This data transmission and reception can be performed via a wired connection such as USB, IEEE 1394, or LAN, or a wireless connection such as Bluetooth or WiFi. The keyboard and mouse interface (KEYBOARD MOUSE I / F) 25 is an interface that controls a human interface device (HID) such as a keyboard or mouse. The display interface (DISPLAY I / F) 26 controls the display on a display (not shown). A network interface (Network I / F) 27 connects the PC 20 to an external network, communicates with one or more external PCs, and issues document ID matching requests, result requests, document data requests, and the like.

[0016] <Original printing process> Next, the flow of printing an original electronic document will be explained with reference to the flowchart in Figure 2. The series of processes shown in the flowchart in Figure 2 begins when a user selects an electronic document to be printed via the UI of a predetermined printing application installed on PC 20 and instructs printing that includes embedding information indicating its authenticity. In the following explanation, the symbol "S" denotes a step.

[0017] In S201, data of an electronic document designated to be printed is acquired. In this embodiment, various original electronic documents are stored and managed in an external PC (not shown). PC 20 transmits an acquisition request for a specific electronic document to the external PC via network interface 27, and receives and acquires data for printing the designated electronic document, specifically PDL data, from the external PC. Here, PDL stands for Page Description Language, and is composed of a set of drawing commands for each page. Types of drawing commands are defined for each PDL specification, and in this embodiment, the following three types are mainly used as examples. TEXT drawing command: (X1, Y1, color, font information, string information) BOX drawing command: (X1, Y1, X2, Y2, color, fill shape) IMAGE drawing command: (X1, Y1, X2, Y2, image file information) In addition to the three types mentioned above, there are also the DOT drawing command for drawing points, the LINE drawing command for drawing lines, and the CIRCLE drawing command for drawing arcs, and these drawing commands are used depending on the application. Commonly used PDLs include PDF (Portable Document Format) proposed by Adobe, XPS proposed by Microsoft, and HP-GL / 2 proposed by HP.

[0018] Figure 3 is a diagram showing an example of an image of one page of an electronic document (hereinafter referred to as a "page image"). The size of the page image 300 shown in Figure 3 is assumed to be 600 pixels wide and 900 pixels high. The PDL corresponding to the page image 300 is shown below. <PAGE=001> <text>50,50, 550, 100, BLACK, STD-18, "ABCDEFGHIJKLMNOPQR”< / text> <text>50,100, 550, 150, BLACK, STD-18, "abcdefghijklmnopqrstuv”< / text> <text>50,150, 550, 200, BLACK, STD-18, "1234567890123456789”< / text> <box> 50, 300, 200, 450, GRAY, STRIPE< / box> 250, 300, 550, 800, “PORTRAIT.jpg”

[0019] The first line of the above PDL<PAGE=001> is a tag that indicates the number of pages. Normally, PDL is designed to be able to describe multiple pages, and tags that indicate page breaks are written in the PDL. In the above example, up to the 9th line indicates that it is the first page. If there is a second page,<PAGE=002> The second line will be written as follows: <text> From the third line< / text> The text is drawn up to the end of the line. In the text drawing command, the first two coordinates indicate the coordinates (X1, Y1) of the top left corner of the drawing area, and the next two coordinates indicate the coordinates (X2, Y2) of the bottom right corner of the drawing area. Next, it is written that the color is "BLACK (black: R=0, G=0, B=0)", the font is "STD (standard)", the font size is 18 points, and the string to be drawn is "ABCDEFGHIJKLMNOPQR". The fourth line <text> From the 5th line< / text> and the sixth line <text> From the 7th line< / text> The strings to be drawn correspond to "abcdefghijklmnopqrstuv" and "1234567890123456789". <box> from< / box>The lines up to are the BOX drawing command, with the first two coordinates indicating the upper left coordinate (X1, Y1) of the drawing start point, and the next two coordinates indicating the lower right coordinate (X2, Y2) of the drawing end point. The color is then specified as "GRAY (gray: R=128, G=128, B=128)" and the fill style as "STRIPE." Line 9 is the IMAGE drawing command. The first two coordinates indicate the upper left coordinate (X1, Y1) of the drawing area, and the next two coordinates indicate the lower right coordinate (X2, Y2) of the drawing area. It also states that the file name of the image in the specified drawing area is "PORTRAIT.jpg." Note that ".jpg" is the file extension, indicating that it is a JPEG file, a commonly used image compression format. Finally, line 10 indicates that the drawing of the page has finished. Note that the document data actually retrieved is often a combination of PDL data with font data and image files added. This is because when font data and image files are managed separately, the text and image portions cannot be formed with drawing commands alone, and there is insufficient information to form the image to be printed. In this step, document data including this PDL data is obtained by accessing an external PC.

[0020] In S202, information indicating the authenticity of the electronic document to be printed is acquired. In this embodiment, a request for information indicating the authenticity of the electronic document is sent to the external PC via the network interface 27, and the information is received and acquired. Here, document ID information is used as the information indicating the authenticity of the electronic document. The document ID information is information of a predetermined number of bits (32 bits in this embodiment) calculated using a hash function based on the document data (a combination of PDL data, font data, and image files). If even one byte of the digital data constituting the electronic document is changed, different document ID information is calculated, and therefore a unique document ID is associated with each electronic document. The risk of document ID information tampering may be reduced by having multiple external PCs collaborate to store and manage electronic document files and document ID information, employing a system configuration such as a blockchain. If the PC 20 has sufficient resources and can manage document ID information internally, internal matching processing may also be performed.

[0021] In S203, rendering processing is performed based on the document data acquired in S201. In this rendering processing, each drawing command included in the PDL data is executed to generate image data in bitmap format, which is composed of color information for each pixel. For example, in the case of the page image 300 shown in FIG. 3, a bitmap image of 600 x 900 pixels is generated. In this embodiment, each pixel of the bitmap image has 256 gradations, each with 8 bits for RGB.

[0022] In S204, the document ID information obtained in S202 is embedded in the bitmap image generated in S203. The process of embedding document ID information in a bitmap image is called "multiplexing" or "multiplexing encoding." When attempting to copy a printout obtained by printing a bitmap image that has undergone this multiplexing process, the document ID information can be extracted (decoded) from a scanned image of the printout, making it possible to determine whether the printout to be copied is the "original printout."

[0023] Handling information on an information processing device such as the PC 20 essentially means handling binary data. Binary data is information expressed as a combination of "0" and "1," and specific meanings are created when these "0"s or "1"s are connected in succession. For example, when handling the English word "hello" as binary data, using Shift-JIS, one of the character codes, the letter "h" corresponds to the binary data "01101000." Similarly, "e" corresponds to the binary data "01100101," "l" to "01101100," and "o" to "01101111." In other words, the string "hello" can be expressed in binary data as "0110100001100101011011000110110001101111." Conversely, if you can obtain the binary data "0110100001100101011011000110110001101111", you can obtain the text information representing the English word "hello." Based on this idea, you can see that multiplexing can be achieved by embedding specified data within an image so that it can be determined to be either "0" or "1."

[0024] <Multiplexing processing> Figures 4(a) and (b) show examples of masks consisting of 8x8 pixels used to generate "0" and "1." Figure 4(a) shows a mask for generating "0," and Figure 4(b) shows a mask for generating "1." By adding the values ​​in the mask to the pixel values ​​of a bitmap image, a periodic pattern can be created for each 8x8 pixel region in the image. As described above, the bitmap image of this embodiment is represented by 8 bits per color, and each pixel is assigned a value between 0 and 255. Values ​​outside the range of 0 to 255 cannot be used as image data, so if the addition result is less than 0 or a value greater than or equal to 256, the value of the addition result is generally replaced with 0 or 255 to bring it within the valid range. In the masks of Figures 4(a) and (b), pixel values ​​are changed by "-10" or "0." However, if all pixel values ​​in the bitmap image corresponding to the masked region are "0," all values ​​in that region will be "0." Note that, although the case of 8 bits per color is described here, this is not limiting. When dealing with digital images, no matter how many bits are used to represent them, there is a valid range, and changes that go outside that range cannot be made.

[0025] Figures 5(a) and (b) are diagrams visually showing the patterns formed in a bitmap image by the masks in Figure 4(a) and (b) above. In the masks in Figures 4(a) and (b), the position of "-10" is represented by solid black, and the position of "0" is represented by hatching. As can be seen from Figures 5(a) and (b), in the bitmap image after the masks are applied, a diagonal line descending to the left appears as a "pattern representing 0" and a diagonal line descending to the right appears as a "pattern representing 1."

[0026] Here, pseudo-code for alternately applying the masks in FIG. 4(a) and FIG. 4(b) to the entire bitmap image is shown below. 01: int i, j, k, l; 02: int width = 600, height=900; 03: unsigned char *data = image data; 04: int **maskA = maskData; 05: bool isMaskA = true; 06: for(j = 0; j < height; j+=8){ 07: for(i = 0; i < width; i+=8){ 08: for(k = 0; k < 8; k++){ 09: for(l = 0; l < 8; l++){ 10: if(isMaskA == true) { 11: data[(i+k)+(j+l)*width] += maskA[k][l]; 12:} 13:} 14:} 15:} 16:}

[0027] In this embodiment, the above pattern data is embedded only in the B value of the RGB values ​​of each pixel in the bitmap image. This is because when printing on a white area of ​​paper using four types of ink (CMYK), the Y ink has lower visibility than the CMY inks. When multiplexing, it is preferable to minimize the impact of the embedded pattern on the original image, i.e., to make it as inconspicuous as possible. RGB, the three primary colors of light, and CMY, the three primary colors of color, are complementary to each other, with R and C, G and M, and B and Y having opposite hues, respectively. This means that modulating the B value of the RGB values ​​significantly changes the amount of Y ink used. Therefore, in this embodiment, only the B value of the RGB values ​​is modulated to control the density fluctuation of the Y ink to be large.

[0028] In the case of the page image 300 shown in Figure 3, the white paper area is sufficiently large, and the area excluding the TEXT drawing area 301, BOX drawing area 302, and IMAGE drawing area 303 is the white paper area. In this case, the masks shown in Figures 4(a) and 4(b) may not properly embed pattern data in each drawing area other than the white paper area. For example, if a drawing area is solid black (R=0, G=0, B=0), the embedding result will also be solid black (R=0, G=0, B=0). Therefore, when embedding pattern data outside the white paper area, especially in the image drawing area, it is preferable to apply masks such as those shown in Figures 6(a) and 6(b). The masks in Figures 6(a) and 6(b) are masks that change each pixel of the bitmap image by "-10," "0," and "+10." In the case of the masks shown in Figures 6(a) and 6(b), even if all pixel values ​​in the bitmap image corresponding to the masked area are "0," the pixel values ​​of the corresponding area will ultimately be "0" or "+10." Therefore, pattern data can be embedded into all pixels in the bitmap image. Figures 7(a) and 7(b) visually illustrate the patterns imparted to the bitmap image by the masks shown in Figures 6(a) and 6(b). In the masks shown in Figures 6(a) and 6(b), the "-10" position is represented by solid black, the "0" position by hatching, and the "+10" position by solid white. As can be seen from Figures 7(a) and 7(b), the bitmap image after the masks are applied will have multiple diagonal lines sloping downward to the left with different densities as the "0" pattern and multiple diagonal lines sloping downward to the right with different densities as the "1" pattern. Note that, since there is a high probability that there will be many blank areas in the text and box drawing areas, it is preferable to apply the masks shown in Figures 4(a) and 4(b). However, even for TEXT and BOX drawing areas, the masks shown in Figures 6(a) and 6(b) may be appropriate depending on the specified color, character thickness, etc. Also, even for IMAGE drawing areas, there may be cases where the area is very close to the white paper area. For this reason, the type of mask to apply may be determined by obtaining a density histogram for each drawing area.For example, if the maximum value of the RGB value is greater than the threshold, the masks in Figures 4(a) and 4(b) are applied, and if it is equal to or less than the threshold, the masks in Figures 6(a) and 6(b) are applied. Alternatively, the input color space is L. * a * b * In this case, the luminance is L * The mask to be applied may be switched by comparing the value with a threshold value, thereby enabling reliable embedding to be achieved.

[0029] Returning to the explanation of the flowchart in FIG.

[0030] In S205, a print image is generated based on the multiplexed bitmap image (hereinafter referred to as the "multiplexed image") generated in S204. A known method may be used to generate this print image. In this embodiment, each pixel of the multiplexed image is subjected to color conversion processing, color separation processing, output characteristic conversion processing, and quantization processing. Each processing will be briefly described below.

[0031] (Color conversion processing) Color conversion processing converts the pixel values ​​(RGB values) of a multiplexed image so that they can be optimally reproduced on the MFP 40. Generally, color values ​​specified in PDL drawing commands are set to be optimally displayed on a display, and if those color values ​​are output to a printer as is, the output will have a different color tone than what is seen on the display. Therefore, color conversion processing is performed to absorb the color differences between the two. This color conversion processing uses a three-dimensional lookup table (LUT) that associates combinations of input pixel values ​​(Rin, Gin, Bin) in the multiplexed image with combinations of optimal output pixel values ​​(Rout, Gout, Bout). Currently, the input values ​​Rin, Gin, and Bin each have 256 gradations. Therefore, it is ideal to prepare a table (Table1

[0256]

[0256]

[0256] [3]) with a total of 16,777,216 pairs of 256 x 256 x 256 output values ​​(Rout, Gout, Bout) as shown below. Rout = Table1[Rin][Gin][Bin][0] Gout = Table1[Rin][Gin][Bin][1] Bout = Table1[Rin][Gin][Bin][2] However, known techniques for reducing the table size may be used, such as reducing the number of grids in the LUT from 256 grids to, for example, 16 grids, and determining the output value by interpolating table values ​​between the grids.

[0032] (Color separation processing) Color separation processing is a process that converts the output values ​​Rout, Gout, and Bout of the color conversion processing into output values ​​for each ink color (here, the four colors CMYK) to be recorded on paper using the inkjet method. There are various ways to achieve this color separation processing. In this embodiment, a three-dimensional lookup table (Table 2

[0256]

[0256]

[0256] [4]) is used that associates the combinations of the above-mentioned output pixel values ​​(Rout, Gout, Bout) with the combinations of ink color pixel values ​​(C, M, Y, K) shown below. C = Table2[Rout][Gout][Bout][0] M = Table2[Rout][Gout][Bout][1] Y = Table2[Rout][Gout][Bout][2] K = Table2[Rout][Gout][Bout][3] In this case, it is preferable that only the Y value of the CMYK values ​​corresponding to the result of modulating the paper-white region in the multiplexing process of S204 has a value greater than 0. More specifically, it is preferable that the Y value of the CMYK values ​​corresponding to the region where pixel values ​​of R=255, G=255, B=255 are converted to pixel values ​​of R=255, G=255, B=245 has a value greater than 0 and a value closer to 0 that is smaller than the Y value, as explained in S204. The reason for this is that it is desirable to reduce the visibility of the embedded pattern. Note that, as with the color conversion process described above, known techniques for reducing the table size may be used.

[0033] (output characteristic conversion processing) The output characteristic conversion process converts the density of each CMYK ink color into a print dot rate. Specifically, for example, the density of 256 levels for each color is converted into a dot rate of 1024 levels for each color. This output characteristic conversion process uses a one-dimensional lookup table (Table 3[4]

[0256] ) that sets the print dot rate (Cout, Mout, Yout, Kout) shown below, which corresponds to the density of each ink color. Cout = Table3[0][C] Mout = Table3[1][M] Yout = Table3[2][Y] Kout = Table3[3][K] As in the case of the color conversion process and color separation process described above, known techniques for reducing the table size may be used.

[0034] (quantization processing) The quantization process converts the printing dot count rates (Cout, Mout, Yout, Kout) of each ink color described above into the quantized values ​​(Cdot, Mdot, Ydot, Kdot) shown below, which represent whether the printing dot of each pixel is on or off. Cdot = Halftone[Cout][x][y] Mdot = Halftone[Mout][x][y] Ydot = Halftone[Yout][x][y] Kdot = Halftone[Kout][x][y] The above are quantized values ​​for the dithering method, and by comparing them with threshold values ​​in the dithering matrix corresponding to each pixel position, a value indicating whether the recording dot of each ink color is on or off can be obtained. In this case, the occurrence probability of each recording dot is Cout / 1023, Mout / 1023, Yout / 1023, Kout / 1023. Note that the quantization processing method is not limited to the dithering method, and other methods such as error diffusion may also be used.

[0035] By executing the above-described processes in order, a print image is generated from the multiplexed image. Returning to the explanation of the flowchart in FIG.

[0036] In S206, the print image data generated in S205 is sent together with the print instruction to the MFP 40 or another printer (not shown), and printing is performed by the destination printer.

[0037] The above is the process leading up to the generation of the original print. In this embodiment, the B value of the RGB values ​​is modulated during the multiplexing process (S204). However, modulation can also be performed on the CMYK values. In this case, the pixel values ​​of the paper white area are Y=0, M=0, C=0, and K=0, so positive values ​​must be used for modulation. In the case of the masks illustrated in Figures 4 and 6, the signs of the modulation values ​​in the mask are inverted, i.e., "-10" becomes "+10" and "+10" becomes "-10." Modulation performed on the CMYK values ​​in this way provides greater control over limiting the ink applied to the paper white area to Y ink only. On the other hand, modulation performed on the RGB values ​​provides greater control over suppressing hue fluctuations when embedding in the image drawing area. Therefore, it is preferable to select an appropriate modulation method depending on the characteristics of the recording process, such as electrophotography or inkjet, and the ratio of paper white, text, and image areas within the page image to be printed.

[0038] <Copying of "original printed matter"> Next, the flow of copying the "original printout" that is the result of the above-mentioned original printing process using the MFP 40 will be described with reference to the flowchart in Fig. 8. The series of processes shown in the flowchart in Fig. 8 begins when a user places the original printout on a platen (not shown) and issues a copy instruction via the UI (user interface) of the MFP 40. In the following description, the symbol "S" denotes a step.

[0039] First, in S811, a printed material to be copied that is set in the MFP 40 is read by a built-in scanner unit (not shown). To read this printed material, LED light is irradiated onto the printed material placed on the platen, and the reflected light is converted into an analog electrical signal by an imaging element such as a CCD that faces each pixel.

[0040] In the next step S812, the analog electrical signal obtained in S811 is digitized to obtain a bitmap image in the RGB color space. At this time, image processing such as MTF correction, input correction, shading correction, and color conversion is performed on each pixel of the bitmap image. Each process is briefly described below.

[0041] <MTF correction processing> MTF (Modulation Transfer Function) correction is a correction process related to the resolution of the scanner unit's reading performance. When an image is optically read using a scanner unit, the image becomes blurred due to factors such as deviation from the focus position or the performance limits of the lens itself, so a certain degree of restoration is performed using filter processing, etc. In this case, if strong enhancement processing is used to completely restore the image, image defects such as blown-out highlights and enhancement of image noise and dust pixels will become more noticeable, so it is necessary to design the filter strength by balancing the image quality improvement with the defects. Below is an example of an edge enhancement filter that multiplies the center of the image by 5 and multiplies the pixel values ​​of the top, bottom, left, and right pixels by -1. R'[x][y] = R[x][y]×5-R[x-1][y]-R[x+1][y]-R[x][y-1]-R[x][y+1] G'[x][y] = G[x][y]×5-G[x-1][y]-G[x+1][y]-G[x][y-1]-G[x][y+1] B'[x][y] = B[x][y]×5-B[x-1][y]-B[x+1][y]-B[x][y-1]-B[x][y+1]

[0042] (Input correction processing) Input correction is the process of converting the output value of the CCD (image sensor), which is originally a photon quantity, into a brightness that matches the sensitivity of the human eye. This converts, for example, an R'G'B' signal with 4096 gradations for each color into a color intensity value (R', G', B') with 1024 gradations for each color. This conversion uses the one-dimensional lookup table shown below (Table 4[4]

[4096] ), which sets the optimal recording dot count rate for each ink color density. R” = Table4[0][R'] G” = Table4[1][G'] B” = Table4[2][B'] However, known techniques for reducing the table size may be used, such as reducing the number of grids in the LUT from 4096 grids to, for example, 256 grids, and determining the output value by interpolating table values ​​between the grids.

[0043] (Shading correction processing) Shading correction reduces uneven color and density caused by differences in reading sensitivity at each pixel position due to manufacturing and assembly variations in the lenses, LEDs, CCDs, and other components that make up the scanner. For example, R"G"B" signals with 1024 gradations for each color are converted into color intensity values ​​(R"', G"', B"') with 256 gradations for each color. This conversion uses the following one-dimensional lookup table (Table 5[x][3]

[1024] ), which specifies the density adjustment value for each pixel position in the X direction (the direction in which the scanner lens is arranged). R”' = Table5[x][0][R”] G”' = Table5[x][1][G”] B”' = Table5[x][2][B”] As in the case of the input correction process described above, known techniques for reducing the table size may be used.

[0044] (Color conversion processing) The 256-level color intensity values ​​(R"', G"', B"') for each color calculated through the processing up to this point are values ​​specific to the scanner unit, as opposed to when printing. The color conversion processing here converts the values ​​specific to the scanner unit into RGB values ​​(Rout, Gout, Bout values) suitable for display on the display. The input values ​​R"', G"', and B"' each have 256 levels. Therefore, for this conversion, the following three-dimensional lookup table (Table 6

[0256]

[0256]

[0256] [3]) is used, which has a total of 16,777,216 sets of output values ​​(256 x 256 x 256). Rout = Table1[R”'][G”'][B”'][0] Gout = Table1[R”'][G”'][B”'][1] Bout = Table1[R”'][G”'][B”'][2] As in the case of the input correction process and the shading correction process described above, a known method for reducing the table size may be used. In this way, a bitmap image is obtained.

[0045] In the next step S813, a process for extracting document ID information is executed for the bitmap image acquired in step S812. Specifically, a process is executed in units of a predetermined area (here, 8x8 pixels) to determine whether the pattern indicating "0" or "1" described above is formed within the bitmap image, and the process extracts the numerical sequence information represented by "0" and "1." By repeatedly applying this process to the entire bitmap image, the document ID information embedded by the multiplexing process when the original was printed is decoded. Information embedded by the multiplexing process is sometimes called "embedded information" or "multiplexed information." The process for decoding the embedded information will now be described in detail.

[0046] (Decoding process of embedded information) First, the location where the embedded information is embedded within the acquired bitmap image is detected. The embedded location can be detected by analyzing the spatial frequency characteristics of each unit area (here, an 8x8 pixel area). Figure 9 shows the spatial frequency characteristics of the pattern used for embedding. The horizontal axis represents horizontal frequency, and the vertical axis represents vertical frequency, with the frequency increasing with increasing distance from the origin. In this embodiment, two types of patterns corresponding to "0" and "1" are embedded in the image (see Figures 5 and 7 above). In this case, a value of "10" is subtracted from the B component of each of the RGB color components (in the case of the mask in Figure 4; in the case of the mask in Figure 6, an addition and subtraction are performed). As a result, for example, the left-sloping pattern formed in the image by the mask in Figure 4(a) generates a large power spectrum on line A02. Similarly, the right-sloping pattern formed in the image by the mask in Figure 4(b) generates a large power spectrum on line A01. Therefore, by detecting this power spectrum, data "0" or "1" can be extracted. It should be noted that, as a preprocessing step for power spectrum detection, edge detection can be performed to enhance the power spectrum.

[0047] The data extraction using frequency analysis described above requires accurate extraction of the analysis area from the image data, so a process to correct for misalignment of the coordinate position is also performed. For example, first, extraction of a unit area from a bitmap image and frequency analysis are repeated vertically and horizontally, shifting the area by one pixel at a time. If the image size is 600 pixels wide and 900 pixels high, this process is repeated a total of 64 times. The point where the spectrum is strongest is then set as the reference position for extraction. Then, by extracting the embedded information based on the reference position, the embedded sequence of "0"s or "1"s can be obtained with high accuracy.

[0048] In this embodiment, as explained in S204, the information to be embedded is text data, and its character code is a value quantified in "Shift JIS." In this case, in 1-byte Shift JIS code (half-width characters), for example, the alphabet "h" corresponds to binary data "01101000," "e" corresponds to "01100101," "l" corresponds to "01101100," and "o" corresponds to "01101111." Therefore, if the extracted numeric sequence of the embedded information is "0110100001100101011011000110110001101110001101111," the English word string "hello" is obtained. In this way, the document ID information embedded as embedded information is extracted.

[0049] Returning to the explanation of the flowchart in FIG.

[0050] In S814, it is determined whether extraction of document ID information was successful in S813. If extraction of document ID information was successful, the process proceeds to S815; if not, the process proceeds to S820. There are two possible cases where extraction fails: one is when document ID information was not embedded in the printed material to be copied (Possibility 1). The other is when document ID information was embedded in the printed material to be copied, but the printed material was dirty or had handwritten characters added later, making it impossible to correctly detect the predetermined pattern representing the embedded information (Possibility 2). In the case of Possibility 1, the process proceeds directly to S820. In the case of Possibility 2, the user may be notified by displaying a message stating, "You are about to copy an authentic document (original printed material) with an embedded document ID." This allows the user to know that they are attempting to copy a potentially inauthentic printed material, and gives them the opportunity to choose whether to cancel the copy operation or otherwise. For example, if the result obtained from the extraction process is between 1 bit and 31 bits inclusive out of the total 32 bits of document ID information, it can be determined that the possibility is 2 and a notification like this can be sent. Note that it is not completely impossible that a single pattern similar to the above-mentioned predetermined pattern is accidentally included in the image. Taking such rare cases into consideration, it is preferable to determine the possibility as 1 when less than 16 bits, half of 32 bits, are extracted, and to determine the possibility as 2 when between 16 bits and 31 bits inclusive are extracted.

[0051] In the following S815, a matching process is performed on the successfully extracted document ID information. In this embodiment, similar to the document ID acquisition process in S202, the MFP 40 accesses the external PC to request a matching and obtains the matching result. Specifically, a matching request to determine whether the extracted document ID information is officially registered and valid is transmitted via a network I / F (not shown), and the matching result is received and obtained from the external PC. Note that if the MFP 40 has sufficient resources and can manage document ID information internally, it is also possible to perform the matching process internally.

[0052] Next, in S816, the process is assigned depending on whether the document ID information extracted in S813 is officially registered and valid as a result of the comparison. If the document ID is valid, the process proceeds to S817, and if the document ID is invalid, the process proceeds to S820. If the document ID is invalid, the user may be notified by displaying a message saying, "You are attempting to copy an unauthentic document with an invalid document ID." This gives the user an opportunity to choose whether to cancel the copying operation, etc.

[0053] In S817, a check is performed to determine whether the printed material to be copied has been tampered with (tamper check process). The general flow of this tamper check process is as follows: First, the MFP 40 sends a document data acquisition request together with document ID information to an external PC via the network I / F 17. Then, the MFP 40 receives and acquires document data (PDL data) linked to the document ID information from the external PC, and performs rendering on the document data. This is the preprocessing. The bitmap image obtained by the rendering process is compared with the bitmap image acquired in S812 to determine whether tampering has occurred. The tamper check process will now be described in detail with reference to another flowchart shown in FIG. 10. Note that in the description of the flow in FIG. 10, the bitmap image obtained by the rendering process will be referred to as a "rendered image," and the bitmap image obtained by the document reading process will be referred to as a "scanned image."

[0054] <Details of the tamper check process> In S1001, a counter that counts pixels determined to be tampered with in S1010 (described later) is initialized. Specifically, the count number of the counter (number of tampered pixels) is set to "0."

[0055] In the next step, S1002, a normalization process is performed on the scanned image acquired in S812. This normalization process is performed to match the dynamic range between the scanned image and the rendered image. For example, the brightest part of a scanned image generally represents the color of the paper of the printed material being scanned, and in principle, has a certain density value. On the other hand, the brightest part of a rendered image is a pixel with pixel values ​​R=255, G=255, and B=255. The darkest part of a scanned image generally represents the black color of ink or toner, and in principle, has a brightness value due to some reflected light. On the other hand, the brightest part of a rendered image is a pixel with pixel values ​​R=0, G=0, and B=0. As such, there is an inherent difference between the brightest and darkest colors in the two images. Furthermore, if the printed material is output in color, there is also a difference in the color tone between the two images. For example, the most vivid red that can be printed has a lower saturation in the scanned image than the pixel values ​​(R=255, G=0, B=0) that represent the most vivid red in the rendered image. Therefore, each pixel value (RGB value) of the scanned image is normalized using the following equations (1) to (3) to find new pixel values ​​(Rnorm, Gnorm, Bnorm). Rnorm = (R - Rd) / (Rw-Rd)×255 Equation (1) Gnorm = (G - Gd) / (Gw-Gd)×255...Equation (2) Bnorm = (B - Bd) / (Bw-Bd)×255...Equation (3) By normalization processing using the above formulas (1) to (3), the RGB values ​​of the brightest color (white) in the scanned image become R=255, G=255, B=255, and the RGB values ​​of the darkest color (black) become R=0, G=0, B=0.

[0056] In the next step S1003, the normalized scanned image is subjected to a filter process, which performs stronger edge enhancement to make it easier to compare with the rendered image.

[0057] In the next step S1004, the image pattern representing the embedded information is removed from the scanned image after filtering. The print image generated in step S205 in the flow of FIG. 2 differs from the original document image (the image of the original electronic document) due to the multiplexing process (S203). Therefore, this process is performed to eliminate these differences as much as possible. Specifically, the pattern data embedded by the multiplexing process is subtracted from the scanned image data. This brings the scanned image closer to the state before the multiplexing process. In this embodiment, this is achieved by multiplying each value in the masks in FIGS. 4 and 6 by "-1" and adding the resulting value to each pixel of the scanned image.

[0058] Next, in S1005, the scanned image from which the embedded information has been removed is compared with the rendered image. Specifically, the rendered image and the scanned image from which the embedded information has been removed are compared pixel by pixel to determine the difference values ​​(ΔR, ΔG, ΔB) expressed by the following equations (4) to (6). ΔR = |R[x][y] of rendered image - R[x][y] of scanned image| Equation (4) ΔG = |G[x][y] of rendered image - G[x][y] of scanned image| Equation (5) ΔB = |B[x][y] of rendered image - B[x][y] of scanned image| Equation (6)

[0059] Next, in S1006, it is determined whether the difference values ​​(ΔR, ΔG, ΔB) obtained in S1005 exceed threshold values. In this embodiment, these are compared with threshold values ​​(Rth, Gth, Bth) previously set for each of the RGB channels, and if any of ΔR, ΔG, or ΔB exceeds the corresponding threshold, the process proceeds to S1007; if not, the process proceeds to S1008. The threshold values ​​are determined taking into consideration the characteristics of the scanner unit and the printing device, and in this embodiment, values ​​such as Rth=Gth=Bth=64 may be set.

[0060] In S1007, the count value of the counter indicating the number of tampered pixels is incremented (+1). After the counter is incremented, the process proceeds to S1008. Then, in S1008, it is determined whether or not comparison of pixel values ​​has been completed for all pixels. If there are any unprocessed pixels, the process returns to S1005 to continue processing, and if comparison of pixel values ​​has been completed for all pixels, the process proceeds to S1009.

[0061] In S1009, it is determined whether the count value of the counter indicating the number of tampered pixels exceeds a threshold value. In this case, the threshold value may be set to, for example, 3% of the total number of pixels, assuming cases where dust is mixed in when the scanned image is acquired. If the count value exceeds the threshold value, the process proceeds to S1010; if not, the process proceeds to S1011.

[0062] In S1010, the result of the tampering check process is determined to be no tampering, and the process ends. On the other hand, in S1011, the result of the tampering check process is determined to be tampered, and the process ends.

[0063] The above is the content of the tampering check process. Note that in this embodiment, the comparison is performed on a pixel-by-pixel basis, but this is not limited to this. For example, the comparison may be performed on a larger scale, such as by comparing the average values ​​in units of multiple pixel blocks. When the tampering check process is completed, the process proceeds to S818 in FIG. 8.

[0064] Returning to the explanation of the flowchart in FIG.

[0065] In S818, the bitmap image obtained in S812 is assigned to a different process depending on the result of the tampering check process. If no tampering is detected, the process proceeds to S819, and if tampering is detected, the process proceeds to S820. If the result of the tampering check process indicates tampering, the user may be notified by displaying a message stating, "You are attempting to copy a tampered document." This gives the user an opportunity to choose whether to cancel the copy operation or not.

[0066] Next, in S819, information indicating that the image is a copy of the "original printout" is added to the bitmap image acquired in S812. By applying an embedding method (e.g., modulating each RGB value) that is more visible than embedding document ID information, it becomes easier to recognize that the image is a photocopied printout rather than an "original printout." In addition, in conjunction with this addition process, the user may be notified by displaying a message indicating that "an attempt is being made to copy an original printout." This gives the user an opportunity to choose whether to cancel the copy operation or otherwise.

[0067] <Highlight embedded patterns in scanned images> The embedded pattern of the bitmap acquired in S812 deteriorates due to printing and scanning. Figure 11(a) shows a simplified representation of pattern deterioration, with a mask 1100 representing the extracted pattern. Region 1101 is a portion of the embedded pattern. Region 1102 is a portion of the embedded pattern, and compared to region 1101, the difference between the pixel values ​​of regions other than the embedded pattern is smaller, resulting in a weaker embedding strength.

[0068] If the embedded pattern in the scanned image is like mask 1100, it can be determined that the extracted embedded pattern is weak. Furthermore, if an error is corrected when extracting document ID information in S813, the pattern can also be determined to be weak. If the embedded pattern is weak, the embedded pattern in the scanned image is emphasized. Mask 1103 in FIG. 11(b) is an emphasis mask created based on pixels with a weak embedded pattern, based on FIG. 4(a).

[0069] Area 1104 sets the strength to be added to pixels with a weak embedded pattern. The strength may be set from the difference between area 1101 and area 1102, or may be set uniformly from a predetermined value. The same multiplexing process as in S204 is performed from the extracted document ID information. At this time, for areas where the embedded pattern is determined to be weak, an emphasis mask is added to the mask that creates the variation in FIG. 4, and the multiplexing process is performed. Alternatively, the embedded pattern can be emphasized by generating a pattern based on the created emphasis mask and emphasizing the scanned image based on that.

[0070] <Reducing the embedded pattern in the scanned image and overlaying a new embedded pattern> Alternatively, the embedded pattern in the scanned image may be reduced and multiplexed again. A method for reducing the embedded pattern involves applying a smoothing filter of a predetermined size to the scanned image. The filter size may be set based on the size of the embedded pattern (here, 8x8). An inverse filter may be calculated from the embedded pattern and used. If the pattern-embedded area is a blank area, a process may be performed to change the pixel values ​​to paper white, such as by removing the background. If the embedded pattern is added to a specific color plane, the reduction process may be performed only on that specific color.

[0071] The image with the reduced embedded pattern is subjected to a multiplexing process to embed the extracted text ID information in the same manner as in S204.

[0072] In the next step S820, a print image is generated based on the bitmap image acquired in step S812 or the bitmap image to which additional processing has been applied in step S819. This process corresponds to step S205 in the flow shown in Figure 2, and there is no particular difference, so a detailed explanation will be omitted. In the following step S821, printing processing is performed using the print image generated in step S820.

[0073] The above is the process flow when copying an original printed material. Note that instead of performing notification processing such as displaying a message in S814, S816, S818, and S819, control may be performed so that the copying process itself is automatically stopped at each point. This saves the user the trouble of having to confirm their intention each time.

[0074] Furthermore, the information added in S819 indicating that it is a copy can be said to be "information indicating that the content is legitimate and a copy" that guarantees that the original has not been tampered with. Therefore, character string information that more clearly expresses that the printed matter to be copied is a genuine document, such as "copy of original printed matter," may be added. Furthermore, even if it is determined in S818 that tampering has occurred, the process may proceed to generating a print image (S820) after adding the text "illegible (tampered) copy."

[0075] Alternatively, the genuine document ID information extracted from the "original printout" to be copied may be updated to document ID information indicating that it is a copy, and then re-embedded in the bitmap image and printed out in S820. This makes it possible to notify the user when an attempt is made to copy a printout in which the updated document ID information output in S820 is embedded. In this case, the document ID information indicating that it is a copy can be acquired and stored together with the document ID information acquired in the original printing process (S202) in a format similar to reservation ID information. This makes it possible to notify the user that it is a copy without having to check with an external PC.

[0076] As described above, according to this embodiment, when a "printed matter of an original" is copied, information indicating that fact is added as highly visible information, so that the user can easily recognize that it is a "copy of an original." Furthermore, when a user attempts to further copy the "copy of an original" obtained by the above-mentioned copying process, the copier can detect the additional information indicating that it is a copy and notify the user of that fact.

[0077] <Variation 1> In the above embodiment, an example has been described in which document ID information is multiplexed and embedded in units of 8x8 pixel regions. With this method, a predetermined pattern is formed in the high-frequency region of the image, making it possible to embed information in a form that is difficult for the user to see. However, the method of embedding information is not limited to this. For example, document ID information may be converted into a two-dimensional code and embedded in a form that is difficult for the user to see.

[0078] FIG. 12(a) shows an example of a two-dimensional code, and FIG. 12(b) shows the corresponding actual print pattern. In the example of FIG. 12(b), one dot is formed per 8x8 pixel area. The dot corresponding to the black pixel 1201 in the two-dimensional code in FIG. 12(a) corresponds to the dot 1202 in the print pattern shown in FIG. 12(b). In other words, no dots of the print pattern are formed at positions corresponding to white pixels in the two-dimensional code in FIG. 12(a). A specific process flow for achieving this is to first convert the document ID information into a two-dimensional code in the multiplexing process (S204) in the flow shown in FIG. 2, then convert it into pattern data of a group of spaced dots and embed it in the rendering image in predetermined area units. Then, in the subsequent print image generation process (S205), the spaced dots are formed using only Y ink, making them less visible.

[0079] When copying a printed material in which document ID information has been embedded using the method of this modified example, the document ID information can be extracted by extracting a two-dimensional code pattern formed of yellow dots in 8x8 pixel units from the read bitmap image and decoding it.

[0080] Furthermore, as an embedding method other than the two-dimensional code, for example, multiplexing may be performed by modulating a threshold value during quantization processing.

[0081] <Variation 2> In the explanation up to this point, it has been assumed that the document ID information indicating the authenticity of the original is embedded in a manner that makes it as indistinguishable as possible from the user's view during printing. However, the document ID information may also be embedded in a manner that makes it easily visible to the user (for example, by modulating the R and G values, which are the areas where the density fluctuations of the CMK inks become large).

[0082] In this case, the copying process may involve printing the rendering image generated during preprocessing of the tamper check process (S817) as is, or printing the bitmap image (S1004) from which the embedded data has been removed as is. Printing using either of these methods results in a printed product without embedded document ID information. From the user's perspective, the white area of ​​the copied printout does not contain visible pattern data, making it easy to recognize that it is not the "original printout." Furthermore, because the embedded data does not exist in the copied printout, if it is further copied, S814 will always return "No."

[0083] Although the method for extracting document ID information from a printed document using a scanner has been described above, information extraction can also be performed using a mobile device such as a smartphone or tablet. In the mobile device, a photograph of the printed document can be taken using a photographing device, and the document ID information can be extracted from the photographed image within the mobile device.

[0084] In the following, text ID information extraction will be described as a process of photographing a printed document using the camera function of a mobile device and extracting text ID information from the captured image. When extracting text ID information from a document printed using an inkjet method, brightening the document when photographing it can improve the accuracy of extraction from dark areas. Dark areas are areas in the printed document where the color is highly concentrated, especially low-brightness areas such as black.

[0085] On the other hand, when photographing dark areas of a document printed on plain paper using electrophotography with the flash on, the wax in the toner strongly reflects light, which can result in overexposure. Therefore, when extracting document ID information, overexposure can degrade the image quality of the captured image, reducing extraction accuracy. To avoid this, this embodiment describes a method for photographing a printed document using electrophotography with the flash on and off, and switching the image from which document ID information is extracted depending on whether or not the printed document contains dark areas.

[0086] FIG. 13 is a block diagram showing the system configuration of a mobile terminal. The CPU 1301 executes processing in accordance with a program stored in the ROM 1302 or RAM 1303. The RAM 1303 is a volatile storage device that temporarily stores programs or data. The ROM 1302 is a non-volatile storage device that also stores programs or data. The NETORK I / F (network interface) 1304 connects the mobile terminal 130 to an external network, communicates with one or more external PCs, and issues document ID matching requests, result requests, document data requests, and the like. The DISPLAY I / F (display interface) inputs user operations via the display and touch panel operations. The SENSOR 1306 is a sensor that measures the distance to the subject and the ambient brightness. The CAMERA 1307 photographs the subject by adjusting exposure and other settings based on the device and shooting conditions measured by the SENSOR 1306. If the amount of light measured by the sensor 1306 is low, the flash 1308 automatically emits a flash when a photograph is taken by the camera 1307. Alternatively, the flash can be forcibly switched on and off by the user or application settings. This concludes the explanation of the system configuration of the mobile terminal.

[0087] Fig. 14 shows a flow for extracting document ID information from an electrophotographically printed document. Fig. 14 explains a method for performing the processes from document reading to document ID information extraction described in S811 to S813 of Fig. 8 using CAMERA 1307 of mobile terminal 130. Note that, in practice, mobile terminal 130 performs the processes of S811 to S821 of Fig. 8, but since the processes from S814 onwards are similar to the processes when PC 20 is used, their explanation will be omitted in this embodiment. Note that the processes in each step of Fig. 14 are performed by CPU 1301 of mobile terminal 130 expanding program code stored in ROM 1302 into RAM 1303 and executing it.

[0088] In S1401, the mobile terminal 130 turns on the flash in response to a user operation and photographs a document. Note that the flash may be switched on and off by, for example, the mobile terminal 130 issuing a notification to turn the flash on or off, and the user then operating to permit the switching. The mobile terminal 130 may also be configured to automatically switch the flash on and off in response to a single instruction from the user. Next, in S1402, the mobile terminal 130 digitizes the analog electrical signal acquired in S1401 to acquire a bitmap image.

[0089] Next, in S1403, the user operates the mobile terminal 130 to turn off the flash and capture a picture of the document. In S1404, the mobile terminal 130 digitizes the analog electrical signal acquired in S1403 to acquire a bitmap image. In S1405, the mobile terminal 130 determines whether the document contains a dark area based on the acquired bitmap image. If no dark area is included, the process proceeds to S1406, and if a dark area is included, the process proceeds to S1407.

[0090] 15 is a diagram illustrating dark areas. Original 1501 shows an original that includes dark areas, and includes a character area 1502, a rectangular drawing area 1503, and an image area 1504. Rectangular drawing area 1503 is a dark area, and is a rectangular area that is solid black. On the other hand, original 1505 shows an original that does not include dark areas, and includes a character area 1506.

[0091] The determination in S1405 as to whether or not a dark area is included is made by calculating the average brightness from the RGB values ​​of N×N pixels based on the bitmap image acquired by photographing the document, and determining that the N×N pixels are dark areas if the average brightness is less than a threshold value. The threshold value is, for example, 64 / 255, which is equivalent to 1 / 4 of the brightness range. This is performed for the rectangular drawing area 1503 and the image area 1504. For the sake of explanation, it is determined that a dark area exists if even one dark area is present.

[0092] Returning to the description of Fig. 14, in S1406 when it is determined that there is no dark part, the mobile terminal 130 sets the bitmap image captured with the flash on in S1402 as the image for extracting text ID information.

[0093] On the other hand, if it is determined that there is a dark portion in S1407, the mobile terminal 130 sets the bitmap image captured with the flash off in S1404 as an image for extracting text ID information. In S1408, the mobile terminal 130 extracts text ID information from the set bitmap image. The subsequent processes performed by the mobile terminal 130 are the same as those in S814 to S821 of FIG. 8.

[0094] As described above, according to this embodiment, it is possible to improve the accuracy of information extraction. Specifically, a printed document is photographed with the flash on and off, and the image from which text ID information is extracted is switched depending on whether or not the document contains dark areas. This makes it possible to extract information with high accuracy even when a document printed using an electrophotographic method contains dark areas.

[0095] In this embodiment, the flash is turned on and off as an example, but photographing or scanning may be performed multiple times with different light intensity settings. Also, a scanner may be used instead of a mobile terminal to scan multiple times with different light intensity settings, and the image from which the document ID information is extracted may be switched using the same process.

[0096] <<Embodiment 2>> Reading of printed documents using a mobile device can be performed not only on printed documents printed using electrophotography, but also on documents printed using inkjet printing, for example. However, because the characteristics of printed documents differ depending on the printing method, using the same reading method for both electrophotography and inkjet printing can result in degradation of image quality and reduced extraction accuracy. Therefore, to maintain high extraction accuracy, the mobile device that performs the reading must perform appropriate processing depending on the printing method of the document.

[0097] For example, for documents printed on plain paper using an inkjet method, turning on the flash can improve the accuracy of extracting document ID information in dark areas. On the other hand, for documents printed using an electrophotographic method, turning on the flash can cause blown-out highlights in dark areas, potentially reducing extraction accuracy. That is, in the case of inkjet printing, photographing with the flash on improves extraction accuracy compared to photographing with the flash off, so it is preferable to photograph with the flash on. On the other hand, in the case of electrophotography, photographing with the flash on reduces extraction accuracy in dark areas, so it is preferable to photograph with the flash off. In this embodiment, a method of switching control during image acquisition by the user specifying the printing method used to print the document will be described.

[0098] Fig. 16 is a flow diagram illustrating how document ID information is extracted from a document printed by electrophotography or inkjet printing using the mobile terminal 130. Similar to Fig. 14, the flow in Fig. 16 is a flow in which the processes from document reading to document ID information extraction described in S811 to S813 in Fig. 8 are performed using the CAMERA 1307 of the mobile terminal 130. In practice, the mobile terminal 130 performs the processes of S811 to S821 in Fig. 8, but the processes from S814 onwards are similar to the processes when the PC 20 is used, and therefore will not be described in this embodiment. The processes in each step in Fig. 16 are performed by the CPU 1301 of the mobile terminal 130 by loading program code stored in the ROM 1302 into the RAM 1303 and executing it.

[0099] In S1601, the user operates the touch panel on the display 1305 of the mobile terminal 130 to input information about the printing method of the document to be photographed. In this embodiment, it is assumed that either the electrophotographic method or the inkjet method is input.

[0100] In S1602, the mobile terminal 130 determines whether to use the flash function when reading the document based on the document printing method input by the user in S1601. If the printing method input by the user is inkjet printing, the process proceeds to S1603 to read the document with the flash function, and if the document is printed using electrophotography, the process proceeds to S1604 to read the document without the flash function.

[0101] In S1603, the user operates the mobile terminal 130 to turn on the flash and take a picture of the document, and in S1604, the user operates the mobile terminal 130 to turn off the flash and take a picture of the document.

[0102] In S1605, the mobile terminal 130 digitizes the analog electrical signal of the print document photographed in the above step to obtain a bitmap image.

[0103] In step S1606, the mobile terminal 130 extracts the document ID information from the bitmap image acquired in step S1605.

[0104] The subsequent processing is the same as that in S814 to S821 in FIG. 8, and therefore a description thereof will be omitted.

[0105] As described above, according to this embodiment, by allowing the user to specify the printing method of the document and switching the capture control when acquiring an image from which to extract the document ID, it is possible to avoid image quality degradation such as blown-out highlights that occur in dark areas when scanning a printed document. As a result, it is possible to suppress a decrease in extraction accuracy. Note that, although this embodiment describes a method using a mobile terminal with the flash on and off, it is also possible to acquire an image from which to extract document ID information by scanning with a scanner at different light intensity settings.

[0106] <<Embodiment 3>> The following describes a method for extracting document ID information from a printed document using a mobile device when the printed document contains a mixture of dark and highlight areas. A highlight area is an area in the printed document that has a brightness value above a certain value.

[0107] When a document is printed using an inkjet method, capturing both dark and highlight areas with the flash on during scanning results in higher extraction accuracy. On the other hand, when a document is printed using an electrophotographic method, capturing dark areas with the flash off and highlight areas with the flash on during scanning results in higher extraction accuracy. Therefore, for documents printed using an inkjet method, extraction is performed from an image captured with the flash on, and for documents printed using an electrophotographic method, extraction is performed using both images captured with the flash on and off. Specifically, scanning is performed from images captured with the flash on and off, and an image from which document ID information is extracted is created by combining the dark and highlight areas. Then, document ID information is extracted from the created image. This embodiment will be described in detail below.

[0108] Fig. 17 shows a flow for switching the method of extracting document ID information based on the printing method specified by the user. Note that, like Fig. 14, the flow in Fig. 17 is a flow for performing the processes from document reading to document ID information extraction described in S811 to S813 in Fig. 8 using CAMERA 1307 of mobile terminal 130. In reality, mobile terminal 130 performs the processes of S811 to S821 in Fig. 8, but the processes from S814 onwards are the same as the processes when PC 20 is used, so their description will be omitted in this embodiment.

[0109] In S1701, the user operates the touch panel on the display 1305 of the mobile terminal 130 to input information about the printing method of the document to be photographed. In this embodiment, it is assumed that either the electrophotographic method or the inkjet method is input.

[0110] In S1702, the mobile terminal 130 turns on the flash in response to a user operation and captures an image of the document. In 1703, the mobile terminal 130 digitizes the analog electrical signal of the print document captured in S1702 to obtain a bitmap image.

[0111] In S1704, the mobile terminal 130 switches the extraction method for document ID information based on the printing method of the document input by the user in S1701. If the printing method input by the user is inkjet printing, the process proceeds to S1705, and if the document is printed by electrophotography, the process proceeds to S1706.

[0112] In S1705, the mobile terminal 130 extracts document ID information from the bitmap image of the document photographed with the flash on and acquired in S1703. The extraction method in this step is referred to as "method 1."

[0113] In S1706, the user operates the mobile terminal 130 to turn off the flash and capture an image of the document.

[0114] In S1707, the mobile terminal 130 acquires a bitmap image by digitizing the analog electrical signal of the print document photographed in S1706. For simplicity of explanation, it is assumed that the bitmaps photographed with the flash on and off have the same image size and that no positional deviation occurs during photographing.

[0115] In S1708, the mobile terminal 130 extracts document ID information using two images: the bitmap image of the document photographed with the flash ON acquired in S1703, and the bitmap image of the document photographed with the flash OFF acquired in S1707. The extraction method in this step is referred to as "method 2." The subsequent processing is the same as that of S814 to S821 in FIG. 8. This concludes the description of the flow for switching the document ID information extraction method based on the print method designated by the user.

[0116] Here, details of method 2 will be explained, i.e., a method of comparing images with flash ON and OFF on a pixel-by-pixel basis, creating a composite image for extracting a sentence ID based on the comparison, and extracting a sentence ID from the composite image.

[0117] FIG. 18 shows a flow in which the mobile terminal 130 compares the flash-on and flash-off images to create a composite image for extraction.

[0118] In S1801, the mobile terminal 130 calculates the average brightness of the entire image with the flash on and with the flash off, and stores the difference (Dth). In 1802, the mobile terminal 130 initializes the coordinate (x, y) values ​​for comparing the flash on and off images pixel by pixel, and designates these as the initial coordinates. In S1803, the mobile terminal 130 calculates the difference D(x, y) between the brightness value of the flash on image and the brightness value of the flash off image at the coordinate (e.g., (0, 0)) designated in S1802.

[0119] In S1804, the mobile terminal 130 determines whether the difference value D(x, y) calculated in S1803 is greater than the average luminance difference Dth calculated in S1801. If the determination is YES (i.e., D(x, y) > Dth), the process proceeds to S1805. If the determination is NO (i.e., D(x, y) < Dth), the process proceeds to S1806.

[0120] In S1805, the mobile terminal 130 sets the pixel value of the flash OFF as the pixel value of the composite image at the designated coordinates. That is, since the difference in luminance between flash ON and OFF at the designated coordinates is greater than the average luminance difference, it is determined that white bleeding has occurred at the designated coordinate position, and at this coordinate, a flash OFF image with less image quality degradation is used for the composite image.

[0121] In S1806, the mobile terminal 130 sets the pixel value of the flash ON as the pixel value of the composite image at the designated coordinates. That is, since the difference in luminance between flash ON and OFF is equal to or less than the average luminance difference, it is determined that no white bleeding has occurred at the designated coordinate position, and at this coordinate, a flash ON image with high extraction accuracy is used for the composite image.

[0122] In S1S1807, the mobile terminal 130 determines whether the generation of the composite image has ended for all pixels. If YES, the process proceeds to S1809. If NO, the process proceeds to S1808. In S1808, the position of the designated coordinates for generating the composite image is updated. After the update, the process returns to S1803 and the same process is repeated.

[0123] In S1809, the mobile terminal 130 performs article ID information extraction processing using the composite image created by comparing the luminance of all pixels of the printed document.

[0124] The above is an explanation of a method for comparing the images of flash ON and OFF for each pixel, creating a composite image for article ID extraction based on this comparison, and extracting the article ID from the composite image. By determining whether the difference at the designated coordinates is greater than the average luminance, local image quality degradation factors including white bleeding can be excluded, and an improvement in extraction accuracy can be expected.

[0125] Although the method described here involves switching based on the difference between flash ON and OFF when creating a composite image, it is also possible to use a method in which each pixel is judged to be in a dark area or not and then switch between flash ON and OFF images. By doing so, it is possible to limit the use of flash OFF images to dark areas, and increase the area in which flash ON images are used, which is expected to further improve extraction accuracy.

[0126] As described above, according to this embodiment, when a printed document contains a mixture of dark and highlight areas, the method for extracting document ID information can be switched depending on the printing method specified by the user. In particular, when extracting a document ID from a document printed using electrophotography, the area of ​​the flash-off image can be limited by determining blown-out highlights and synthesizing and extracting the flash-on and flash-off images. As a result, a decrease in extraction accuracy can be suppressed. While the method of using a mobile device with the flash on and off has been described here, images from which document ID information is extracted can also be obtained by scanning with a scanner at different light intensity settings.

[0127] <<Embodiment 4>> In this embodiment, a method for determining the printing method from a scanned print document, rather than manually setting it by the user, will be described. Two types of printing methods, inkjet printing and electrophotographic printing, will be determined.

[0128] Fig. 19 shows a flow for capturing images with the mobile terminal with the flash on and off, determining the printing method using the images, and switching the method for extracting document ID information. Note that, like Fig. 14, the flow in Fig. 19 is a flow for performing the processes from document reading to document ID information extraction described in S811 to S813 in Fig. 8 using CAMERA 1307 of mobile terminal 130. In reality, mobile terminal 130 performs the processes of S811 to S821 in Fig. 8, but the processes from S814 onwards are the same as the processes when PC 20 is used, so their description will be omitted in this embodiment.

[0129] In S1901, the mobile terminal 130 turns on the flash and photographs the document in response to a user operation. In S1902, the mobile terminal 130 digitizes the analog signal of the print document photographed in S1901 to obtain a bitmap image with the flash on.

[0130] In S1903, the user operates the mobile terminal 130 to turn off the flash and capture a photograph of the document. In S1904, the mobile terminal 130 digitizes the analog signal of the print document captured in S1903 and acquires a bitmap image with the flash off. For simplicity of explanation, it is assumed that the bitmaps captured with the flash on and off have the same image size and that no positional deviation occurs during capture.

[0131] In S1905, the mobile terminal 130 determines the printing method of the document using the flash-on and flash-off images acquired in S1902 and S1904. The determination method will be described in detail below with reference to FIG.

[0132] Figure 20 shows the flow for determining the printing method using images both with the flash on and off. The printing method is determined by taking advantage of the fact that blown-out highlights occur when an original printed using the electrophotographic method is read with the flash on. If blown-out highlights occur, the method is determined to be electrophotographic, and if not, the method is determined to be inkjet.

[0133] In S2001, the mobile terminal 130 calculates the average brightness of the entire image with the flash on and with the flash off, and stores the difference (Dth). In S2002, the mobile terminal 130 resets a counter for counting the number of pixels determined to be blown out. It also initializes the coordinate (x, y) values ​​for comparing the flash on and off images pixel by pixel, and designates these as the initial coordinates.

[0134] In S2003, the mobile terminal 130 calculates the difference D(x, y) between the luminance value of the flash-on image and the luminance value of the flash-off image at the specified coordinates. In S2004, the mobile terminal 130 determines whether the difference value D(x, y) calculated in S2003 is greater than the average luminance difference Dth calculated in S2001. If the determination is YES (i.e., D(x, y) > Dth), the process proceeds to S2005. If the determination is NO (i.e., D(x, y) < Dth), the process proceeds to S2006.

[0135] In S2005, the mobile terminal 130 increments the count by one. This is because the difference between flash-on and flash-off is greater than the average luminance difference, so it is determined that white flashing occurs at the specified coordinate position. In S2006, the mobile terminal 130 determines whether the generation of the composite image has ended for all pixels. If YES, the process proceeds to S2008; if NO, the process proceeds to S2007.

[0136] In S2007, the mobile terminal 130 updates the position of the specified coordinates for which white flashing determination is performed. After the update, it returns to S2003 and repeats the same process. In S2008, the mobile terminal 130 determines whether the number of white-flashing pixels is equal to or greater than a predetermined threshold TH using the result of counting white flashing for all pixels. The predetermined threshold TH is set relatively from the number of pixels in the dark part of the printed manuscript. For the dark part of the printed manuscript, the shadow point is set from the luminance histogram of the flash-off image, and the number of pixels with luminance values below the shadow point is counted. A predetermined ratio may be multiplied by the obtained number of pixels in the dark part to provide a margin. If the determination result is YES (i.e., white-flashing counter > TH), the process proceeds to S2009; if NO (i.e., white-flashing counter < TH), the process proceeds to S2010.

[0137] In S2009, since there is a high possibility that white flashing has occurred, the mobile terminal 130 determines that the printed manuscript was printed by an electrophotographic method and returns to the flow of FIG. 19.

[0138] In S2010, since there is little possibility of blown-out highlights, it is determined that the print document was printed using the inkjet method, and the process returns to the flow in Fig. 19. This concludes the description of the flow for determining the printing method using images with the flash on and off.

[0139] Returning to the description of Fig. 19, in S1906, the mobile terminal 130 switches processing based on the printing method of the document determined in S1905. If the printing method determined in S1905 is inkjet, the process proceeds to S1907, and if it is electrophotography, the process proceeds to S1908.

[0140] In S1907, the mobile terminal 130 extracts the document ID information using "method 1" described in the third embodiment. On the other hand, in S1908, the mobile terminal 130 extracts the document ID information using "method 2" described in the third embodiment.

[0141] The above is an explanation of the flow for capturing images with the flash on and off, determining the printing method using the images, and switching the method for extracting document ID information. According to this embodiment, the printing method of the print document when reading document ID information is determined by the mobile terminal 130, not by the user, which saves the user time and effort. Furthermore, since the determination is made mechanically using the flowchart in Figure 20 rather than by the user, the accuracy of extracting document ID information can be improved.

[0142] In this embodiment, the printing method of an original document is determined to be electrophotographic or inkjet by comparing images with the flash on and off. However, this is not limiting. The screen ruling included in the captured image may be analyzed for patterns, and if a screen is detected, the method may be determined to be electrophotographic. Alternatively, the inkjet method or electrophotographic method may be determined based on a predetermined contrast ratio between white paper and black text. Even in the case of an inkjet method, if the plain paper is coated in advance with clear ink, the method described above may be used to determine the method as electrophotographic, and extraction processing may be performed based on this determination. However, because the coating may cause overexposure with the flash on, extraction processing using images captured with the flash off, as with electrophotography, is preferable.

[0143] <<Other embodiments>> In the above embodiment, the reading of the printed document and the extraction of the document ID information are described as being performed by a mobile terminal, but this is not limited to this. For example, a device with a photographing or copying function, such as a digital camera or a copier, may be used. Furthermore, regarding the flash function used when reading with a mobile terminal, other methods of irradiating light onto the printed document may be used.

[0144] In the above-described embodiment, a rendered image based on a PDL was used as the image data into which the embedded information used for printing is embedded, but this is not limited to this. For example, the embedded information may be embedded in RGB image data acquired from an external device by the PC 20 via USB, IEEE 1394, LAN, etc. As described above, the embedded information is not limited to a form that guarantees authenticity, and may be embedded in a form that makes it difficult for the user to see the specified information on the paper.

[0145] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

1. An image processing device that extracts embedded information from a print document, a first acquisition means for acquiring information indicating that the print document is printed using a first printing method or a second printing method; If the print document is printed by a first printing method based on the information, first image data is acquired by reading the print document while irradiating the print document with light; a second acquisition means for acquiring second image data read without irradiating the print document with light when the print document is printed by a second printing method; an extracting unit that extracts the embedded information using at least one of the first image data and the second image data; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the printing method is an inkjet method or an electrophotographic method.

3. The image processing device described in claim 1 or 2, characterized in that the extraction means extracts the embedded information using the first image data when the printing method indicated by the information acquired by the first acquisition means is an inkjet method, and extracts the embedded information using the second image data when the printing method indicated by the information acquired by the first acquisition means is an electrophotographic method.

4. The image processing device described in claim 1 or 2, characterized in that the extraction means extracts the embedded information using both the first image data and the second image data when the printing method indicated by the information acquired by the first acquisition means is an electrophotographic method.

5. The image processing device described in claim 4, characterized in that the extraction means synthesizes the first image data and the second image data based on a comparison of the brightness difference of each pixel of the first image data and the second image data with the average brightness difference of the entire image, and extracts the embedded information from the synthesized image data.

6. 6. The image processing apparatus according to claim 1, wherein the first acquisition unit acquires the information by accepting an instruction from a user.

7. the first acquisition unit acquires the information by determining a printing method of the print document based on the first image data and the second image data; 3. The image processing apparatus according to claim 1, wherein the extracting means extracts the embedded information using image data corresponding to the printing method.

8. 8. The image processing device according to claim 1, further comprising a photographing function.

9. 9. The image processing device according to claim 1, wherein the irradiation of the light is controlled by turning on or off a flash.

10. 10. The image processing apparatus according to claim 1, wherein the second acquisition unit reads the print document by capturing an image multiple times with different light intensity settings for the irradiated light.

11. 8. The image processing apparatus according to claim 1, wherein the second acquisition unit reads the print document by scanning the print document a plurality of times with different light intensity settings.

12. 12. The image processing apparatus according to claim 1, wherein the embedded information is information indicating authenticity of the print document.

13. A control method for an image processing device that extracts embedded information from a print document, comprising: a first acquisition step of acquiring information indicating that the print document is printed using a first printing method or a second printing method; If the print document is printed by a first printing method based on the information, first image data is acquired by reading the print document while irradiating the print document with light; a second acquisition step of acquiring second image data read without irradiating the print document with light when the print document is printed by a second printing method; an extraction step of extracting the embedded information using at least one of the first image data and the second image data; 1. A control method for an image processing apparatus, comprising:

14. The method for controlling an image processing device according to claim 13, wherein the printing method is an inkjet method or an electrophotographic method.

15. A control method for an image processing device as described in Claim 13 or 14, characterized in that the extraction step extracts the embedded information using the first image data when the information acquired by the first acquisition step is an inkjet method, and extracts the embedded information using the second image data when the information acquired by the first acquisition step is an electrophotographic method.

16. A control method for an image processing device as described in Claim 13 or 14, characterized in that the extraction step extracts the embedded information using both the first image data and the second image data when the information acquired by the first acquisition step is electrophotographic.

17. The control method for an image processing device described in Claim 16, characterized in that the extraction step synthesizes the first image data and the second image data based on a comparison between the brightness difference of each pixel of the first image data and the second image data and the average brightness difference of the entire image, and extracts the embedded information from the synthesized image data.

18. A control method for an image processing device described in any one of claims 13 to 17, characterized in that the first acquisition step acquires the information by accepting instructions from a user.

19. The first obtaining step obtains the information by determining a printing method of the print document based on the first image data and the second image data, 15. The method of claim 13, wherein the extraction step extracts the embedded information using image data corresponding to the printing method.

20. A method for controlling an image processing device according to claim 13, wherein the image processing device has a photographing function.

Images