Image processing device, image processing method and program

Abstract

To provide a copying function with ensured intensity of embedded information.SOLUTION: In copying a printed matter, embedded information is extracted from a scanned image obtained by scanning the printed matter to be copied. When the printed matter to be copied is confirmed to be a genuine document with the extracted embedded information, the embedded information extracted from the scanned image is reconstituted and printed out.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a technology for copying printed materials. [Background technology]

[0002] Traditionally, in Japan and other countries, paper documents have been treated as official originals. However, with the advancement of an advanced information and communications society, legislation has been established (e.g., the Electronic Signature Act), and systems and deregulations have been implemented to allow electronic documents to be treated as originals. Furthermore, technologies have been developed to utilize and preserve electronic documents as originals. Given this social background, it is now common for electronic documents to be treated as originals, and paper printouts of such documents to be treated as copies (e.g., copies of resident registration cards). Furthermore, information is often embedded in such printed materials in a manner that makes it difficult to detect. For example, Patent Document 1 discloses a technology for detecting tampering in printed materials by embedding digital watermark data in the printed materials. [Patent Document 1] Japanese Patent Publication No. 2020-088780 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-270972 Summary of the Invention [Problem to be solved by the invention]

[0003] When copying an original printed material with embedded information using a copier, the embedded information may not be properly reproduced in the output printed material (copy of the original). Embedding information in a printed material is achieved by forming a predetermined pattern on the paper at a density that is difficult for the human eye to discern. If the predetermined pattern is not formed with sufficient strength when the original is printed, for example, the predetermined pattern representing the embedded information may not be properly read when the original printed material is scanned. Furthermore, due to limitations in the performance of a scanner device, the predetermined pattern representing the embedded information may not be properly read. If the strength of the information embedded in the original printed material deteriorates, the embedded information in the copy will also be degraded. The technology disclosed herein has been developed in light of these issues and aims to provide a copying function that ensures the strength of embedded information. [Means for solving the problem]

[0004] The printing system according to the present disclosure includes a scanning unit for scanning a printed matter to be copied, and a scanning image obtained by scanning the printed matter to be copied. and based on the detected pattern, an extraction means for extracting embedded information; and when the printed matter to be copied is confirmed to be a genuine document based on data linked to the embedded information extracted by the extraction means, the scanned image is The pattern related to the embedded information is enhanced by reinforcing pixels at positions where the intensity of the pattern is weak among the patterns detected from and a printing unit that performs printing processing using the print image generated by the generating unit. [Effects of the Invention]

[0005] According to the technology of the present disclosure, even if the information embedded in the original printout has deteriorated, it is possible to obtain a copy in which the necessary information is embedded with sufficient strength. [Brief explanation of the drawings]

[0006] [Figure 1] Block diagram showing the configuration of a printing system [Figure 2]Flowchart showing the process for printing an original electronic document [Figure 3] A diagram showing an example of a page image [Figure 4] 1(a) and 1(b) are diagrams showing an example of a mask; [Figure 5] 1(a) and 1(b) are diagrams showing patterns formed by a mask. [Figure 6] 1(a) and 1(b) are diagrams showing an example of a mask; [Figure 7] 1(a) and 1(b) are diagrams showing patterns formed by a mask. [Figure 8] Flowchart showing the process for copying an original printout [Figure 9] A diagram showing the spatial frequency characteristics of the pattern used for embedding. [Figure 10] Flowchart showing details of tamper check processing [Figure 11] (a) and (b) are explanatory diagrams for embedding document ID information in a QR code (registered trademark). [Figure 12] 11(a) is a diagram showing an example of a deteriorated pattern, and FIG. 11(b) is a diagram showing a pattern reinforcement mask corresponding to FIG. 11(a). DETAILED DESCRIPTION OF THE INVENTION

[0007] The printing system according to the present disclosure includes a scanning unit for scanning a printed matter to be copied, and a scanning image obtained by scanning the printed matter to be copied. The embedded information in Embedded in position an extraction unit that detects a pattern and extracts embedded information based on the detected pattern; and when the printed matter to be copied is confirmed to be an authentic document based on data linked to the embedded information extracted by the extraction unit, extracting the embedded information detected from the scanned image. The position where the embedded information is embedded The method is characterized by comprising a generating means for generating a print image by reconstructing the pattern relating to the embedded information by reinforcing pixels in the pattern at positions where the strength of the pattern is weak, and a printing means for performing a printing process using the print image generated by the generating means.

[0008] In this specification, "original" refers to an electronic document (digital document) registered and managed together with information indicating that its contents are authentic, and "printed original" refers to a printed matter produced using the data of the electronic document. Therefore, for example, a public document such as the aforementioned "copy of resident registration card" issued by a government office would be considered a "printed original." Furthermore, "copy of original" refers to a printed matter obtained by copying the "printed original" using an image processing device with a copying function. Therefore, for example, a copy of the aforementioned "copy of resident registration card" would be considered a "copy of original."

[0009] [Embodiment 1] <System configuration> FIG. 1 is a block diagram showing the configuration of a printing system according to this embodiment. As shown in FIG. 1, this printing system has an MFP (Multi Function Printer) 10 as a printing device and a PC 20 as its host device. The MFP 10 has multiple functions, such as a printer function and a scanner function, and also has a copy function that links these two functions. The MFP 10 may also have a function to save and transmit image data to be printed and a function to send and receive faxes. The hardware configurations of the MFP 10 and the PC 20 will be described below.

[0010] The MFP main body 10 is mainly composed of the following elements: The CPU 11 is an arithmetic processing unit that controls the entire MFP 10 and executes, for example, a copy process (described below) in accordance with programs stored in the ROM 13 and RAM 12. The RAM 12 is a volatile storage device that temporarily stores programs and data. The ROM 13 is a non-volatile storage device that stores table data and programs used in various processes (described below). The data transfer interface (DATA TRANSFER I / F) 14 controls the sending and receiving of data to and from the PC 20. The head controller (HEAD Controller) 15 controls the heating operation of a heater mounted on a print head (not shown) based on print data to eject ink. The image processing accelerator (IMAC) 16 is an arithmetic processing unit that can execute image processing faster than the CPU 11. Note that the image processing accelerator 16 is not necessarily required; the table parameter creation process and image processing described above may be executed solely by the CPU 11, depending on the printer specifications. A scanner controller 17 controls the illumination of an LED mounted on a scanner unit (not shown), acquires light intensity information from the scanner unit, and controls writing to RAM 12. This allows the image of a document set on a document table (not shown) to be read. A motor controller 18 controls multiple motor units (not shown) to move the recording head relative to the recording paper and move the scanner unit relative to the document. The configuration of MFP 10 is not limited to that shown in the figure, and may include, for example, a network interface for connecting to an external network and communicating with other external PCs, etc.

[0011] The PC 20 is primarily composed of the following elements: The CPU 21 is an arithmetic processing unit that controls the entire PC 20 and executes, for example, the printing of an original document (described later) in accordance with programs stored in the HDD 23 and RAM 22. The RAM 22 is volatile storage and temporarily stores programs and data. The HDD 23 is nonvolatile storage and also stores programs and data. The data transfer interface (DATA TRANSFER I / F) 24 controls the transmission and reception of data with the MFP 10. This data transmission and reception can be performed using a wired connection such as USB, IEEE 1394, or LAN, or a wireless connection such as Bluetooth or WiFi. The keyboard and mouse interface (KEYBOARD MOUSE I / F) 25 is an interface that controls a human interface device (HID) such as a keyboard or mouse. The display interface (DISPLAY I / F) 26 controls the display on a display (not shown). A network interface (Network I / F) 27 connects the PC 20 to an external network, communicates with one or more external PCs, and issues document ID matching requests, result requests, document data requests, and the like.

[0012] <Original printing process> Next, the flow of printing an original electronic document will be explained with reference to the flowchart in Figure 2. The series of processes shown in the flowchart in Figure 2 begins when a user selects an electronic document to be printed via the UI of a predetermined printing application installed on PC 20 and instructs printing that includes embedding information indicating its authenticity. In the following explanation, the symbol "S" denotes a step.

[0013] In S201, data of an electronic document designated to be printed is acquired. In this embodiment, various original electronic documents are stored and managed in an external PC (not shown). PC 20 transmits an acquisition request for a specific electronic document to the external PC via network interface 27, and receives and acquires data for printing the designated electronic document, specifically PDL data, from the external PC. Here, PDL stands for Page Description Language, and is composed of a set of drawing commands for each page. Types of drawing commands are defined for each PDL specification, and in this embodiment, the following three types are mainly used as examples. TEXT drawing command: (X1, Y1, color, font information, string information) BOX drawing command: (X1, Y1, X2, Y2, color, fill shape) IMAGE drawing command: (X1, Y1, X2, Y2, image file information) In addition to the three types mentioned above, there are also the DOT drawing command for drawing points, the LINE drawing command for drawing lines, and the CIRCLE drawing command for drawing arcs, and these drawing commands are used depending on the application. Commonly used PDLs include PDF (Portable Document Format) proposed by Adobe, XPS proposed by Microsoft, and HP-GL / 2 proposed by HP.

[0014] Figure 3 is a diagram showing an example of an image of one page of an electronic document (hereinafter referred to as a "page image"). The size of the page image 300 shown in Figure 3 is assumed to be 600 pixels wide and 900 pixels high. The PDL corresponding to the page image 300 is shown below. <PAGE=001> <text>50,50, 200, 100, BLACK, STD-18, "ABCDEFGHIJKLMNOPQR”< / text> <text>50,100, 200, 150, BLACK, STD-18, "abcdefghijklmnopqrstuv”< / text> <text>50,150, 200, 825, BLACK, STD-18, "1234567890123456789”< / text> <box> 50, 300, 200, 450, GRAY, STRIPE< / box> 250, 300, 550, 800, “PORTRAIT.jpg”

[0015] The first line of the above PDL<PAGE=001> is a tag that indicates the number of pages. Normally, PDL is designed to be able to describe multiple pages, and tags that indicate page breaks are written in the PDL. In the above example, up to the 9th line indicates that it is the first page. If there is a second page,<PAGE=002> The second line will be written as follows: <text> From the third line< / text> The text is drawn up to the end of the line. In the text drawing command, the first two coordinates indicate the coordinates (X1, Y1) of the top left corner of the drawing area, and the next two coordinates indicate the coordinates (X2, Y2) of the bottom right corner of the drawing area. Next, it is written that the color is "BLACK (black: R=0, G=0, B=0)", the font is "STD (standard)", the font size is 18 points, and the string to be drawn is "ABCDEFGHIJKLMNOPQR". The fourth line <text> From the 5th line< / text> and the sixth line <text> From the 7th line< / text> The strings to be drawn correspond to "abcdefghijklmnopqrstuv" and "1234567890123456789". <box> from< / box>The lines up to are the BOX drawing command, with the first two coordinates indicating the upper left coordinate (X1, Y1) of the drawing start point, and the next two coordinates indicating the lower right coordinate (X2, Y2) of the drawing end point. The color is then specified as "GRAY (gray: R=128, G=128, B=128)" and the fill style as "STRIPE." Line 9 is the IMAGE drawing command. The first two coordinates indicate the upper left coordinate (X1, Y1) of the drawing area, and the next two coordinates indicate the lower right coordinate (X2, Y2) of the drawing area. It also states that the file name of the image in the specified drawing area is "PORTRAIT.jpg." Note that ".jpg" is the file extension, indicating that it is a JPEG file, a commonly used image compression format. Finally, line 10 indicates that the drawing of the page has finished. Note that the document data actually retrieved is often a combination of PDL data with font data and image files added. This is because when font data and image files are managed separately, the text and image portions cannot be formed with drawing commands alone, and there is insufficient information to form the image to be printed. In this step, document data including this PDL data is obtained by accessing an external PC.

[0016] In S202, information indicating the authenticity of the electronic document to be printed is acquired. In this embodiment, a request for information indicating the authenticity of the electronic document is sent to the external PC via the network interface 27, and the information is received and acquired. Here, document ID information is used as the information indicating the authenticity of the electronic document. The document ID information is information of a predetermined number of bits (32 bits in this embodiment) calculated using a hash function based on the document data (a combination of PDL data, font data, and image files). If even one byte of the digital data constituting the electronic document is changed, different document ID information is calculated, and therefore a unique document ID is associated with each electronic document. The risk of document ID information tampering may be reduced by having multiple external PCs collaborate to store and manage electronic document files and document ID information, employing a system configuration such as a blockchain. If the PC 20 has sufficient resources and can manage document ID information internally, internal matching processing may also be performed.

[0017] In S203, rendering processing is performed based on the document data acquired in S201. In this rendering processing, each drawing command included in the PDL data is executed to generate image data in bitmap format, which is composed of color information for each pixel. For example, in the case of the page image 300 shown in FIG. 3, a bitmap image of 600 x 900 pixels is generated. In this embodiment, each pixel of the bitmap image has 256 gradations, each with 8 bits for RGB.

[0018] In S204, the document ID information obtained in S202 is embedded in the bitmap image generated in S203. The process of embedding document ID information in a bitmap image is called "multiplexing" or "multiplexing encoding." When attempting to copy a printout obtained by printing a bitmap image that has undergone this multiplexing process, the document ID information can be extracted (decoded) from a scanned image of the printout, making it possible to determine whether the printout to be copied is the "original printout."

[0019] Handling information on an information processing device such as the PC 20 essentially means handling binary data. Binary data is information expressed as a combination of "0" and "1," and specific meanings are created when these "0"s or "1"s are connected in succession. For example, when handling the English word "hello" as binary data, using Shift-JIS, one of the character codes, the letter "h" corresponds to the binary data "01101000." Similarly, "e" corresponds to the binary data "01100101," "l" to "01101100," and "o" to "01101111." In other words, the string "hello" can be expressed in binary data as "0110100001100101011011000110110001101111." Conversely, if you can obtain the binary data "0110100001100101011011000110110001101111", you can obtain the text information representing the English word "hello." Based on this idea, you can see that multiplexing can be achieved by embedding specified data within an image so that it can be determined to be either "0" or "1."

[0020] <Multiplexing processing> Figures 4(a) and (b) show examples of masks consisting of 8x8 pixels used to generate "0" and "1." Figure 4(a) shows a mask for generating "0," and Figure 4(b) shows a mask for generating "1." By adding the values ​​in the mask to the pixel values ​​of a bitmap image, a periodic pattern can be created for each 8x8 pixel region in the image. As described above, the bitmap image of this embodiment is represented by 8 bits per color, and each pixel is assigned a value between 0 and 255. Values ​​outside the range of 0 to 255 cannot be used as image data, so if the addition result is less than 0 or a value greater than or equal to 256, the value of the addition result is generally replaced with 0 or 255 to bring it within the valid range. In the masks of Figures 4(a) and (b), pixel values ​​are changed by "-10" or "0." However, if all pixel values ​​in the bitmap image corresponding to the masked region are "0," all values ​​in that region will be "0." Note that, although the case of 8 bits per color is described here, this is not limiting. When dealing with digital images, no matter how many bits are used to represent them, there is a valid range, and changes that go outside that range cannot be made.

[0021] Figures 5(a) and (b) are diagrams visually showing the patterns formed in a bitmap image by the masks in Figure 4(a) and (b) above. In the masks in Figures 4(a) and (b), the position of "-10" is represented by solid black, and the position of "0" is represented by hatching. As can be seen from Figures 5(a) and (b), in the bitmap image after the masks are applied, a diagonal line descending to the left appears as a "pattern representing 0" and a diagonal line descending to the right appears as a "pattern representing 1."

[0022] Here, pseudo-code for alternately applying the masks in FIG. 4(a) and FIG. 4(b) to the entire bitmap image is shown below.

[0023] 01: int i, j, k, l; 02: int width = 600, height=900; 03: unsigned char *data = image data; 04: int **maskA = maskData; 05: bool isMaskA = true; 06: for(j = 0; j < height; j+=8){ 07: for(i = 0; i < width; i+=8){ 08: for(k = 0; k < 8; k++){ 09: for(l = 0; l < 8; l++){ 10: if(isMaskA == true) { 11: data[(i+k)+(j+l)*width] += maskA[k][l]; 12:} 13:} 14:} 15:} 16:}

[0024] In this embodiment, the above pattern data is embedded only in the B value of the RGB values ​​of each pixel in the bitmap image. This is because when printing on a white area of ​​paper using four types of ink (CMYK), the Y ink has lower visibility than the CMY inks. When multiplexing, it is preferable to minimize the impact of the embedded pattern on the original image, i.e., to make it as inconspicuous as possible. RGB, the three primary colors of light, and CMY, the three primary colors of color, are complementary to each other, with R and C, G and M, and B and Y having opposite hues, respectively. This means that modulating the B value of the RGB values ​​significantly changes the amount of Y ink used. Therefore, in this embodiment, only the B value of the RGB values ​​is modulated to control the density fluctuation of the Y ink to be large.

[0025] In the case of the page image 300 shown in Figure 3, the white paper area is sufficiently large, and the area excluding the TEXT drawing area 301, BOX drawing area 302, and IMAGE drawing area 303 is the white paper area. In this case, the masks shown in Figures 4(a) and 4(b) may not properly embed pattern data in each drawing area other than the white paper area. For example, if a drawing area is solid black (R=0, G=0, B=0), the embedding result will also be solid black (R=0, G=0, B=0). Therefore, when embedding pattern data outside the white paper area, especially in the image drawing area, it is preferable to apply masks such as those shown in Figures 6(a) and 6(b). The masks in Figures 6(a) and 6(b) are masks that change each pixel of the bitmap image by "-10," "0," and "+10." In the case of the masks shown in Figures 6(a) and 6(b), even if all pixel values ​​in the bitmap image corresponding to the masked area are "0," the pixel values ​​of the corresponding area will ultimately be "0" or "+10." Therefore, pattern data can be embedded into all pixels in the bitmap image. Figures 7(a) and 7(b) visually illustrate the patterns imparted to the bitmap image by the masks shown in Figures 6(a) and 6(b). In the masks shown in Figures 6(a) and 6(b), the "-10" position is represented by solid black, the "0" position by hatching, and the "+10" position by solid white. As can be seen from Figures 7(a) and 7(b), the bitmap image after the masks are applied will have multiple diagonal lines sloping downward to the left with different densities as the "0" pattern and multiple diagonal lines sloping downward to the right with different densities as the "1" pattern. Note that, since there is a high probability that there will be many blank areas in the text and box drawing areas, it is preferable to apply the masks shown in Figures 4(a) and 4(b). However, even for TEXT and BOX drawing areas, the masks shown in Figures 6(a) and 6(b) may be appropriate depending on the specified color, character thickness, etc. Also, even for IMAGE drawing areas, there may be cases where the area is very close to the white paper area. For this reason, the type of mask to apply may be determined by obtaining a density histogram for each drawing area.For example, if the maximum value of the RGB value is greater than the threshold, the masks in Figures 4(a) and 4(b) are applied, and if it is equal to or less than the threshold, the masks in Figures 6(a) and 6(b) are applied. Alternatively, the input color space is L. * a * b * In this case, the luminance is L * The mask to be applied may be switched by comparing the value with a threshold value, thereby enabling reliable embedding to be achieved.

[0026] Returning to the explanation of the flowchart in FIG.

[0027] In S205, a print image is generated based on the multiplexed bitmap image (hereinafter referred to as the "multiplexed image") generated in S204. A known method may be used to generate this print image. In this embodiment, each pixel of the multiplexed image is subjected to color conversion processing, color separation processing, output characteristic conversion processing, and quantization processing. Each processing will be briefly described below.

[0028] <Color conversion processing> Color conversion processing converts the pixel values ​​(RGB values) of a multiplexed image so that they can be optimally reproduced on the MFP 10. Generally, color values ​​specified in PDL drawing commands are set to be optimally displayed on a display, and if those color values ​​are output to a printer as is, the output will have a different color tone than what is seen on the display. Therefore, color conversion processing is performed to absorb the color differences between the two. This color conversion processing uses a three-dimensional lookup table (LUT) that associates combinations of input pixel values ​​(Rin, Gin, Bin) in the multiplexed image with combinations of optimal output pixel values ​​(Rout, Gout, Bout). Currently, the input values ​​Rin, Gin, and Bin each have 256 gradations. Therefore, it is ideal to prepare a table (Table1

[0256]

[0256]

[0256] [3]) with a total of 16,777,216 pairs of 256 x 256 x 256 output values ​​(Rout, Gout, Bout) as shown below. Rout = Table1[Rin][Gin][Bin][0] Gout = Table1[Rin][Gin][Bin][1] Bout = Table1[Rin][Gin][Bin][2] However, known techniques for reducing the table size may be used, such as reducing the number of grids in the LUT from 256 grids to, for example, 16 grids, and determining the output value by interpolating table values ​​between the grids.

[0029] <Color separation processing> Color separation processing is a process that converts the output values ​​Rout, Gout, and Bout of the color conversion processing into output values ​​for each ink color (here, the four colors CMYK) to be recorded on paper using the inkjet method. There are various ways to achieve this color separation processing. In this embodiment, a three-dimensional lookup table (Table 2

[0256]

[0256]

[0256] [4]) is used that associates the combinations of the above-mentioned output pixel values ​​(Rout, Gout, Bout) with the combinations of ink color pixel values ​​(C, M, Y, K) shown below. C = Table2[Rout][Gout][Bout][0] M = Table2[Rout][Gout][Bout][1] Y = Table2[Rout][Gout][Bout][2] K = Table2[Rout][Gout][Bout][3] In this case, it is preferable that only the Y value of the CMYK values ​​corresponding to the result of modulating the paper-white region in the multiplexing process of S204 has a value greater than 0. More specifically, it is preferable that the Y value of the CMYK values ​​corresponding to the region where pixel values ​​of R=255, G=255, B=255 are converted to pixel values ​​of R=255, G=255, B=245 has a value greater than 0 and a value closer to 0 that is smaller than the Y value, as explained in S204. The reason for this is that it is desirable to reduce the visibility of the embedded pattern. Note that, as with the color conversion process described above, known techniques for reducing the table size may be used.

[0030] <<Output characteristic conversion processing>> The output characteristic conversion process converts the density of each CMYK ink color into a print dot rate. Specifically, for example, the density of 256 levels for each color is converted into a dot rate of 1024 levels for each color. This output characteristic conversion process uses a one-dimensional lookup table (Table 3[4]

[0256] ) that sets the print dot rate (Cout, Mout, Yout, Kout) shown below, which corresponds to the density of each ink color. Cout = Table3[0][C] Mout = Table3[1][M] Yout = Table3[2][Y] Kout = Table3[3][K] As in the case of the color conversion process and color separation process described above, known techniques for reducing the table size may be used.

[0031] Quantization processing The quantization process converts the printing dot count rates (Cout, Mout, Yout, Kout) of each ink color described above into the quantized values ​​(Cdot, Mdot, Ydot, Kdot) shown below, which represent whether the printing dot of each pixel is on or off. Cdot = Halftone[Cout][x][y] Mdot = Halftone[Mout][x][y] Ydot = Halftone[Yout][x][y] Kdot = Halftone[Kout][x][y] The above are quantized values ​​for the dithering method, and by comparing them with threshold values ​​in the dithering matrix corresponding to each pixel position, a value indicating whether the recording dot of each ink color is on or off can be obtained. In this case, the occurrence probability of each recording dot is Cout / 1023, Mout / 1023, Yout / 1023, Kout / 1023. Note that the quantization processing method is not limited to the dithering method, and other methods such as error diffusion may also be used.

[0032] By executing the above-described processes in order, a print image is generated from the multiplexed image. Returning to the explanation of the flowchart in FIG.

[0033] In S206, the print image data generated in S205 is sent together with the print instruction to the MFP 10 or another printer (not shown), and the print process is executed by the destination printer.

[0034] The above is the process leading up to the generation of the original print. In this embodiment, the B value of the RGB values ​​is modulated during the multiplexing process (S204). However, modulation can also be performed on the CMYK values. In this case, the pixel values ​​of the paper white area are Y=0, M=0, C=0, and K=0, so positive values ​​must be used for modulation. In the case of the masks illustrated in Figures 4 and 6, the signs of the modulation values ​​in the mask are inverted, i.e., "-10" becomes "+10" and "+10" becomes "-10." Modulation performed on the CMYK values ​​in this way provides greater control over limiting the ink applied to the paper white area to Y ink only. On the other hand, modulation performed on the RGB values ​​provides greater control over suppressing hue fluctuations when embedding in the image drawing area. Therefore, it is preferable to select an appropriate modulation method depending on the characteristics of the recording process, such as electrophotography or inkjet, and the ratio of paper white, text, and image areas within the page image to be printed.

[0035] <Copying of "original printouts"> Next, the flow of copying the "original printout" that is the result of the above-mentioned original printing process using the MFP 10 will be described with reference to the flowchart in Fig. 8. The series of processes shown in the flowchart in Fig. 8 begins when a user places the original printout on a platen (not shown) and issues a copy instruction via the UI (user interface) of the MFP 10. In the following description, the symbol "S" denotes a step.

[0036] First, in S811, a printed material to be copied that is set in the MFP 10 is read by a built-in scanner unit (not shown). To read this printed material, LED light is irradiated onto the printed material placed on the platen, and the reflected light is converted into an analog electrical signal by an imaging element such as a CCD that faces each pixel.

[0037] In the next step S812, the analog electrical signal obtained in S811 is digitized to obtain a bitmap image in the RGB color space. At this time, image processing such as MTF correction, input correction, shading correction, and color conversion is performed on each pixel of the bitmap image. Each process is briefly described below.

[0038] <MTF correction processing> MTF (Modulation Transfer Function) correction is a correction process related to the resolution of the scanner unit's reading performance. When an image is optically read using a scanner unit, the image becomes blurred due to factors such as deviation from the focus position or the performance limits of the lens itself, so a certain degree of restoration is performed using filter processing, etc. In this case, if strong enhancement processing is used to completely restore the image, image defects such as blown-out highlights and enhancement of image noise and dust pixels will become more noticeable, so it is necessary to design the filter strength by balancing the image quality improvement with the defects. Below is an example of an edge enhancement filter that multiplies the center of the image by 5 and multiplies the pixel values ​​of the top, bottom, left, and right pixels by -1. R'[x][y] = R[x][y]×5-R[x-1][y]-R[x+1][y]-R[x][y-1]-R[x][y+1] G'[x][y] = G[x][y]×5-G[x-1][y]-G[x+1][y]-G[x][y-1]-G[x][y+1] B'[x][y] = B[x][y]×5-B[x-1][y]-B[x+1][y]-B[x][y-1]-B[x][y+1]

[0039] <Input correction processing> Input correction is the process of converting the output value of the CCD (image sensor), which is originally a photon quantity, into a brightness that matches the sensitivity of the human eye. This converts, for example, an R'G'B' signal with 4096 gradations for each color into a color intensity value (R', G', B') with 1024 gradations for each color. This conversion uses the one-dimensional lookup table shown below (Table 4[4]

[4096] ), which sets the optimal recording dot count rate for each ink color density. R” = Table4[0][R'] G” = Table4[1][G'] B” = Table4[2][B'] However, known techniques for reducing the table size may be used, such as reducing the number of grids in the LUT from 4096 grids to, for example, 256 grids, and determining the output value by interpolating table values ​​between the grids.

[0040] <Shading correction processing> Shading correction reduces uneven color and density caused by differences in reading sensitivity at each pixel position due to manufacturing and assembly variations in the lenses, LEDs, CCDs, and other components that make up the scanner. For example, R"G"B" signals with 1024 gradations for each color are converted into color intensity values ​​(R"', G"', B"') with 256 gradations for each color. This conversion uses the following one-dimensional lookup table (Table 5[x][3]

[1024] ), which specifies the density adjustment value for each pixel position in the X direction (the direction in which the scanner lens is arranged). R”' = Table5[x][0][R”] G”' = Table5[x][1][G”] B”' = Table5[x][2][B”] As in the case of the input correction process described above, known techniques for reducing the table size may be used.

[0041] <Color conversion processing> The 256-level color intensity values ​​(R"', G"', B"') for each color calculated through the processing up to this point are values ​​specific to the scanner unit, as opposed to when printing. The color conversion processing here converts the values ​​specific to the scanner unit into RGB values ​​(Rout, Gout, Bout values) suitable for display on the display. The input values ​​R"', G"', and B"' each have 256 levels. Therefore, for this conversion, the following three-dimensional lookup table (Table 6

[0256]

[0256]

[0256] [3]) is used, which has a total of 16,777,216 sets of output values ​​(256 x 256 x 256). Rout = Table1[R”'][G”'][B”'][0] Gout = Table1[R”'][G”'][B”'][1] Bout = Table1[R”'][G”'][B”'][2] As in the case of the input correction process and the shading correction process described above, a known method for reducing the table size may be used. In this way, a bitmap image is obtained.

[0042] In the next step S813, a process for extracting document ID information is executed for the bitmap image acquired in step S812. Specifically, a process is executed in units of a predetermined area (here, 8x8 pixels) to determine whether the pattern indicating "0" or "1" described above is formed within the bitmap image, and the process extracts the numerical sequence information represented by "0" and "1." By repeatedly applying this process to the entire bitmap image, the document ID information embedded by the multiplexing process when the original was printed is decoded. Information embedded by the multiplexing process is sometimes called "embedded information" or "multiplexed information." The process for decoding the embedded information will now be described in detail.

[0043] <Decoding process of embedded information> First, the location where the embedded information is embedded within the acquired bitmap image is detected. The embedded location can be detected by analyzing the spatial frequency characteristics of each unit area (here, an 8x8 pixel area). Figure 9 shows the spatial frequency characteristics of the pattern used for embedding. The horizontal axis represents horizontal frequency, and the vertical axis represents vertical frequency, with the frequency increasing with increasing distance from the origin. In this embodiment, two types of patterns corresponding to "0" and "1" are embedded in the image (see Figures 5 and 7 above). In this case, a value of "10" is subtracted from the B component of each of the RGB color components (in the case of the mask in Figure 4; in the case of the mask in Figure 6, an addition and subtraction are performed). As a result, for example, the left-sloping pattern formed in the image by the mask in Figure 4(a) generates a large power spectrum on line A02. Similarly, the right-sloping pattern formed in the image by the mask in Figure 4(b) generates a large power spectrum on line A01. Therefore, by detecting this power spectrum, data "0" or "1" can be extracted. It should be noted that, as a preprocessing step for power spectrum detection, edge detection can be performed to enhance the power spectrum.

[0044] The data extraction using frequency analysis described above requires accurate extraction of the analysis area from the image data, so a process to correct for misalignment of the coordinate position is also performed. For example, first, extraction of a unit area from a bitmap image and frequency analysis are repeated vertically and horizontally, shifting the area by one pixel at a time. If the image size is 600 pixels wide and 900 pixels high, this process is repeated a total of 64 times. The point where the spectrum is strongest is then set as the reference position for extraction. Then, by extracting the embedded information based on the reference position, the embedded sequence of "0"s or "1"s can be obtained with high accuracy.

[0045] In this embodiment, as explained in S204, the information to be embedded is text data, and its character code is a value quantified in "Shift JIS." In this case, in 1-byte Shift JIS code (half-width characters), for example, the alphabet "h" corresponds to binary data "01101000," "e" corresponds to "01100101," "l" corresponds to "01101100," and "o" corresponds to "01101111." Therefore, if the extracted numeric sequence of the embedded information is "0110100001100101011011000110110001101110001101111," the English word string "hello" is obtained. In this way, the document ID information embedded as embedded information is extracted.

[0046] Returning to the explanation of the flowchart in FIG.

[0047] In S814, it is determined whether extraction of document ID information was successful in S813. If extraction of document ID information was successful, the process proceeds to S815; if not, the process proceeds to S820. There are two possible cases where extraction fails: one is when document ID information was not embedded in the printed material to be copied (Possibility 1). The other is when document ID information was embedded in the printed material to be copied, but the printed material was dirty or had handwritten characters added later, making it impossible to correctly detect the predetermined pattern representing the embedded information (Possibility 2). In the case of Possibility 1, the process proceeds directly to S820. In the case of Possibility 2, the user may be notified by displaying a message stating, "You are about to copy an authentic document (original printed material) with an embedded document ID." This allows the user to know that they are attempting to copy a potentially inauthentic printed material, and gives them the opportunity to choose whether to cancel the copy operation or otherwise. For example, if the result obtained from the extraction process is between 1 bit and 31 bits inclusive out of the total 32 bits of document ID information, it can be determined that the possibility is 2 and a notification like this can be sent. Note that it is not completely impossible that a single pattern similar to the above-mentioned predetermined pattern is accidentally included in the image. Taking such rare cases into consideration, it is preferable to determine the possibility as 1 when less than 16 bits, half of 32 bits, are extracted, and to determine the possibility as 2 when between 16 bits and 31 bits inclusive are extracted.

[0048] In the following S815, a process of verifying the successfully extracted document ID information is performed. In this embodiment, similar to the document ID acquisition process in S202, the MFP 10 accesses the external PC to request a verification and obtains the verification result. Specifically, a verification request as to whether the extracted document ID information is officially registered and valid is transmitted via the network I / F 17, and the verification result is received and obtained from the external PC. Note that, similar to S202 described above, multiple external PCs may cooperate to manage document ID information, and a system configuration such as a blockchain may be adopted.

[0049] Next, in S816, the process is assigned depending on whether the document ID information extracted in S813 is officially registered and valid as a result of the comparison. If the document ID is valid, the process proceeds to S817, and if the document ID is invalid, the process proceeds to S820. If the document ID is invalid, the user may be notified by displaying a message saying, "You are attempting to copy an unauthentic document with an invalid document ID." This gives the user an opportunity to choose whether to cancel the copying operation, etc.

[0050] In S817, a check is performed to determine whether the printed material to be copied has been tampered with (tamper check process). The general flow of this tamper check process is as follows: First, the MFP 10 sends a document data acquisition request together with document ID information to an external PC via the network I / F 17. Then, the MFP 10 receives and acquires document data (PDL data) associated with the document ID information from the external PC, and performs rendering on the document data. This is the preprocessing. The bitmap image obtained in the rendering process is compared with the bitmap image acquired in S812 to determine whether tampering has occurred. The tamper check process will now be described in detail with reference to another flowchart shown in FIG. 10. In the description of the flow in FIG. 10, the bitmap image obtained in the rendering process will be referred to as a "rendered image," and the bitmap image obtained in the document reading process will be referred to as a "scanned image."

[0051] <Details of the tamper check process> In S1001, a counter that counts pixels determined to be tampered with in S1010 (described later) is initialized. Specifically, the count number of the counter (number of tampered pixels) is set to "0."

[0052] In the next step, S1002, a normalization process is performed on the scanned image acquired in S812. This normalization process is performed to match the dynamic range between the scanned image and the rendered image. For example, the brightest part of a scanned image generally represents the color of the paper of the printed material being scanned, and in principle, has a certain density value. On the other hand, the brightest part of a rendered image is a pixel with pixel values ​​R=255, G=255, and B=255. The darkest part of a scanned image generally represents the black color of ink or toner, and in principle, has a brightness value due to some reflected light. On the other hand, the brightest part of a rendered image is a pixel with pixel values ​​R=0, G=0, and B=0. As such, there is an inherent difference between the brightest and darkest colors in the two images. Furthermore, if the printed material is output in color, there is also a difference in the color tone between the two images. For example, the most vivid red that can be printed has a lower saturation in the scanned image than the pixel values ​​(R=255, G=0, B=0) that represent the most vivid red in the rendered image. Therefore, each pixel value (RGB value) of the scanned image is normalized using the following equations (1) to (3) to find new pixel values ​​(Rnorm, Gnorm, Bnorm). Rnorm = (R - Rd) / (Rw-Rd)×255 Equation (1) Gnorm = (G - Gd) / (Gw-Gd)×255...Equation (2) Bnorm = (B - Bd) / (Bw-Bd)×255...Equation (3)

[0053] By normalization processing using the above formulas (1) to (3), the RGB values ​​of the brightest color (white) in the scanned image become R=255, G=255, B=255, and the RGB values ​​of the darkest color (black) become R=0, G=0, B=0.

[0054] In the next step S1003, the normalized scanned image is subjected to a filter process, which performs stronger edge enhancement to make it easier to compare with the rendered image.

[0055] In the next step S1004, the image pattern representing the embedded information is removed from the scanned image after filtering. The print image generated in step S205 in the flow of FIG. 2 differs from the original document image (the image of the original electronic document) due to the multiplexing process (S203). Therefore, this process is performed to eliminate these differences as much as possible. Specifically, the pattern data embedded by the multiplexing process is subtracted from the scanned image. This brings the scanned image closer to the state before the multiplexing process. In this embodiment, this is achieved by further embedding values ​​obtained by multiplying each value in the masks in FIGS. 4 and 6 by "-1" into the scanned image.

[0056] Next, in S1005, the scanned image from which the embedded information has been removed is compared with the rendered image. Specifically, the rendered image and the scanned image from which the embedded information has been removed are compared pixel by pixel to determine the difference values ​​(ΔR, ΔG, ΔB) expressed by the following equations (4) to (6). ΔR = |R[x][y] of rendered image - R[x][y] of scanned image| Equation (4) ΔG = |G[x][y] of rendered image - G[x][y] of scanned image| Equation (5) ΔB = |B[x][y] of rendered image - B[x][y] of scanned image| Equation (6)

[0057] Next, in S1006, it is determined whether the difference values ​​(ΔR, ΔG, ΔB) obtained in S1005 exceed threshold values. In this embodiment, these are compared with threshold values ​​(Rth, Gth, Bth) previously set for each of the RGB channels, and if any of ΔR, ΔG, or ΔB exceeds the corresponding threshold, the process proceeds to S1007; if not, the process proceeds to S1008. The threshold values ​​are determined taking into consideration the characteristics of the scanner unit and the printing device, and in this embodiment, values ​​such as Rth=Gth=Bth=64 may be set.

[0058] In S1007, the count value of the counter indicating the number of tampered pixels is incremented (+1). After the counter is incremented, the process proceeds to S1008. Then, in S1008, it is determined whether or not comparison of pixel values ​​has been completed for all pixels. If there are any unprocessed pixels, the process returns to S1005 to continue processing, and if comparison of pixel values ​​has been completed for all pixels, the process proceeds to S1009.

[0059] In S1009, it is determined whether the count value of the counter indicating the number of tampered pixels exceeds a threshold value. In this case, the threshold value may be set to, for example, 3% of the total number of pixels, assuming cases where dust is mixed in when the scanned image is acquired. If the count value exceeds the threshold value, the process proceeds to S1010; if not, the process proceeds to S1011.

[0060] In S1010, the result of the tampering check process is determined to be no tampering, and the process ends. On the other hand, in S1011, the result of the tampering check process is determined to be tampered, and the process ends.

[0061] The above is the content of the tampering check process. Note that in this embodiment, the comparison is performed on a pixel-by-pixel basis, but this is not limited to this. For example, the comparison may be performed on a larger scale, such as by comparing the average values ​​in units of multiple pixel blocks. When the tampering check process is completed, the process proceeds to S818 in FIG. 8.

[0062] Returning to the explanation of the flowchart in FIG.

[0063] In S818, the bitmap image obtained in S812 is assigned to a different process depending on the result of the tampering check process. If no tampering is detected, the process proceeds to S819, and if tampering is detected, the process proceeds to S820. If the result of the tampering check process indicates tampering, the user may be notified by displaying a message stating, "You are attempting to copy a tampered document." This gives the user an opportunity to choose whether to cancel the copy operation or not.

[0064] Next, in S819, a process is performed to reconstruct information indicating the authenticity of the document that was embedded in the bitmap image acquired in S812. Specifically, a process is performed to reinforce weak portions of the specified pattern data embedded in the bitmap image. As described in the problem to be solved, this is because the embedded pattern data obtained by reading the printed material may be degraded. FIG. 11(a) shows an example of a degraded pattern representing the "0" in FIG. 5(a). In the original pattern in FIG. 5(a), a diagonal line sloping downward to the left should appear due to solid black pixels, but a hatched pixel 1101 appears in part of it. This indicates a pixel where the desired density could not be achieved. This reduces the difference with other hatched pixels, resulting in a broken diagonal line sloping downward to the left. In other words, the embedding strength of the pattern data is weakened accordingly. If a pattern like that shown in FIG. 11(a) is detected in the bitmap image acquired in S812, it can be determined that the strength of the embedded information is weakened. When the strength of the embedded information is weakened in this way, a pattern reinforcement mask is used to reinforce the deteriorated portions of the extracted pattern data. Figure 11(b) shows the pattern reinforcement mask corresponding to Figure 11(a). The pattern reinforcement mask in Figure 11(b) further changes the pixel position 1102 where the desired density could not be achieved by "-5." The degree of change may be set based on the difference between the pixel 1101 where the desired density could not be achieved and the solid black pixel, or a predetermined value may be set in advance. Then, the pattern reinforcement mask described above is applied to image areas of the bitmap image where the strength of the embedded information is determined to be weak, and the pattern data is reinforced. Note that even if error correction is performed during document ID extraction in S813, the embedded information may be determined to be weakened, and a similar reinforcement process may be performed on the corrected portions. Alternatively, the pattern reinforcement mask may be applied only to specific color planes where pattern data is embedded.

[0065] In the next step S820, a print image is generated based on the bitmap image for which the embedded information has been reconstructed in step S819. This process corresponds to step S205 in the flow shown in Figure 2, and there are no particular differences, so a detailed explanation will be omitted. In the next step S821, a print process is performed using the print image generated in step S820.

[0066] The above is the process flow when copying an original printout. In this embodiment, an example has been described in which a pattern reinforcement mask is applied to the bitmap image read in S812 to reinforce the already embedded pattern data, thereby realizing sufficiently strong embedded information in the copy. However, this can also be achieved by the following method.

[0067] <Variation 1> After removing the embedded pattern data from the bitmap image acquired in S812, a new multiplexing process may be performed to embed information indicating the authenticity of the document. One method for removing the embedded pattern data from the bitmap image is to apply a smoothing filter of a predetermined size to the bitmap image scanned in S812. The filter size is set, for example, based on the size of the embedded pattern (e.g., 8x8 pixels). Alternatively, an inverse filter may be calculated from the embedded pattern and applied to the bitmap image. If the pattern is embedded in a white area of ​​the paper, a process may be performed to restore the pixel values ​​to the white paper state by background removal or the like. After removing the embedded pattern, the document ID information extracted in S813 may be embedded in the bitmap image using a method similar to S204 in the flow shown in Figure 2. If the pattern data is embedded only in a specific color plane, the pattern data removal process may be applied only to that specific plane.

[0068] <Variation 2> One method is to acquire the original document data based on the document ID information extracted in S813 and embed new document ID information in it. In this case, the document ID information extracted in S813 is sent to an external PC, the document data (PDL data) linked to the document ID information is acquired, rendering processing is performed using the acquired PDL data to obtain a bitmap image, and multiplexing processing is performed on the bitmap image to embed new document ID information.

[0069] By using any of the above methods, it is possible to embed information with sufficient strength in the copy.

[0070] [Embodiment 2] In the first embodiment, an example was described in which document ID information is multiplexed and embedded in units of 8x8 pixel regions. With this technique, a predetermined pattern is formed in the high frequency region of the image, making it possible to embed information in a form that is difficult for the user to see. Next, a mode in which document ID information is converted into a QR code and embedded in a form that is difficult for the user to see will be described as the second embodiment. Note that the following will only describe the embedding method, which is a difference from the first embodiment, and other descriptions will be omitted.

[0071] FIG. 12(a) shows an example of a QR code, and FIG. 12(b) shows the corresponding actual print pattern. In the example of FIG. 12(b), one dot is formed per 8x8 pixel area. The dot corresponding to the black pixel 1201 in the QR code of FIG. 12(a) corresponds to the dot 1202 in the print pattern shown in FIG. 12(b). In other words, no dots of the print pattern are formed at positions corresponding to white pixels in the QR code of FIG. 12(a). A specific process flow for achieving this is to first convert the document ID information into a QR code in the multiplexing process (S204) in the flow of FIG. 2, and then convert it into pattern data of a group of spaced dots and embed it in the rendering image in predetermined area units. Then, in the subsequent print image generation process (S205), the spaced dots are formed using only Y ink, making them less visible.

[0072] When copying a printed material in which document ID information has been embedded using the method of this embodiment, if yellow dots are formed in 8x8 pixel units in the bitmap image obtained by reading it, the QR code pattern can be extracted and decoded to extract the document ID information.

[0073] Furthermore, as an embedding method other than the QR code, for example, multiplexing may be performed by performing threshold modulation during quantization processing (see Patent Document 2).

[0074] (Other Examples) The technology of the present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

1. an extraction means for detecting a pattern at a position where embedded information is embedded in a scanned image obtained by scanning a printed matter to be copied, and extracting the embedded information based on the detected pattern; a generating means for generating a print image by reconstructing the pattern relating to the embedded information by reinforcing pixels at positions where the intensity of the pattern is weak, among the patterns at positions where the embedded information detected from the scanned image is embedded, when the printed matter to be copied is confirmed to be an authentic document based on the data linked to the embedded information extracted by the extracting means; a printing means for performing a printing process using the print image generated by the generating means; A printing system comprising:

2. the extraction means repeatedly performs a process of cutting out a unit area of ​​a predetermined size from the scanned image and a process of spatial frequency analysis of the cut-out unit area while shifting a coordinate position for cutting out the unit area, and extracts the embedded information based on the pattern detected at the reference position, using the coordinate position of the unit area where the power spectrum resulting from the spatial frequency analysis process is strongest as a reference position for cutting out; When the printout to be copied is confirmed to be an authentic document based on the data linked to the embedded information extracted by the extraction means, the generation means generates the print image by reconstructing the pattern related to the embedded information by reinforcing pixels at positions where the intensity of the pattern is weak among the patterns detected at the reference position in the scanned image.

2. The printing system according to claim 1.

3. The printing system described in claim 2, characterized in that the generation means performs the reinforcement using a mask that changes pixels that form the pattern detected at the reference position but did not achieve the desired density, treating those pixels as pixels at positions where the intensity of the pattern is weak.

4. The embedded information is expressed as binary data represented by a combination of 0 and 1, 4. The printing system according to claim 1, wherein the pattern detected from the scanned image is a pattern indicating 0 or 1.

5. It is connected to an external device via a network, The method further includes a verifying unit that verifies whether the embedded information extracted by the extracting unit is valid by transmitting the embedded information to the external device, When the result of the comparison indicates that the embedded information is valid and the printed matter to be copied is confirmed to be an authentic document based on the document data linked to the embedded information, the generating means generates the printed image by reconstructing the pattern related to the embedded information by reinforcing pixels at positions where the pattern strength is weak among the patterns detected from the scanned image.

2. The printing system according to claim 1.

6. A method for controlling a printing system, comprising: an extraction step of detecting a pattern at a position where embedded information is embedded in a scanned image obtained by scanning a printed matter to be copied, and extracting the embedded information based on the detected pattern; a generating step of generating a print image by reconstructing a pattern relating to the embedded information by reinforcing pixels at positions where the intensity of the pattern is weaker among the patterns at positions where the embedded information detected in the scanned image is embedded, when the printed matter to be copied is confirmed to be an authentic document based on the data linked to the embedded information extracted in the extracting step; a printing step of performing a printing process using the print image generated in the generating step; A control method comprising:

7. A program for causing a computer to function as each of the means of the printing system according to any one of claims 1 to 5.

Images