Digital watermark embedding and extraction method

Abstract

To provide a single digital watermarking method that embeds a watermark with printing resistance and enables authentication of a printed matter using watermark information, and in particular, to provide a method for accurately authenticating a primary printed matter (original) and a secondary printed matter (copy) by performing determination using the embedded digital watermark.SOLUTION: An image is divided into multiple regions, and a watermark is embedded in each region using a green noise pattern of different roughness. Watermark information and reliability are calculated for each region using a different discriminator, and the authenticity of the image is determined from the information.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a method for embedding and extracting invisible electronic watermarks that embed information in digital image data, and in particular to a method for embedding and extracting a watermark that allows the authenticity of a printed image to be determined by photographing the printed image with an embedded watermark using a camera such as a smartphone and reading the electronic watermark. [Background technology]

[0002] In recent years, the development of printers and printing devices has made it possible to output high-quality images, making it easy to reproduce document images that are exactly the same as the original, and has greatly contributed to the development of society in various fields. However, on the other hand, crimes such as counterfeit printing of banknotes and securities, and mass reproduction and sale of copyrighted creative works such as paintings and photographs have become rampant. In addition, in recent years, electronic government and local government electronic issuance of various certificates over the Internet, electronic tickets, etc., have often used paper media, and tampering and counterfeiting of printed documents have also increased, becoming a major social problem.

[0003] One such security technology is DRM (Digital Rights Management). DRM encrypts and encapsulates digital content and incorporates copy protection to prevent illegal duplication, modification, and unauthorized use of the content. However, when viewing the content, the encryption is decrypted and the raw image is viewed on a monitor, which means that it is easy to copy the content by taking screenshots, for example, creating a security loophole.

[0004] Furthermore, because DRM is intended for use with digital content, it cannot provide copy protection against the duplication or counterfeiting of printed securities or paper documents.

[0005] On the other hand, NFT (Non-Fungible Token) is a method to prove unique ownership of digital assets and content using blockchain technology. Since NFT is recorded on the blockchain, ownership and transaction history of digital assets and content can be confirmed in a transparent and reliable manner. Generally, NFT is applied to digitized electronic assets such as Bitcoin, but for physical assets (e.g. branded goods, expensive products, valuable items), NFT is used by linking physical items to NFT using NFC (Near Field Communication) using RFID (Radio Frequency Identification) tags.

[0006] Attaching an RFID tag to an object allows for individual identification of the object, linking physical items to the digital world, and combining it with an NFT allows the ownership of the physical item to be associated with the NFT on the blockchain, enabling original proof. However, because an RFID tag is embedded or attached to a physical item, if it is physically removed, the security function is easily lost.

[0007] On the other hand, unlike DRM, NFT, etc., digital watermarking methods embed information directly into document or image information and are always attached to the image data, making them difficult to remove, and making it possible to prevent and track unauthorized use or leakage. Among these, invisible digital watermarks embed information so that it cannot be detected by the human eye, so changes or deterioration in image quality cannot be detected, expanding the scope of their use.

[0008] Various methods have been proposed for digital watermarking technology depending on the form of use. A robust embedding algorithm with high resistance is required to prevent embedded information from being lost against various attacks that modify image data, such as image editing, processing, conversion, and compression. In particular, a watermarking method with an even more robust and special algorithm is required to prevent the watermark from being lost even in printed images output by a printer or other device.

[0009] On the other hand, when it comes to preventing tampering or authenticity verification, the authenticity is usually verified by losing the watermark information through copying, making the watermark unreadable. Since copying involves the double operation of scanning the original and outputting it on a printer, the spatial frequency response characteristics (MTF) of both the scanner and the printer are multiplied, and the high frequency components are greatly reduced. To achieve this efficiently, fragile watermarks are usually used. This is achieved, for example, by inserting watermark information into the LSB (lowest bit) of the image data. Of course, there is no print / scan resistance.

[0010] Because these are contradictory embedding methods, it has traditionally been necessary to switch between a highly robust digital watermarking method and a weaker watermarking method using dedicated equipment or software with different algorithms depending on the application or purpose, which makes operation and management complicated and is a factor preventing their use and widespread adoption. Therefore, a method is needed to extract and detect both using the same method or algorithm without changing the method.

[0011] Patent Document 1 and Non-Patent Document 1 have proposed a digital watermarking method that simultaneously incorporates embedding of such a highly resistant digital watermark and embedding of a weak digital watermark for tamper detection. This method has a means for multiplexing and simultaneously embedding two types of information, main information and sub information. The main information is embedded in the image data as a green noise pattern, which realizes a robust watermark that does not disappear even in printed images. The sub information is embedded in a high spatial frequency (checkerboard pattern), which is a weak watermark that allows tampering and the position of tampering to be detected. However, since the sub information is not print / scan resistant, tampering detection is performed using electronic data. For this reason, it cannot be used to determine the authenticity of printed matter.

[0012] In other words, this method cannot be used to determine the authenticity or genuineness of primary and secondary prints when a printed item such as banknotes, securities, or printed electronic tickets is used as the primary print (original) and a counterfeit copy made using a copying device or the like is used as the secondary print (copy). [Prior art documents] [Patent documents]

[0013] [Patent Document 1] Patent application 2018-88962 [Non-patent literature]

[0014] [Non-Patent Document 1] Naoto Kawamura: “Green noise diffusion type digital watermarking method enabling multiple embedding of information”, Journal of the Institute of Image Electronics Engineers of Japan, Vol. 48, No. 2, pp. 221-227 (2019). Summary of the Invention [Problem to be solved by the invention]

[0015] The challenge is to solve these problems and to obtain a single digital watermarking method that can embed a watermark with printing resistance and at the same time enable authentication of printed matter using the watermark information. In particular, the challenge is to provide a method for accurately authenticating a primary print (original) and a secondary print (copy) by identifying the primary print (original) and the secondary print (copy) using the embedded digital watermark. [Means for solving the problem]

[0016] Therefore, we attempt to develop a digital watermarking method that combines strong and weak embedding in a single digital watermarking technique, allowing both watermarks to be extracted in the primary print, while in the secondary print the strong watermark can be extracted but the weak watermark disappears, making it possible to verify authenticity. Therefore, the present invention provides a digital watermarking method in which the whole or part of an image is divided into a plurality of regions and watermark information is embedded in each region using a green noise pattern, the watermark being embedded using a green noise pattern of different roughness for each region, and the watermark being extracted using a different classifier for each region.

[0017] The classifiers are classifiers obtained by machine learning using neural networks corresponding to each level of roughness, and the watermark extraction is characterized in that watermark information and reliability are extracted for each region using a classifier corresponding to each level of roughness, and the authenticity of the image is determined from the watermark information and reliability extracted by the classifier for each region.

[0018] The same watermark information is embedded in each area, and authenticity is determined based on the degree of coincidence of the watermark information extracted from each area.

[0019] Further, among the plurality of regions, image link information is embedded in the region where watermark information is embedded in the coarsest pattern, and an individual identification number of the image is embedded in the region where watermark information is embedded in the finest pattern, and authenticity is determined based on the degree of match between the individual identification number obtained from the link destination and the individual identification number extracted from the region where watermark information is embedded in the finest pattern. Effect of the Invention

[0020] By doing the above, it becomes possible to determine the authenticity of printed images. In other words, it becomes possible to determine whether various printed images such as securities, tickets, and paintings that one owns are genuine or copied or counterfeited. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a system configuration for embedding a watermark according to the present invention. [Diagram 2] FIG. 2 is a diagram showing a flow of embedding and extracting watermark information according to the present invention. [Diagram 3] Diagram of green noise dot pattern generation. [Figure 4] A diagram showing the generated green noise dot pattern and its spectral distribution. [Diagram 5] A diagram showing the generated fine green noise dot pattern and its spectral distribution. [Figure 6] A diagram showing the generated coarse green noise dot pattern and its spectral distribution. [Figure 7] FIG. 4 is a diagram showing a flow of embedding watermark information. [Figure 8] FIG. 4 is a diagram showing a flow of extracting watermark information. [Figure 9] A diagram showing multiple embedding regions. [Figure 10] FIG. 2 is a configuration diagram of a neural network used in the present invention. [Figure 11] A 384x384 image is divided into two areas and a watermark is embedded. [Figure 12] FIG. 1 is a diagram showing a spectral pattern when a watermark is extracted from image data (electronic data). [Figure 13] FIG. 13 is a diagram showing the spectral pattern when extracting a watermark from a primary print. [Figure 14] FIG. 13 is a diagram showing a spectral pattern when extracting a watermark from a secondary print. [Figure 15] A 640x640 image divided into two regions. [Figure 16] An image showing a watermark embedded in part of an image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Fig. 1 shows the system configuration for embedding a watermark, in which an image to be embedded is loaded into memory 5 or data memory 7 such as a hard disk in a personal computer 3. Watermark information is entered from a keyboard 9, and an image processing program for embedding the watermark is stored in a program memory 6, and the embedding process is executed using a CPU 11, ROM 4, RAM 5, etc. The image with the watermark embedded is displayed on a monitor 8. The embedded image is printed by a printer 2, and is also distributed over the net via a communication function 10.

[0023] Figure 2 shows the process of embedding and extracting watermark information, where an image (13) with watermark information 12 embedded in it is displayed on a display (15) or printed by a printer (14). This image is photographed by an input device such as a smartphone camera (16), and the watermark extraction process is carried out inside the smartphone (17). The extracted information can be used in various ways, such as linking with a specific application (18) or leading to a specific homepage (HP) from the extracted URL (19). EXAMPLES

[0024] The following is an explanation based on the examples. First, the generation of the dispersed dot pattern used for embedding is explained. Figure 3 shows the generation flow of a dot pattern that exhibits green noise characteristics as a dispersed dot pattern. The green noise dot pattern is generated in the following steps. Step 1: First, as an initial value, (R^2) / 2 random dots are placed in an area with a block size of R x R, and the result is pi(x, y) (20). Here, the subscript i represents a multi-value number. For example, for binary values, i = 0, 1, and for quaternary values, i = 0, 1, 2, 3. Also, ^ represents exponentiation. For R = 32, 512 black and white dots are randomly placed. The initial value exhibits white noise characteristics. Step 2: Next, perform a two-dimensional Fourier transform of the dot pattern. We obtain Pi(fx, fy) = F{pi(x, y)} (21), where fx and fy are the spatial frequencies and F{} is the Fourier transform. Step 3: Multiply Pi(fx, fy) by an elliptical ring filter Di(fx, fy) that limits the frequency band of fx, fy to fx, min≦fx≦fx, max, fy, min≦fy≦fy, max to obtain a new spectral characteristic P'i(fx, fy) (22). Step 4: Perform an inverse Fourier transform on P'i(fx, fy) to obtain a multi-value dot pattern. p'i(x,y)=F -1 We obtain {P'i(fx, fy)}(23), where F -1 {} represents the inverse Fourier transform. Step5: Error: e(x, y)= p'i(x, y)-pi(x, y) Then, N pixels with the largest error are inverted (24). Steps 2 to 5 are repeated until the error e becomes equal to or smaller than a certain value Th, at which point the process is terminated to obtain the final pattern (25).

[0025] The dot pattern obtained by this method is a cluster-type dispersed dot pattern whose spectrum is distributed in the intermediate frequency band between the maximum frequency fmax and the minimum frequency fmin, and which exhibits so-called green noise characteristics. By skillfully adjusting this intermediate frequency band, a dot pattern can be obtained that has low response (is hard to see) to the human eye when observed at a clear vision distance, but is sufficiently responsive (resolved or visible) to a printer or camera.

[0026] In a dot pattern with a block size of R × R, the frequency due to the average dot spacing at a blackening rate of 1 / 2 is f0 = sqrt(0.5) R / 2 where sqrt() represents the square root. Δfmax and Δfmin are the differences based on f0 and are given by the following formula. Δfmax = fmax - f0 Δfmin = fmin - f0 In addition, the y direction is set to be an ellipse that is β times larger than the x direction. The green noise pattern defined in this way is GN(β, Δfmax, Δfmin) It is expressed as:

[0027] Figure 4 shows the green noise pattern and its spectrum for GN (1.6, -2, -6) when R=32 and there are four values. In the case of four values, four types of masks corresponding to the embedded bits "00", "01", "11", and "10" are used to obtain the respective green noise patterns. The spectra of the four types of patterns are elliptical ring shapes, each rotated by 45°. The code allocation is a Gray code with a Hamming distance of 1. The patterns are p0, p1, p2, and p3. We will call these patterns the standard patterns.

[0028] Figure 5 shows the green noise pattern and its spectrum for GN(1.5, 1, -3). Compared to Figure 4, Δfmax and Δfmin are larger (shifted more toward the + side), so the spectrum shifts to the higher frequency side, and the green noise pattern is finer. This GN(1.5, 1, -3) pattern is called the fine pattern. As with the standard pattern, the patterns are p0, p1, p2, and p3, and are represented by the 2-bit Gray code 00, 01, 11, and 10.

[0029] Figure 6 shows the green noise pattern and its spectrum for GN(1.5, -4, -7). Compared to Figure 4, Δfmax and Δfmin are smaller (shifted more towards the - side), so the spectrum shifts to the lower frequency side, and the green noise pattern is coarser. This GN(1.5, -4, -7) pattern is called the coarse pattern. As with the standard pattern, the patterns are p0, p1, p2, and p3, and are represented by the 2-bit Gray code 00, 01, 11, and 10.

[0030] The above is for the case of tetravalent noise, but in the case of binary noise, p0 and p2 can be used to assign the binary codes 0 and 1. Also, in step 1 of Fig. 3, different dot patterns can be obtained by changing the initial random number. In other words, it is possible to change the dot pattern by changing the seed value for random number generation in a pseudorandom number generator. In an RxR block, only (RxR)C(RxR / 2) different dot patterns are possible, and if R is large, the number can be said to be almost infinite (here, ()C() represents combination). However, the spectral shape does not change. For this reason, the extraction software can be used in common without modification. In this way, green noise patterns with different roughness are generated.

[0031] Next, the camouflage pattern will be described. The camouflage pattern is composed of an inversion pattern and a phase shift pattern. The inverse pattern is created as follows. σ=0: pi(x,y) pattern (basic pattern) σ=1: pi(x,y) upside down pattern σ=2: pi(x,y) left-right inverted pattern σ=3: pi(x,y) inverted pattern σ=4: Negative pattern of pi(x,y) σ=5: Negative pattern of pi(x,y) σ=6: Negative pattern of pi(x,y) σ=7: Negative pattern of pi(x,y) flipped upside down and left and right The up / down and left / right directions are performed along the long and short axis of the ellipse, respectively. A negative pattern is a pattern in which the white and black dots are inverted, and since the number of white dots and black dots is the same, the number does not change even when inverted. A total of eight different dot patterns σj (j = 0, 1, 2, ... 7) are generated for one pi(x, y), and all show the same spectral shape.

[0032] Here, the dot pattern pi(x, y) is expressed as a three-dimensional array, pi(σ, x, y), with σ as a parameter. As described later, a random inversion pattern can be obtained by selecting σ (an integer value between 0 and 7) as a random number.

[0033] Next, a phase shift pattern is a pattern in which the starting position of the pattern is shifted. Now, if the amount of shift is r, r has components in the x and y directions and is expressed as (X, Y). Here, 0≦X, Y≦R-1. Such (X, Y) is determined randomly by generating random numbers. The phase shift pattern in which the phase is shifted is as follows, based on the pattern without phase shift (basic pattern). A pattern with a phase shift of (X, Y) of pi(σ, x, y): pi(σ, (x+X) mod R, (y+Y) mod R ) Here, mod R represents the remainder modulo R.

[0034] Camouflage patterns are variations on the green noise pattern, preventing it from being detected and increasing security from attacks. They all have the same spectral characteristics, so extraction software can be used without modification when using camouflage patterns.

[0035] Next, we will explain how to embed watermark information. The watermark information is embedded in the luminance signal of the color image data. This is to prevent crosstalk between colors that occurs when a watermarked printed image is read by a camera or scanner because the color separation performance of the printing ink and color separation filter is not ideal, resulting in a decrease in the accuracy of watermark extraction.

[0036] The embedding is done by dividing the image data I(x, y) into R×R blocks and embedding the green noise pattern corresponding to the watermark bit information in each block. If the image size is W×H, the total number of blocks N is W / R×H / R (where ' indicates truncation to the nearest whole number).

[0037] Next, the specific embedding process is shown in the flow chart of FIG. 7. First, as pre-processing (30) before the embedding operation, a small amount of image smoothing is performed on the image. This image smoothing eliminates image dependency for various images. It maintains stability, similar to the low-pass filters used in single-lens reflex cameras. For example, if two-pixel smoothing is performed, the maximum frequency of the image after smoothing is halved. In other words, the Nyquist frequency of the image in the block is halved, the overlap with the green noise spectrum is reduced, and accurate watermark extraction is possible. This improves image stability for various images. Three-pixel smoothing may also be performed with a Gaussian filter. This smoothing is performed by convolving the smoothing filter with the image.

[0038] Next, the character string serving as the watermark information is converted into a bit string (32). The characters are coded in ASCII. An error correction code based on the Reed-Solomon code, which will be described later, is also added to generate the one-dimensional watermark bit string wm(i).

[0039] Next, the image is divided into R x R blocks (32) and watermark information is embedded in each block. The block size R is usually 32. A small block size (e.g. R = 16) can increase the amount of embedded information, but the accuracy of watermark extraction from the spectral distribution decreases. Conversely, a large block size (e.g. R = 64) improves extraction accuracy but reduces the amount of embedded information. R is usually a power of 2 so that spectral calculations using FFT can be performed quickly.

[0040] Embedding begins with the first block number n=1 (33). In this block, two bits are embedded in the case of four values, so two bits are extracted from the bit string of the watermark information (34). Next, a pattern corresponding to the two bits is embedded (36), and the values ​​(σ, X, Y) given to the inversion pattern and phase shift pattern are given as random numbers. The index σ of the inversion pattern is generated as a random integer value sequence from 0 to 7 using a random number generator with Seed1 as the seed, and the phase shift amounts X and Y are generated as random integer value sequences from 0 to R-1 using a random number generator with Seed2. In other words, these values ​​are different for each block. Also, if Seed1 and Seed2 are changed for each image, they will also be different between images. The pattern used for embedding is expressed by the following formula. p'i(σ, x, y)= pi(σ, (x + X) mod R, (y + Y) mod R) Then, embedding is performed for each block as shown in the following equation. W(x, y)=I(x, y)+ g·p'i(σ, x, y) where W(x, y) is the image with the watermark embedded, I(x, y) is the image before embedding, and g is the embedding strength. In this way, watermark information is embedded into image data. When embedding in one block (n-th block) is finished, the process moves to the next block (n+1-th block) (37), and when embedding in all blocks is finished (38), embedding is complete.

[0041] Next, adaptive embedding is explained. In adaptive embedding, when embedding of one block is completed, it is checked whether the embedded watermark has been extracted correctly and embedded with sufficient reliability. First, embedding is performed with embedding strength g, and the image of that block is immediately read out, the Fourier spectrum of that block is detected by FFT, and the watermark information and reliability are extracted by a discriminator. If the extracted watermark information is different from that at the time of embedding, or if the same watermark information is used but the reliability is below the reference value, the embedding strength is increased by Δg and the embedding operation is performed again in the same block. This operation is repeated, and when the detected watermark information is the same as the embedded information and the reliability is above the reference value, embedding of that block is completed and the process moves to the next block.

[0042] In areas where the image has locally high spatial frequency (for example, areas where fine patterns are densely packed), the image spectrum overlaps with the green noise spectrum to a large extent, resulting in extraction errors. In such areas, however, strong embedding can be used to reduce the extraction error rate. This reduces image dependency, enabling consistently stable embedding and extraction.

[0043] However, adaptive embedding strength changes for each block, so to make it a reversible digital watermark, it is necessary to store the embedding strength for each block. By dividing the embedding strength into several stages and storing the "stages," the storage capacity can be reduced. For example, if the embedding strength is divided into three stages and the three-value information is "0," "1," and "2," this three-value information can be stored for each block. If the above-mentioned image smoothing is performed and the embedding strength is appropriate, the areas of "1" and "2" are usually few and the majority are "0", so by compressing the data, it is possible to store it in the key with a small capacity. However, if there is no need for a reversible type depending on the application, there is no need to store the embedding strength for each block even in adaptive embedding.

[0044] Next, we will explain how to extract watermarks. Figure 8 is a block diagram showing how watermarks are extracted. After the image data containing the watermark is photographed using a smartphone camera or the like, it is corrected to a specified size. Then, just like when the watermark is embedded, it is divided into blocks (41), and the embedded information is extracted for each block. First, the first block (block number n=1) is started (42). The Fourier spectrum of this block's image data is obtained by FFT (43). Next, the obtained Fourier spectrum is subjected to pattern classification based on the spectral shape using a neural network classifier (described later) (44), and two bits of watermark information are extracted (45). Next, the process moves to the next block (46), and the watermark information is extracted using the same process. When all blocks have been processed (47), the extracted information is subjected to error correction decoding processing, and the obtained bit string is converted into a character string to obtain the embedded watermark information (48).

[0045] Next, adaptive extraction will be explained. If block synchronization is lost during extraction, extraction errors will occur. In order to reduce such extraction errors, adaptive processing is performed. Adaptive extraction processing gives a small positional shift to the block to be extracted. In other words, an R×R block is set as a reference block, and multiple R×R blocks with small displacements (Δx, Δy) in the x and y directions are set around the reference block, watermark information and reliability are extracted from these blocks, and the one with the highest reliability is selected, and the extracted information from this block is used as the watermark information. This operation makes it possible to extract more accurate information even if block synchronization is lost. The displacement (Δx, Δy) is usually sufficient to be about 2 pixels.

[0046] Next, embedding into multiple regions will be explained. Figure 9 shows an image divided into multiple regions and then divided into blocks (26). The size of each region may be different depending on the purpose. Also, the regions do not have to be rectangular, and they may be divided in the middle of a block. Each region is filled with a green noise pattern of a different coarseness. For example, region 1 (49) is filled with the coarsest green noise pattern GN(1.5, -4, -7), region 2 (50) is filled with the next coarsest pattern (standard pattern) GN(1.6, -2, -6), and region 3 (51) is filled with the finer pattern GN(1.5, 1, -3).

[0047] Since the size of the regions is determined in advance, the number of blocks per region is fixed. The embedding operation is performed sequentially, and when a predetermined number of blocks is reached, the pattern is changed to a green noise pattern of different roughness, allowing the embedding to be performed all at once.

[0048] When authenticity is determined based on the degree of agreement of the watermark information in each area (described later), the size of each area is made the same and the same watermark information is embedded. When authenticity is determined based on link information provided by the watermark, the size of each area does not need to be the same, and a size appropriate for the purpose is selected. In this case, a URL is embedded as link information in area 1, the name of the image owner and additional information about the image in area 2, and an individual identification number, etc., is embedded in area 3.

[0049] The image with the embedded watermark is output from the printer at a specific image size. In the primary printing, which is output from electronic data to a printer, the embedding strength (g) and print size are adjusted in advance so that the watermark can be extracted even from an image filled with the finest patterns in the printed image from the printer. If the print output image size is made small, the green noise pattern is reduced and cannot be read by a camera or scanner. Conversely, if the print image size is made large, the authenticity determination effect cannot be achieved. To make authenticity determination possible, it is necessary to set the output image size so that it can be read from fine patterns in the primary printing but cannot be read in the secondary printing.

[0050] The output size of a printer is usually specified in ppi (pixels per inch). For example, if an image of (X pixels) x (X pixels) is output in Yppi, the image size will be Image size [mm] = (X / Y) 25.4 For example, if you output a 512 x 512 image at 200 ppi, it will be approximately 65 mm x 65 mm.

[0051] As a result of various experiments, it was found that when outputted on a home inkjet printer at 175 ppi with an embedding strength of g = 0.25, the watermark can be extracted with high accuracy in the primary print even with fine patterns on plain paper. Naturally, the watermark information can also be extracted correctly with coarse and standard patterns. On photo paper and glossy paper, this is possible at 200 ppi. It was also confirmed that in the secondary print, extraction is possible with coarse and standard patterns, but not with fine patterns. These characteristics vary between individual cameras, scanners, printers, etc., so it is necessary to understand them in advance.

[0052] The watermark is extracted separately for each region. The coarsest pattern classifier is used for region 1, the standard pattern classifier for region 2, and the finest pattern classifier for region 3. These classifiers are derived by machine learning using a neural network, which will be described later.

[0053] Extraction of watermarks from each region is performed for each region according to the extraction flow shown in Figure 8. The size of each region is determined in advance, so the classifier is set to change when the region changes. Extracted watermark information and reliability are obtained from each classifier. A high accuracy rate is obtained for each region from the image printed out as a primary print at 175 ppi. Even if errors occur in several blocks due to scratches or dirt, the extracted character string shows an accuracy rate of almost 100% due to the error correction code.

[0054] Next, we attempt to extract the watermark in the same way from a secondary print (i.e., an illegally copied counterfeit product) that is a copy of the printed output image. The copy is made by a copier, but since it is read by a scanner and output by a printer, the high frequency components of the image are reduced. As a result, fine patterns cannot be read, and it becomes difficult to extract the watermark from area 3, which is embedded with fine patterns.

[0055] Next, we will explain how to determine authenticity. Suppose the same watermark information is embedded in areas 1, 2, and 3. In the primary print output at 175 ppi, the watermark can be read in all areas. Therefore, the watermark information extracted from area 3 and the watermark information extracted from area 1 are completely identical. However, in the case of a secondary print, the watermark information extracted in area 1 does not match the watermark information extracted in area 3. For this reason, if there is a perfect match, the image is determined to be genuine, that is, the original, and if there is a mismatch, it is determined to be non-genuine, that is, a secondary print copied from the original, and thus a counterfeit.

[0056] However, the degree of match varies depending on the characteristics of image input / output devices such as printers, scanners, and cameras, and on the environment. It also depends on the print image size and the image. This causes errors in the extracted information. Therefore, errors may occur in the extracted data from area 3 even in the primary printing. Error correction can correct up to three words of errors, but if there are more than that, it will not be a perfect match and will be considered inauthentic. However, if the average reliability of area 3 is 0.9 or more, it can be judged using the watermark information in area 2.

[0057] In other words, if the watermark information extracted from area 2 exactly matches the watermark information extracted from area 1, it is judged to be genuine and the print is determined to be the original. If they do not exactly match, it is judged to be non-genuine and the print is determined to be a counterfeit.

[0058] The more times watermark information is printed, the more difficult it becomes to extract it. In particular, the finer the pattern, the more rapidly it deteriorates and the more difficult it becomes to extract. Therefore, fine patterns in secondary printing are almost impossible to extract. However, there are cases where a primary print is judged to be non-authentic due to variations in equipment or the environment, and judgment using area 2 can be said to be a remedy in such cases.

[0059] Next, we will explain the pattern discriminator for extracting watermark information from the spectral distribution. Figure 10 shows the configuration of pattern discrimination by a neural network (NN). To perform machine learning, a large number of labeled spectral data extracted from an embedded image are prepared in advance as a dataset (52). For example, 2000 spectral patterns each of p0, p1, p2, and p3 are prepared, for a total of 8000 datasets. This dataset is divided into learning data and test data, and learning is performed using the NN method. The image size to be discriminated is 32 x 32, the number of neurons in the input layer (53) is 32 x 32 = 1024 corresponding to each pixel, the hidden layer (middle layer) (54) is 50, and the output layer (57) is 4 (in the case of a 4-value green noise pattern), a Sigmoid function (55) is used as the activation function, and the output is obtained by Softmax (57) in the final stage. For the dataset, a camouflage pattern watermark pattern was used for the image. As a result of machine learning using the backpropagation method, a 99.7% accuracy rate was achieved.

[0060] This machine learning is performed for green noise patterns with different roughness, and three types of network parameters are obtained by performing machine learning for three types of spectral patterns: a coarse pattern, a standard pattern, and a fine pattern.

[0061] By incorporating the network parameters obtained by such machine learning as CSV data into the watermark extraction program, an embedded neural network classifier can be implemented. The network parameters are a total of four dictionary-type data sets, consisting of two networks W1 and W2 with biases bi and b2, and are incorporated into the digital watermark extraction program as array data of W1[1024, 50], W2[50, 4], b1

[50] , b2[4]. The same is true for network parameters for patterns with different coarseness. The processing time of the implemented program is extremely short, and pattern classification with a significantly higher accuracy rate is possible compared to classification methods using conventional pattern matching methods.

[0062] The output value from the neural network is transformed by softmax in the final stage to represent the predicted value (probability of occurrence) of the output value. Therefore, the closer this value is to 1, the higher the probability that it is the target pattern, and this can be used as an index of reliability. In adaptive embedding, if the embedding strength is controlled so that this predicted value is equal to or greater than a reference value for reliability (e.g., 0.95), embedding can be performed with a high probability of reliability in all blocks regardless of the image, resulting in an embedding that can obtain a high accuracy rate.

[0063] Next, we will explain error correction. There are various factors that can cause extraction errors, such as the spatial frequency characteristics of the original image, stains and scratches on the printed matter, external light and shadows when shooting, internal distortion aberration of the camera, bending and wrinkles of the subject, position detection errors when generating a corrected image, projective transformation accuracy, and synchronization errors. For these reasons, error correction codes are necessary.

[0064] Therefore, we introduce Reed-Solomon (RS) code as an error correcting code and extract watermark information with error correcting capability. Using the irreducible polynomial x^8+x^4+x^3+x+1 in the extended Galois field GF(2^8) consisting of 2^8 elements, we incorporate RS(16,6) code that can correct up to 3 errors in 16 words. In this case, the generating polynomial is x^6+a^166 x^5+a^0 x^4+a^134 x^3+a^5 x^2+a^176 x+a^15. When embedding information using such RS code, the amount of embeddable information is reduced to 10 / 16 compared to the case without error correcting code, but it is possible to obtain a high accuracy rate regardless of the image.

[0065] Next, an example of division into two regions is explained with reference to Fig. 11. Fig. 11 shows a watermark embedded in a 384x384 Mandrill image. The image is divided into 32x32 blocks, 64x64 markers are placed at the four edges of the image, and the remaining region is divided into two regions of 64 blocks. Since 2 bits of information are embedded per block, 16 characters can be embedded. The watermark information is embedded in the standard pattern of GN(1.6, -2, -6) in region 1, and in the fine pattern of GN(1.5, 1, -3) in region 2. As an example, the watermark information is embedded with 10 characters of "abcdefghij" in region 1, and 10 characters of "0123456789" in region 2. Reed-Solomon code RS(16, 6) is used as the error correction code. This allows for error correction of up to three words out of 16 words, and requires six error detection words, resulting in 10 characters of watermark information.

[0066] The markers are written to the four edges of a rectangular embedded image, and use a radial pattern for positioning. The marker positions are detected from the embedded image, the position coordinates of the four edges of the image are calculated, and the trapezoidal distortion is corrected by projective transformation to generate a corrected image.

[0067] From this corrected image, the image is divided into blocks of size R x R, and the watermark is extracted. Figure 12 shows the spectrum of each block when watermark extraction is performed from the image (digital data) with the watermark information embedded in Figure 11. Region 1 shows the spectrum from the standard pattern, and region 2 shows the spectrum from the finer pattern. As the pattern becomes finer, the frequency becomes higher, and it can be seen that the elliptical ring-shaped spectrum becomes larger. The four edges of the image are excluded by the marker parts. Watermark information is extracted from the spectral pattern of each region using the respective classifiers.

[0068] In the primary printing, the image data with the embedded watermark is output from the printer at 175ppi. The print size is 55.7mm x 55.7mm. This printed image is photographed with a smartphone or other camera, a corrected image is created, and the watermark is extracted. Figure 13 shows the extracted spectral image. Although it is somewhat degraded, 100% correct answers were obtained. A reliability is obtained from each block, and the average reliability obtained by averaging the blocks within the region was 0.992 in region 1 and 0.935 in region 2, which were high reliability values. Experiments with various images showed that the average reliability was 0.9 or higher when the answer was correct.

[0069] Next, the primary printout with the embedded watermark is reproduced using a copier or MFP printer to create a secondary printout. This printout is then photographed in the same way using a smartphone or other camera, a corrected image is created, and the watermark is extracted. Figure 14 shows the extracted spectral image. As degradation progresses, the correct answer was obtained in area 1, but in area 2, all 10 embedded characters were incorrect, even with error correction. The average reliability in area 1 was 0.910, but in area 2 it was 0.810.

[0070] In the above example, different watermark information was embedded in area 1 and area 2 to verify the accuracy of extraction, but authenticity can be determined by embedding the same information. For example, the same information "abcdefghij" is embedded in area 2, and if the extraction results match perfectly, it can be determined to be authentic, and if they are different, it can be determined to be a counterfeit. If printed out at 175 ppi, the primary print will match almost perfectly. However, errors may occur due to scratches or stains on the printed matter. Therefore, the authenticity is expressed probabilistically and is called the degree of match. In other words, if the average reliability of area 2 is 0.910, the degree of match is 0.91, and it can be expressed as being authentic with a 91% probability. As mentioned above, if the average reliability of area 2 is 0.90 or more, it can be said that it is not a counterfeit.

[0071] As another example of embedding, image link information is embedded in area 1 and an individual identification number is embedded in area 2, and it is determined whether the individual identification number obtained from the link information matches the individual identification number obtained from area 2. Since the link information is searched from the management ledger on the homepage, a long number of characters is required for the URL, so a large image is used.

[0072] Figure 15 shows a 640 x 640 image divided into two areas, with link information embedded in area 1 and an individual identification number embedded in area 2. Area 1 has 320 blocks and can embed 50 characters (80 words with error correction code added), and can embed the URL of the website where authenticity is judged and the individual number, etc. Area 2 has 80 blocks and can embed 10 characters (16 words with error correction code added), and can embed an individual identification number of up to 10 characters.

[0073] The management ledger for the corresponding website is accessed from the URL extracted from area 1, and an individual identification number is obtained by providing the individual number. In other words, the website is accessed, the individual identification number is searched for from the corresponding individual number in the management ledger, and compared with the individual identification number extracted from area 2. If there is a perfect match, it is determined to be genuine. The average reliability of area 2 is then taken as the probability of authenticity.

[0074] Figure 16 shows the watermark embedded in part of an image, together with the company name, owner name, logo, etc. In the figure, it is embedded in the lower right part of the image. A corrected image is generated based on the markers at the four edges, and the watermark is extracted. The image quality is high because the watermark is embedded in only part of the image. This makes it possible to determine the authenticity of high-quality prints such as paintings and photographs. [Industrial Applicability]

[0075] As described above, the present invention makes it possible to determine the authenticity of printed matter by embedding a digital watermark. In particular, it makes it possible to determine the authenticity of primary printed matter (original) and secondary printed matter (copy), and it is possible to detect illegal copies. [Explanation of symbols]

[0076] Reference number 1 represents a camera or scanner, 2 a printer, 3 a personal computer (PC), 4 a ROM within the PC, 5 a RAM, 6 a program memory that contains the digital watermark program etc., 7 a data memory, 8 a monitor, 9 a keyboard, 10 a communication function and 11 a CPU. 12 represents watermark information, 13 an embedded image, 14 a printed image, 14 a displayed image, 16 an image captured with a camera, 17 a watermark extraction, 18 a link with application software and 19 a link to a homepage (HP). 26 represents a divided block and 49, 50 and 51 represent the various areas when the image is divided.

Claims

1. A digital watermarking method in which an entire image or a part of an image is divided into a plurality of regions, and watermark information is embedded in each region using a green noise pattern, The watermark is embedded in each region using a green noise pattern with different roughness. The watermark is extracted using a different classifier for each region.

2. A digital watermark embedding and extraction method comprising the steps of:

2. the classifier is a classifier obtained by machine learning using a neural network corresponding to each roughness, The watermark is extracted by using a classifier corresponding to the roughness of each region to extract the watermark information and its reliability.

2. The method for embedding and extracting a digital watermark according to claim 1, further comprising the step of determining the authenticity of the image based on the watermark information and reliability extracted from the classifiers for each region.

3. 4. The method for embedding and extracting a digital watermark according to claim 3, wherein the same watermark information is embedded in each area, and the authenticity is judged based on the degree of coincidence of the watermark information extracted from each area.

4. The method for embedding and extracting electronic watermarks as described in claim 3, characterized in that, among the multiple regions, image link information is embedded in the region where watermark information is embedded in the coarsest pattern, and an individual identification number of the image is embedded in the region where watermark information is embedded in the finest pattern, and authenticity is determined from the degree of match between the individual identification number obtained from the link destination and the individual identification number extracted from the region where watermark information is embedded in the finest pattern.

Images