Digital watermark analysis device and digital watermark analysis method

Abstract

To provide an electronic watermark analyzing device that picks up an image of printed matter and determines where to embed additional information in the picked-up image.SOLUTION: An image processing device which analyzes an electronic watermark comprises: an internal image pickup device which picks up an image of printed matter having additional information embedded therein by superposing a pattern having a predetermined period or frequency on an image so as to acquire an obtained image as a picked-up image; an image characteristic calculation part which analyzes a plurality of small areas in the picked-up image to calculate the spatial frequency characteristics of the individual small areas; a signal intensity estimation part which estimates an embedding position and embedding signal intensity of the additional information in the picked-up image based upon the spatial frequency characteristics; a marker detection area determination part which determines the position of a marker detection area for detecting a marker as a reference position for acquiring the additional information in the picked-up image based upon embedding position information and embedding signal intensity information; and an additional information acquisition part which acquires the additional information in the picked-up image based upon the position of the detected marker.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates primarily to a digital watermark analysis device. [Background technology]

[0002] There is a technology called digital watermarking (see Patent Documents 1 and 2). Digital watermarking is a technology that embeds additional information related to an image in an image so that it is difficult to visually distinguish, and may also be called a watermark. A similar technology is called steganography, which embeds additional information unrelated to an image. Images with the embedded additional information are distributed through networks such as the Internet, and the additional information is extracted from the images with the embedded additional information.

[0003] In addition, an image with embedded additional information may be printed on paper by an image output device such as a copier or printer, and the additional information may be extracted from the resulting printout. The additional information may be extracted from the printout by analyzing the image of the printout captured using an image capture function of a scanner or a camera function of a mobile terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-229508 A [Patent Document 2] Patent No. 4645457 Summary of the Invention [Problem to be solved by the invention]

[0005] According to Patent Document 1, it is necessary to evaluate the reading strength in advance, and according to Patent Document 2, it is necessary to specify at which position in the captured image the embedding is performed and at what level of embedding signal strength. Therefore, the configurations of Patent Documents 1 and 2 have room for improvement in making it possible to read unknown digital watermarks from printed matter.

[0006] The present invention has been made in response to the above-mentioned problems, and has an exemplary object to provide a technique that is advantageous for diversifying the reading of digital watermarks. [Means for solving the problem]

[0007] One aspect of the present invention relates to a digital watermark analysis device, the digital watermark analysis device comprising: an imaging unit that captures an image of a print in which additional information is embedded by superimposing a pattern having a predetermined period or frequency on the image, and obtains the image thus obtained as a captured image; a calculation unit that analyzes a plurality of small areas in the captured image and calculates spatial frequency characteristics of each small area; a specifying unit that specifies an embedding position and an embedding signal intensity of the additional information in the captured image based on the spatial frequency characteristic; a determination unit that determines a position of a marker detection area in the captured image for detecting a marker that serves as a reference position for acquiring the additional information, based on the embedding position information and the embedded signal strength information; an acquisition unit that detects a marker within the marker detection area and acquires the additional information within the captured image based on the position of the detected marker. It is characterized by: Effect of the Invention

[0008] The present invention is advantageous in diversifying the reading of digital watermarks. [Brief description of the drawings]

[0009] [Figure 1]FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing system. [Diagram 2] FIG. 2 is a schematic diagram of a recording head of a printer. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of a control circuit of a printer. [Figure 4] FIG. 2 is a block diagram showing an example of a system configuration. [Diagram 5] FIG. 2 is a diagram illustrating an example of a screen configuration of an application. [Figure 6] 13 is an example of an embedding information input dialog for an application. [Figure 7] 13 is an example of a print setting dialog of an application. [Figure 8] FIG. 13 is a diagram showing an example of SVG description. [Figure 9] 13 is a flowchart of a print data generating process. [Figure 10] FIG. 11 is a diagram showing an example of a print command. [Figure 11] 13 is a flowchart of a process of embedding additional information. [Figure 12] FIG. 13 is a diagram showing an embedding pattern. [Figure 13] FIG. 2 is a diagram illustrating a configuration example of a unit block. [Figure 14] 11A and 11B are diagrams illustrating an example of embedding unit blocks in a printed matter. [Figure 15] FIG. 2 is a block diagram including an additional information reading processing unit. [Figure 16] 11A and 11B are diagrams illustrating an example of a captured image and a marker detection area when a printed material is read. [Figure 17] FIG. 13 is a diagram showing an example of a plurality of small areas in a captured image. [Figure 18] FIG. 11 is a diagram illustrating an example of calculation of image characteristics. [Figure 19] FIG. 13 is a diagram illustrating an example of analysis in the spatial frequency domain. [Figure 20] 13 is a flowchart of a process for determining a position of a marker detection area. [Figure 21] FIG. 13 is a diagram showing an example of a notification on a display. [Figure 22]FIG. 2 is a block diagram including an additional information reading processing unit. [Figure 23] FIG. 13 is a diagram illustrating an example of an embedded signal strength table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <<First embodiment>> <Image processing system overview> The image processing device according to the first embodiment (assumed to be image processing device 100; image processing device 100 may also be expressed as information processing device 100), details of which will be described later, includes an application 400 (see FIG. 4) capable of embedding and reading additional information. The application 400 is a program that creates image data as content with the additional information embedded therein, and may also be expressed as an application program. As will be described later with reference to FIG. 1 etc., the image processing device 100 generates print data from image data as content, and transmits the print data to a printer 112. The printer 112 performs printing using the print data, and outputs a printed matter with the additional information embedded therein.

[0012] The term "additional information" as used herein refers to information embedded in a printed matter. The additional information is embedded, added, or multiplexed in a printed matter so as to be visually inconspicuous or difficult to visually recognize. Such additional information may also be expressed as multiplexed information, watermark information, or embedded information.

[0013] <Hardware configuration of image processing device> Fig. 1 shows an example of the configuration of an image processing system SY according to this embodiment. The image processing system SY includes an image processing device 100, an external imaging device 111, a printer 112, and a server 114. The image processing device 100 and the printer 112 can be connected to each other via a network 113 or directly. In Fig. 1, the server 114 is connected via the network 113, but may be omitted in other embodiments.

[0014] A known portable or non-portable terminal may be used as the image processing device 100, and typically includes, for example, a smartphone, a tablet PC (personal computer), a notebook PC, a desktop PC, etc. In this embodiment, the image processing device 100 is a device equipped with a touch panel display, that is, a so-called smartphone.

[0015] The image processing device 100 includes a CPU (Central Processing Unit) 102, a ROM (Read Only Memory) 103, and a RAM (Random Access Memory) 104. The image processing device 100 further includes a secondary storage device 105, a display 106, IFs (Interfaces) 107, 108, and 109, and an internal imaging device 110.

[0016] The CPU 102 loads a predetermined program in memory and executes it to realize various processes described below. Although a single CPU 102 is shown in Fig. 1, multiple CPUs or CPU cores may be provided. The ROM 103 stores the program executed by the CPU 102. The RAM 104 temporarily stores various information when the CPU 102 executes the program.

[0017] The secondary storage device 105 is a known non-volatile memory or storage medium such as a hard disk drive (HDD), a solid-state drive (SSD), a flash memory, etc. The secondary storage device 105 stores files, data such as a database that holds processing results such as image analysis, and various programs.

[0018] The display 106 displays a UI (user interface) for receiving operations for implementing various processes, and displays various information such as the results of the executed processes. In this embodiment, a touch panel display is used for the display 106, that is, the display 106 is equipped with a touch sensor. This allows the user to input a desired operation to the display 106 by touch operations such as tapping, swiping, pinching in, and pinching out.

[0019] An internal imaging device (imaging unit) 110 captures an image. Image data obtained by capturing an image is subjected to predetermined image processing and then stored in the secondary storage device 105. Image data may also be acquired from an external imaging device 111 via the IF 108 and stored in the secondary storage device 105.

[0020] The IF 109 can communicate with external devices (for example, a printer 112 and a server 114) via a network 113.

[0021] The IFs 107 to 109 may employ either wired or wireless communication. The image processing device 100 can acquire image data from a server 114 connected to a network 113 or from an external imaging device 111, and output image data (print data) to a printer 112, via the IFs 107 to 109. Examples of wired communication include USB and Ethernet (registered trademark). Examples of wireless communication include wireless LAN, NFC (Near Field Communication), Bluetooth (registered trademark), and infrared communication.

[0022] The above-mentioned elements are connected to each other via an internal bus 101, and a CPU 102 controls these elements via the internal bus 101. In this embodiment, a program is executed by the CPU 102, that is, the image processing device 100 can be said to be a place where software is executed (a software execution environment).

[0023] <Printer hardware configuration> 2 is a schematic diagram showing a part of the hardware configuration of the printer 112. In this embodiment, the printer 112 employs an inkjet system, but other recording systems such as an electrophotographic system may also be employed. The printer 112 includes a recording head 201, an ink unit 202, and a control circuit unit 203.

[0024] The recording head 201 is configured to be capable of scanning (moving back and forth) in the width direction of the print medium 204, and performs recording by ejecting ink received from ink units 202 mounted on the recording head 201 onto the print medium 204. Here, it is assumed that four ink units 202 are provided, but they may be of any color, or may be of the same color or type.

[0025] The control circuit unit 203 includes a storage unit, a calculation unit, and a communication unit required for driving the recording head 201. The recording head 201 receives a recording signal and a control signal from the control circuit unit 203, and performs recording according to the recording signal based on the control signal. A print medium 204, which is a recording medium, is transported by a transport roller (not shown) while recording is being performed by the recording head 201, and an image is formed on the print medium 204 in this manner.

[0026] 3 is a block diagram showing the configuration of the control circuit unit 203. The control circuit unit 203 includes an input IF 301, a CPU 302, an output IF 303, a ROM 304, and a RAM 305. The input IF 301 receives image data and a control signal for driving the print head 201 as input signals from an image processing device or the like external to the printer 112. The input IF 301 sends the image data and the control signal to the RAM 305 and the CPU 302.

[0027] In response to this, the CPU 302 executes a control program stored in a non-volatile memory, ROM 304, to perform signal processing on the image data. The image data that has undergone this signal processing is output as recording data together with a control signal from the output IF 303. The recording head 201 is driven based on the output recording data and control signal, thereby printing an image on the print medium 204.

[0028] <System configuration> 4 shows an example of the configuration of an image processing system including an application 400 and a printer 112. The application 400 runs on the image processing device 100, and in this embodiment includes a display processing unit 401, an image acquisition unit 402, an additional information setting unit 403, a print setting unit 404, a print data generation unit 405, a data transmission / reception unit 406, and an additional information reading unit 407. The application 400 may further include other units (which may be expressed as blocks, etc.) as necessary.

[0029] The application 400 is stored as a program in the ROM 103 or the secondary storage device 105, and is read and executed by the CPU 102, which causes the CPU 102 to function as the above-mentioned individual units of the application 400. The application 400 has a function of generating image data in which additional information is embedded based on the image data to be printed and the input additional information, and also has a function of transmitting print data for printing the generated image data to the printer 112.

[0030] The printer 112 includes a print processing unit 408 and a data transmission / reception unit 409. The print processing unit 408 and the data transmission / reception unit 409 are realized by the CPU 302 reading out and executing a program stored in the ROM 304.

[0031] Fig. 5 shows an example of a screen of the application 400 operating on the image processing device 100. Hereinafter, the processing performed by each element shown in Fig. 4 will be described with reference to Fig. 5.

[0032] The display processing unit 401 performs processing for displaying character information to be embedded as additional information, images to be used in print processing, and the like on the display 106 of the image processing device 100. The display processing unit 401 also has a function for displaying information related to a user interface (UI), such as buttons for selecting images and characters, and a list of print settings.

[0033] 5, an image selection button 501, an embedded information setting button 502, and a print setting button 503 may be displayed on the display 106 of the image processing device 100. An image that will become the content being edited may be displayed in the content editing area 500. Note that the buttons 501 to 503 are not limited to the above example, and may be given other names or may be symbolized icons.

[0034] The image acquisition unit 402 acquires an image into which additional information is to be embedded. For example, when the image selection button 501 is selected, an image selection dialogue (a dialogue box, or a window as another example) is launched, and the user can use this to select an image available in the image processing device 100. In this embodiment, the user selects an image from the image selection dialogue, but in another embodiment, the user can also use a captured image by separately activating the internal imaging device 110. The image acquired by the image acquisition unit 402 is displayed in the content editing area 500, and can be enlarged, reduced, rotated, or moved by a touch operation. Any other editing may be additionally performed.

[0035] 6 shows an example of an embedded information setting dialog 600. When an embedded information setting button 502 is pressed, the embedded information setting dialog 600 is displayed on the display 106. The additional information setting unit 403 sets the information edited through the embedded information setting dialog 600 as additional information. When the user taps an area 601, the user can edit the information to be embedded (additional information). As an example, it is assumed that the character string "hello" is embedded as additional information.

[0036] Although text input is exemplified here, in other embodiments, attribute information related to the image, such as copyright, image capture date and time, image capture location, photographer, etc., may be embedded as additional information. Alternatively, link destination information, such as a uniform resource locator (URL), may be embedded as additional information. In still other embodiments, information other than text, such as audio or video, may be embedded as additional information. Furthermore, in the case where there is a common data table, identification information for identifying data in the data table may be embedded as additional information.

[0037] When the user has completed the input, he or she can press a decision button 602, or when canceling, he or she can press a cancel button 603.

[0038] The embedding of additional information may be performed in a manner in which the user consciously inputs the additional information he or she wishes to embed (see FIG. 6), or in a manner in which the application 400 inputs the additional information without the user being aware of it. For example, if the selected image is a JPEG image with information embedded in an Exif (Exchangeable Image File Format) tag, information such as the date and time the image was taken and the location where it was taken may be automatically embedded.

[0039] 7 shows an example of a print setting dialog 700. When the print setting button 503 is pressed, the print setting dialog 700 is displayed on the display 106. Through the print setting dialog 700, the user can select a printer, a paper type, a paper size, and set bordered / borderless printing.

[0040] When the printer selection list 701 is tapped, a list of available printers is displayed, and the user can select any printer. When the paper type list 702 is tapped, a list of available paper types is displayed, and the user can select any paper type. Examples of paper types include glossy paper, matte paper, and plain paper. When the paper size selection list 703 is tapped, a list of paper sizes available for the paper type selected in the paper type list 702 is displayed, and the user can select any paper size. In addition, the user can set the borders (bordered / borderless) when printing by tapping the print border setting 704. Bordered printing is a printing method that creates margins on all four sides of the printed matter, and borderless printing is a printing method that prints without creating margins on the paper.

[0041] When the print button 705 is pressed, the print setting unit 404 transmits the print settings to the print data generating unit 405 and proceeds to the printing process. When the cancel button 706 is pressed, printing is not executed and the screen returns to the previous screen.

[0042] The information relating to the print settings of the printer displayed in the print setting dialog 700 may be stored in advance in an information processing device, or may be downloaded as needed from the server 114 or the printer 112 via the network 113 .

[0043] The print data generation unit 405 performs content rendering processing, additional information embedding processing, and processing for generating data required for printing by the printer 112. In the rendering processing, the print data generation unit 405 generates image data to be used for printing from the content acquired by the image acquisition unit 402 and displayed in the content editing area 500. In the additional information embedding processing, the print data generation unit 405 embeds the additional information set by the additional information setting unit 403 in the image data generated in the rendering processing. The additional information embedding processing may be expressed as a multiplexing processing.

[0044] The content used to generate the print data can be written using a web standard language (HTML (Hyper Text Markup Language), CSS, JavaScript (registered trademark), etc.). In this embodiment, SVG (Scalable Vector Graphics), which is one of the graphic representation methods in HTML, is used. However, this is not limited to this, and for example, Canvas, which is used to describe graphics in HTML, may also be used.

[0045] Figures 8(a) and 8(b) show examples of content description used for printing in this embodiment. Figure 8(a) shows an outline of an example of SVG description of content when only one image is placed. Note that the numbers on the left side of the figure indicate line numbers.

[0046] According to Fig. 8(a), the first line indicates that the content is described in SVG and indicates the size of the SVG. The second line indicates a description of an image added to the content. In this embodiment, the application 400 creates content consisting of only one image, but it may also be a form in which text, stamps, etc. are added to the image. In that case, information on the added text, stamps, etc. is described in SVG.

[0047] Next, with reference to FIG. 9, a process in which the print data generating unit 405 executes rendering processing of the above content described in SVG, embeds additional information, and generates print data will be described.

[0048] FIG. 9 is a flowchart showing an example of the print data generation process performed by the print data generating unit 405.

[0049] First, in S902 (hereinafter simply referred to as "S902"; the same applies to other steps described below), the print data generation unit 405 determines a rendering size according to the paper settings. For example, if the paper settings are L size, bordered, and the print resolution is 300 dpi, the rendering size will be 950px (pixels) wide x 1400px high.

[0050] If a print resolution other than 300 dpi can be set as the print setting, the rendering size corresponding to the other print resolution is determined. For example, as described above, if the paper setting is L size, bordered, and the print resolution is 600 dpi, the rendering size is calculated as 1900px x 2800px.

[0051] In S903, the print data generation unit 405 rewrites a part of the SVG content for printing. Here, the width and height of the SVG are changed based on the image size to be sent to the printer 112 so as to match the rendering size determined in S902. Specifically, the width on the second and third lines of the SVG in Fig. 8(a) is changed to "950", the height to "1400", and the viewBox on the second line to "0 0 950 1400". An example of the rewritten SVG is shown in Fig. 8(b).

[0052] In S904, the print data generating unit 405 requests a screen generation from an OS (Operating System) (not shown) of the image processing device 100. Here, information on an area of ​​950px x 1400px is acquired by referring to the SVG rewritten in S903. Then, the OS executes screen generation (950px x 1400px) for the SVG. Here, information on the size of the SVG may be held in the application 400, not in the SVG itself. After that, the OS generates a screen that is not displayed on the UI as an off-screen screen.

[0053] In S905, the print data generation unit 405 requests the OS to draw the SVG on the screen generated in step S904. The OS loads the SVG information. When the SVG information has been loaded, the application 400 receives a load completion notification from the OS. This notification may be made using a standard function provided in the OS. When the application 400 receives the load completion notification, it determines that the SVG drawing has been completed and proceeds to the next step.

[0054] In S906, the print data generation unit 405 requests image data of the content from the OS. The image data here is RGBA data of the content displayed on the off-screen screen. S906 can be simply said to be a step of executing a screen capture. In response to a request from the application 400, the OS executes a screen capture of the off-screen screen on which the content is drawn, and transmits the acquired RGBA data to the application 400.

[0055] In S907, the print data generation unit 405 performs processing for embedding additional information into the RGBA data acquired in S906. The additional information is embedded in the image so as to be difficult to see, and the information set by the additional information setting unit 403 is used. Details of the processing for embedding additional information will be described later. The RGBA data acquired from the OS in S906 includes information on A (Alpha, here transparency). In this embodiment, this transparency information is not used for printing, so it is deleted and the processing for embedding additional information is performed as RGB data.

[0056] In S908, the print data generating unit 405 converts the RGB data after the additional information embedding process into JPEG image data. Note that, although the embodiment illustrates an example in which image data is converted into JPEG, other known formats may be adopted, such as PDF (Portable Document Format).

[0057] In S909, the print data generation unit 405 adds a print command to be sent to the printer 112 to the JPEG image data generated in S908. Here, the data to be added to the JPEG image data is generated based on the above-mentioned print setting information. Additionally, in S909, the print data generation unit 405 may add a command for controlling the printer 112 as necessary.

[0058] Fig. 10 shows an example of a print command for print data. Here, a command written in XML format is used as the print command. Lines 3 to 5 describe the printer settings indicating "printing in standard mode on L-size glossy paper." Lines 7 to 9 describe that the print data is "JPEG data with a width of 950 and a height of 1400," and line 10 describes that the JPEG data converted in S909 will be inserted.

[0059] The printer 112 executes printing by receiving the print command as described above together with the image data (JPEG data). Through the above steps, the rendering process, the embedding process of additional information, and the generation process of print data performed by the print data generation unit 405 are completed.

[0060] Referring again to FIG. 4, the data transmission / reception unit 406 on the image processing device 100 side transmits print data to the printer 112. The application 400 requests the OS to transmit print data, and the OS transmits the data received from the application to the printer 112. The data transmission may be realized by a known method capable of communicating between the image processing device 100 and the printer 112, and is assumed to be performed by wireless WiFi here. As another example, the data transmission may be performed by a wired method, for example, by connecting the image processing device 100 and the printer via a USB port. In this case, the data may be converted by a printer driver of the image processing device 100 into raster data printable by the printer 112, and the raster data may be transmitted to the printer 112 via the USB port. Alternatively, the raster data may be converted into data printable by the printer 112, and printing may be performed.

[0061] In the printer 112, the data transmission / reception unit 409 receives print data from the image processing device 100. The data transmission / reception unit 409 analyzes the print command included in the received print data, and sends print setting information indicating the paper size, paper type, etc. obtained by the analysis to the print processing unit 408. The data transmission / reception unit 409 also decodes the JPEG file included in the print data to convert it into image data, and sends the converted image data to the print processing unit 408.

[0062] Here, during normal printing, if the print setting information includes correction processing information, image correction processing can be performed on the image data. Examples of image correction processing include brightness adjustment, contrast adjustment, color balance adjustment, as well as well-known image processing such as backlight correction and red-eye correction for photo printing. When printing an image with embedded additional information, the embedded information may be lost or erased due to image processing, so execution of image correction processing may be suppressed. When executing the above-mentioned image correction processing, it is preferable to execute the image correction processing before generating print data by the application 400 (before embedding the additional information).

[0063] The print processing unit 408 converts the image data received from the data transmission / reception unit 409 into ink color data by separating the colors so that the image data is output in suitable colors when it is output by the printer 112. The conversion to ink color data may be performed using any of the known color conversion processes that convert from an image format (RGB) used for screen display to ink colors (CMYK) for printing.

[0064] For example, assume that the printer 112 has four ink colors: cyan (C), magenta (M), yellow (Y), and black (K). The print processing unit 408 converts input data consisting of three colors, red (R), green (G), and blue (B), into four ink color data of C, M, Y, and K. A color separation table may be used during this conversion. The color separation table is a three-dimensional lookup table that receives input values ​​of R, G, and B and converts them into ink color values ​​of C, M, Y, and K. In this way, the print processing unit 408 converts the values ​​of R, G, and B into ink color values ​​of C, M, Y, and K for each pixel of the input image data using the color separation table.

[0065] Furthermore, the print processing unit 408 can perform image processing such as output tone correction and halftoning using image processing parameters set in a lookup table or the like, thereby converting the ink color data into data for print output.

[0066] The converted print output data is sent to the control circuit unit 203. The control circuit unit 203 forms an image by ejecting ink onto the print medium 204 based on the print output data obtained as described above. Through the above steps, the print processing of the print data is completed.

[0067] The additional information is embedded in the print obtained in the above manner (print medium 204 on which image formation has been completed). The additional information reading unit 407 of the application 400 has a function of reading and extracting the additional information from the print in which the additional information is embedded. The details of the additional information extraction process will be described later.

[0068] In the example of FIG. 4, the application 400 is capable of both embedding and reading the additional information, but the embedding and reading of the additional information may be performed separately by different applications.

[0069] <Additional information embedding process> 11 is a flowchart of an embedding process performed by the application 400 (mainly the print data generating unit 405) to embed additional information into rendered image data. In short, this flowchart shows the details of step S907 in FIG.

[0070] In S1101, the print data generating unit 405 converts the additional information into binary data. Here, as an example of the additional information, consider the case where the character string "hello" is input. Binary data is information of "0" or "1", and by arranging these consecutively, it is possible to give the binary data a specific meaning. The correspondence between binary data and characters is defined by a code also called a "character code". For example, according to "Shift JIS", which is an example of a character code, "h" can correspond to the binary data "01101000". Similarly, "e" can correspond to "01100101", "l" can correspond to "01101100", and "o" can correspond to "01101111".

[0071] That is, the character "hello" can be expressed as binary data "0110100001100101011011000110110001101111". On the other hand, when binary data "0110100001100101011011000110110001101111" is acquired, the character "hello" is acquired. In this way, the binary data is embedded so that "0" or "1" can be determined, thereby realizing embedding of additional information.

[0072] 12(a) and 12(b) show some examples of methods for embedding binary data "0" or "1" information into an image.

[0073] The pattern shown in Fig. 12(a) represents binary data "0," and the pattern shown in Fig. 12(b) represents binary data "1," both of which are configured as 8px x 8px (each rectangle in the figure represents 1px (1 pixel).) The print data generation unit 405 uses these patterns to embed additional information configured as binary data of "0" and "1" into the image, and superimposes or adds a periodic pattern (or a periodic pattern) to an 8 x 8 area in the image.

[0074] One example of a method for embedding a pattern in an image is to directly superimpose the periodic pattern on RGB values ​​(luminance information). As another example, the embedding may be realized by separating the RGB values ​​into other color space information such as luminance-color difference information (e.g., CIE L*a*b*, YCrCb signals). As yet another example, the embedding may be realized by separating the RGB values ​​into ink colors (e.g., CMYK signals).

[0075] In this embodiment, for the sake of simplicity, a method is adopted in which a pattern having the above periodicity is superimposed on the B component of the RGB components. Here, each component is represented by 8 bits (0 to 255). In embedding a pattern in an image, for example, 30 is subtracted from the pixel value (B value) of the B component for the hatched pixel parts in Figs. 12(a) and 12(b).

[0076] Additionally, it is more preferable if there are pixel portions to be added. When the number of pixel portions to be subtracted and the number of pixel portions to be added are substantially equal, the sense of incongruity in the image after the additional information is embedded can be reduced.

[0077] 12(c) and 12(d) show patterns in which the subtracted pixel portion and the added pixel portion are mixed (the added pixel portion is shown with lighter hatching than the subtracted pixel portion). In this way, the embedding of additional information is realized by performing processing using a predetermined pattern on any of the RGB data of the image to be embedded.

[0078] Here, when embedding a pattern having the above-mentioned periodicity in an image by superimposing it, it is preferable to embed it with a pattern period (or color amplitude change) that is unlikely to exist in nature. For example, in this embodiment, the B component increases and decreases, but a pattern in which the B component increases and decreases at a relatively high period is unlikely to exist in nature. By implementing embedding in this manner, it is possible to reduce interference between the embedded signal and the original image that may occur when reading additional information.

[0079] Alternatively / additionally, it is also conceivable to perform a filtering process on the image to be embedded in advance for a frequency corresponding to the period of the pattern to be superimposed. This can be realized using a known filter, and as an example, it can be realized by a filtering process using an FFT (Fast Fourier Transform) filter. That is, an image to be embedded is subjected to a filtering process on a specific frequency by performing an FFT on the image to be embedded and converting it into an image in the spatial frequency domain, and performing a filtering process on the frequency corresponding to the pattern to be superimposed and performing an inverse FFT. By performing the above embedding on the image obtained in this way, it is possible to reduce the above interference that may occur when reading the additional information.

[0080] 13 shows a unit block (multiplexed block) of additional information according to this embodiment. According to this embodiment, the embedded additional information is obtained by reading all information in a unit block 1300. In this embodiment, the unit block 1300 is configured to be 240px x 240px including a marker section 1301 and a data section 1302.

[0081] The marker section 1301 is an 80px x 80px block used to identify the position of the unit block 1300 when reading additional information, and has embedded therein information identifying the marker section 1301. In this example, a specific pattern consisting of a combination of Figures 12(a) and 12(b) is given as information identifying the marker section 1301, and the marker section 1301 can be identified by detecting this pattern. As an alternative method, it is also possible to configure the marker section 1301 with a pattern having a periodicity (or a direction of periodicity) different from that of Figures 12(a) and 12(b).

[0082] In addition, data loss or reading errors when reading the additional information may be detected and corrected by including error correction or error detection data in the data of the 100-bit marker section 1301. Examples of error correction methods include Reed-Solomon codes and Golay codes, and examples of error detection methods include CRC (Cyclic Redundancy Check).

[0083] In the data section 1302, a pattern 1303 (i.e., a binary pattern representing "0" or "1") of FIG. 12(a) or FIG. 12(b) is arranged. In this embodiment, 800 patterns 1303 are arranged in the data section 1302. Each pattern 1303 can represent 1 bit of information. Therefore, the entire block 1300 contains 800 bits of information. As with the binary data of the marker section 1301, error correction and error detection data may be provided for this 800-bit binary data.

[0084] In this embodiment, the marker section 1301 is disposed in the upper left part of the unit block 1300, but may be disposed in other positions such as the upper right part or the center. The marker section 1301 may be provided in a shape other than a rectangle, and may be disposed in a dispersed manner within the unit block.

[0085] 14 shows the print medium 204 in which unit blocks 1300 are repeatedly embedded. According to this embodiment, unit blocks 1300 expressing the same additional information are repeatedly embedded over the entire surface of the print medium 204. It is assumed that the print settings are L size, 300 dpi printing, and bordered printing.

[0086] In this embodiment, the print image size in this case is 950px wide x 1400px long as a result of the processing in S902. Therefore, unit blocks 1300 of 240px x 240px can be arranged in 3 rows x 5 columns (a total of 15 blocks). To read the additional information embedded in the print medium 204, it is sufficient to read the data portion 1302 of one of the multiple unit blocks 1300 repeatedly embedded.

[0087] Note that the robustness when reading the additional information can be improved by embedding a large amount of the same information (unit blocks 1300) all over the surface of the print medium 204. This is because even if the additional information cannot be read from a certain unit block 1300, it may be possible to read the additional information from other unit blocks 1300.

[0088] Note that, although an example is shown here in which the same information (unit block 1300) is repeatedly embedded, this is not limited to this example. Different information (blocks with different contents) may be embedded all over the image. Also, in FIG. 14, there are areas in the right and bottom parts of the print medium where the block size is insufficient, but part of the block may be embedded in these areas as well.

[0089] 11 again, in S1102, the print data generating unit 405 determines the arrangement of the unit blocks 1300. The print data generating unit 405 arranges the unit blocks 1300 starting from the upper left corner of the print image. The example in Fig. 14 described above shows an example in which the unit blocks 1300 are arranged starting from the upper left corner of the print image.

[0090] In S1103, the print data generating unit 405 embeds the additional information into the image in which the blocks 1300 are arranged in the above-determined arrangement. The additional information is embedded by superimposing the pattern of FIG. 12 on each pixel while associating the image to be embedded with the arrangement of the blocks 1300 in FIG. 14. Specifically, the data section 1302 in each block 1300 in FIG. 14 is configured with the pattern of FIG. 12(a) or FIG. 12(b). Therefore, the above-mentioned subtraction process (process of subtracting 30 from the B value) is performed on the hatched pixel portion in each pattern. If the B value is below 0, the pixel value after subtraction is set to 0. By applying this to the entire image, the embedding of the additional information into the image is completed.

[0091] In this manner, the additional information can be embedded in the image and printed by the printer 112. The patterns in Figures 12(a) to 12(d) can also be expressed as embedded patterns.

[0092] <Additional information embedded signal strength> Next, the embedding signal strength related to the ease of reading additional information will be described. In the above description, an example was shown in which a periodic pattern was formed by subtracting (and incidentally adding) the B value in the embedding process of additional information, but if the B value falls below 0 or exceeds 255 due to subtraction or addition, it is limited to 0 or 255, respectively. This reduces the color amplitude, so in this example, the embedding signal strength is reduced when the B value is in the range of 0 to 29 or 226 to 255. Also, in the white color where R value=G value=B value=0 and its neighboring color ranges, the addition of the B value results in a color on the white background (the R value indicates the pixel value of the R component, and the G value indicates the pixel value of the G component). Therefore, in the embedding of additional information, the embedding signal strength is often lowered for the white color and its neighboring color ranges, and the above-mentioned subtraction or addition is not performed (or the amount of subtraction or addition is reduced). If the embedded signal strength is low, the additional information is difficult to read, and if the embedded signal strength is high, the additional information is difficult to read. Therefore, a printed matter generally contains a mixture of areas where the additional information is easy to read and areas where the additional information is difficult to read.

[0093] 16 shows an example of image data of a printout 1601 indicating the magnitude of the embedded signal strength. In the figure, a lighter color indicates a lower embedded signal strength, and a darker color indicates a higher embedded signal strength.

[0094] <Additional information reading process> Next, a process for reading additional information from a printed matter will be described. The process for reading additional information is executed by the additional information reading unit 407 (see FIG. 4) of the application 400, but may be executed by an application of another device.

[0095] 15 is a block diagram showing an example of the configuration of an image processing device 100 for performing additional information reading processing. An additional information reading unit 411 included in an application 400 of the image processing device 100 includes an image acquisition unit 1501, an image characteristic calculation unit 1502, a signal intensity identification unit 1503, a marker detection area determination unit 1504, and an additional information acquisition unit 1505.

[0096] The image acquisition unit 1501 controls the internal imaging device 110 to capture an image of the printed matter 1601 in which the additional information is embedded, and acquires image data of the printed matter 1601. The image data may be still image data, or still image data extracted from video data acquired at a predetermined frame rate. The image data thus obtained is sent to the image characteristic calculation unit 1502. The image data is assumed to be obtained as 8-bit data in RGB format, and its size is 1080×1920 (2K size).

[0097] Fig. 16 shows an example of image data obtained as described above. As shown in Fig. 16, a printout 1601 in which additional information is embedded is imaged by the internal imaging device 110 of the image processing device 100, and image data of an imaging area 1604 is obtained. The additional information is obtained by analyzing a marker detection area 1605 in the imaging area 1604. To reduce the amount of calculation, the additional information is obtained not for the entire imaging area 1604 but for a partial area.

[0098] The image characteristic calculation unit 1502 acquires image characteristics from a plurality of small areas in the image data of the printout 1601. When calculating the image characteristics, spatial frequency information obtained by frequency analysis can be used.

[0099] 17(a) is an explanatory diagram of the above-mentioned frequency analysis. In the figure, a plurality of small areas 1701 arranged within an imaging area 1604 are used for frequency analysis.

[0100] 18 is a flowchart showing an example of a method for calculating image characteristics. In S1800, a parameter i=0 is set.

[0101] In S1801, a trimming process is performed on a plurality of small areas 1701. The small areas 1701 are set as an analysis size, and are trimmed to a size of 128px x 128px. In this embodiment, analysis is performed at a size of 128px x 128px, but this size is not necessarily required. However, for the process of converting to the spatial frequency domain described later, it is preferable to use a size that is a power of 2 (for example, 64px x 64px, 256px x 256px, etc.).

[0102] In S1802, the trimmed image obtained in S1801 is converted into the spatial frequency domain, and in this embodiment, a two-dimensional FFT is performed. The image obtained in this manner is expressed as an FFT image. Note that any known technology may be used for the FFT algorithm.

[0103] FIG. 19(a) shows an example of an FFT image actually obtained by converting a part of image data into the spatial frequency domain. With the center of the image as the origin, the horizontal axis represents the horizontal frequency and the vertical axis represents the vertical frequency, with the farther away from the origin the higher the frequency range. The FFT image shown here is obtained by calculating the sum of the absolute values ​​of the real and imaginary parts for each pixel corresponding to each frequency, normalizing the sum with the maximum and minimum values ​​(zero) to obtain the pixel value of each pixel. Normalization is performed with a range of 0 to 255 (maximum value) to produce a monochrome 8-bit image. The monochrome 8-bit FFT image obtained in this way is used as information representing the image characteristics of the small area 1701.

[0104] In S1803, it is determined whether or not calculation of image characteristics of all areas is completed based on the parameter i. If calculation is not completed, in S1804, 1 is added to the parameter i, and the process returns to S1801 to calculate image characteristics of the next area. If calculation of image characteristics of all areas is completed (if the parameter i reaches a predetermined value), the FFT image is transmitted to the signal intensity identification unit 1503 as the image characteristics.

[0105] The signal strength identification unit 1503 uses the characteristic information (FFT image) of the small area 1701 obtained by the image characteristic calculation unit 1502 to calculate the feature amount for each small area 1701, and identifies or estimates the embedding position and embedded signal strength of the additional information based on a relative comparison of the calculated feature amounts. The signal strength identification unit 1503 may be expressed as a signal strength estimation unit 1503, or may simply be expressed as an identification unit or an estimation unit.

[0106] As shown in FIG. 16, the image data of the printed matter 1601 contains the two embedding patterns (see FIGS. 12(a) and 12(b)) embedded in the printed matter 1601. Therefore, when a portion cropped from the image data is transformed into the spatial frequency domain, a power spectrum peak appears at a frequency corresponding to the periodicity (or direction of the period) of the two embedding patterns. The FFT image also exhibits the same power spectrum peak on the opposite side of the origin. As can be seen from FIG. 19(a), in the spatial frequency domain, in addition to the DC component in the center, four power spectrum peaks appear.

[0107] It is conceivable to use a calculated value (such as an average value or a maximum value) of the power spectrum of the small area 1701 as the feature amount of the small area 1701 described above, but in that case, components different from the embedding pattern (such as components having periodicity contained in the current image) will be mixed in as noise. Therefore, in this embodiment, the components of the peak (and, incidentally, the vicinity thereof) of the power spectrum of the embedding pattern are extracted and used as the feature amount. Note that the peak of the power spectrum may be simply referred to as a peak in the following explanation.

[0108] FIG. 19(b) shows another example of the division of small areas in an FFT image. When there is no tilt (rotation) between the internal imaging device 110 and the printed matter 1601, the peaks corresponding to the embedded pattern appear on the lines 1901 and 1902. However, the frequency at which the peaks appear varies depending on the distance between the internal imaging device 110 and the printed matter 1601. For example, when the distance is short, the peak appears on the low frequency side, and when the distance is long, the peak appears on the high frequency side. In the example of FIG. 19(b), six small areas A to F are set in a ring shape from the low frequency side, excluding the low frequency region near the origin.

[0109] In this example, it is determined which of these ring-shaped small areas contains the peak. For example, the maximum value of the power spectrum contained in each of the small areas is calculated, and the small area with the largest maximum value is determined as the small area with the peak. Alternatively, the average value of the power spectrum contained in each of the small areas is calculated, and the small area with the largest average value is determined as the small area with the peak.

[0110] Next, a peak corresponding to the inclination (rotation) is extracted. If there is an inclination (rotation) between the internal imaging device 110 and the printed matter 1601, the position of the peak is rotated between positions a to i as shown in Fig. 19(c). That is, the lines 1901 and 1902 rotate. Note that the positions a to i are arranged at intervals of 10°.

[0111] As an example, consider a case where a peak is detected in area C of the ring-shaped small areas 1903 (areas A to F). If the above tilt (rotation) does not occur, the four peaks in the small area 1903 appear at position a. In contrast, if the above tilt (rotation) occurs 10° to the left (counterclockwise), the peak appears at position b. Similarly, if the above tilt (rotation) occurs 30° to the right (clockwise), the peak appears at position g.

[0112] In this way, the position of the peak is detected, thereby making it possible to determine the above-mentioned tilt (rotation). Specifically, the average value X of the power spectrum contained in each of the positions a to i is calculated for each position, and it is determined that a tilt occurs corresponding to the position where the average value X is maximum. The value of this average value X can be used as the aforementioned feature amount.

[0113] In this embodiment, the inclination is determined after determining the ring-shaped small area 1903. This makes it possible to reduce erroneous detection of the inclination when a locally strong power spectrum peak occurs as noise outside the ring-shaped small area 1903.

[0114] Fig. 17(b) shows the result of calculating the feature amount for each small area 1701 in Fig. 17(a). The small areas 1701 with darker colors have higher embedded signal strength and therefore have higher feature amount values, while the small areas 1701 with lighter colors have lower embedded signal strength and therefore have lower feature amount values. In other words, it is possible to identify the embedded position and embedded signal strength of additional information based on the position of each small area 1701 and its feature amount. In this example, the small area 1701 including the circle in the upper right corner has a relatively low embedded signal strength, while the small area 1701 including the rectangle in the lower right corner has a relatively high embedded signal strength.

[0115] The information indicating the specified embedding position and embedded signal strength is expressed as embedding position information and embedded signal strength information, respectively. In this embodiment, the embedded signal strength stored in the two-dimensional array INFO[X][Y] is used. The signal strength is expressed as an integer from 0 to 255, and X and Y are index values ​​corresponding to the horizontal and vertical coordinate values ​​of the small area 1701 shown in FIG. 17(b), respectively. The embedding position information and embedded signal strength information are transmitted to the marker detection area determination unit 1504.

[0116] The marker detection area determination unit 1504 determines the position of the marker detection area 1605 in the captured image 1604 in Fig. 16 based on the above-mentioned embedding position information and embedded signal strength information. More specifically, the position of the marker detection area 1605 is determined so that the average value of the embedded signal strength of the small area 1701 included in the marker detection area 1605 becomes the largest.

[0117] 20 is a flowchart showing an example of a method for determining the position of the marker detection area 1605. For simplicity, the marker detection area 1605 is assumed to have a size in which small areas 1701 are arranged in 4 rows by 4 columns, and the margin between the small areas 1701 is assumed to be 0. In other words, the size of the marker detection area 1605 is assumed to be 512px by 512px.

[0118] In S2001, variables (Max, Ave, X, Y, Xmax, and Ymax) are initialized. The variable Max is a variable that stores the maximum value of the embedded signal strength, and its initial value is 0. The variable Ave is a variable that stores the average value of the embedded signal strength, and its initial value is 0. The variables X and Y are variables that correspond to the horizontal and vertical coordinate values ​​of the small area 1701 in the captured image 1604, respectively, and their initial values ​​are both 0. In FIG. 17(b), the small area 1701 located in the upper left of the captured image 1604 has X=0 and Y=0, and the small area 1701 located in the lower right of the captured image 1604 has X=7 and Y=13. The variables Xmax and Ymax are variables for storing the X value and Y value at which the embedded signal strength is maximized, and their initial values ​​are both 0.

[0119] In S2002, the average value Ave of the embedded signal strengths INFO[X][Y] of a small area 1701 of 4 rows and 4 columns with variables X and Y as starting points at the upper left corner is calculated.

[0120] In S2003, the average value Ave is compared with the maximum value Max. If the average value Ave is greater than the maximum value Max, the process proceeds to S2004. Since the initial value of the maximum value Max is 0, the average value Ave is initially determined to be greater than the maximum value Max unless the average value Ave is 0. If the average value Ave is equal to or less than the maximum value Max, the process proceeds to S2005.

[0121] In S2004, the maximum value Max is updated. The average value Ave is stored in the maximum value Max, and the coordinate values ​​X and Y at that time are stored in variables Xmax and Ymax, respectively, and then the process proceeds to S2005. The search for the single small area 1701 is completed by S2002 to S2004.

[0122] In S2005, it is determined whether or not the search has been completed for all of the marker detection area 1605. In this example, it is completed when X=4 and Y=10 are reached. If the search has been completed, this flowchart ends. If the search has not been completed, the coordinate values ​​X and Y are updated and the process returns to S2002. For the coordinate values ​​X and Y, for example, the coordinate value X is incremented by 1 each time, and when it exceeds X=4, which is the right end of the search, the coordinate value X is returned to 0 and the coordinate value Y is incremented by 1. In a series of searches, the order in which the coordinate values ​​X and Y are incremented may be reversed.

[0123] As a result, the coordinate values ​​X and Y positions where the average value Ave of the embedded signal strength of the small area 1701 included in the marker detection area 1605 is maximum are obtained as variables Xmax and Ymax, respectively. These variables Xmax and Ymax are converted into a coordinate system in the captured image 1604 and are determined as the position of the marker detection area 1605. In this example, variables Xmax=4 and Ymax=9 are obtained, and the position of the marker detection area 1605 in the captured image 1604 is x=512+α and y=1152+β. Here, the above-mentioned x and y are coordinates when the upper left coordinate of the captured image 1604 is x=0 and y=0, respectively. Note that α and β are offset amounts for correcting the deviation between the upper left coordinate of FIG. 17(b) and the upper left coordinate of the captured image 1604 in FIG. 17(a).

[0124] In this example, the offset amount α is α=28px, which is calculated by subtracting 128px×8=1024px from the horizontal size of the captured image 1604, 1080px, to get 56px, and dividing the result by 2. Similarly, the offset amount β is β=64px, which is calculated by subtracting 128px×14=1792px from the vertical size of 1920px, to get 128px, and dividing the result by 2. Therefore, the position of the marker detection area 1605 is x=540px and y=1216px.

[0125] In the above description, an embodiment has been exemplified in which the marker detection area 1605 is searched while being shifted by 128px, which is the size of the small area 1701, to search for the position where the embedded signal strength is maximum. However, the size of the marker detection area 1605 does not necessarily have to be a multiple of the size of the small area 1701, and the search may be performed while being shifted by 1px. Note that there may be cases in which the entire small area 1701 does not fall within the marker detection area 1605. In that case, the embedded signal strength of the entire marker detection area 1605 may be calculated based on the area ratio of the portion of the small area 1701 included in the marker detection area 1605.

[0126] In addition, in this embodiment, the position of the marker detection area 1605 is determined so that the average value of the small areas 1701 contained within the marker detection area 1605 is maximized, but other calculated values ​​(e.g., median, maximum, etc.) may be used instead of the average value.

[0127] The additional information acquisition unit 1505 executes marker detection for the marker detection area 1605 determined by the marker detection area determination unit 1504, and performs control for obtaining additional information.

[0128] First, the detection of the marker position will be described. In this embodiment, two patterns shown in Figs. 12(a) and 12(b) are embedded in the image. In the embedding example, 30 is subtracted from the B component of RGB. As a result, the patterns in Figs. 12(a) and 12(b) exhibit relatively large power spectra in two specific directions, which is also clear from Figs. 18(a) and 18(b). The additional information acquisition unit 1505 detects this power spectrum in a local area of ​​8px x 8px, thereby extracting data of "0" or "1". It is also possible to enhance the power spectrum by performing edge detection and sharpness processing as preprocessing of the detection.

[0129] When extracting the data by frequency analysis, it is necessary to accurately cut out the marker from the marker detection area 1605 of the image data. In particular, in the case of imaging using a mobile terminal such as a smartphone, since the imaging distance is difficult to be constant, a process is performed to correct the distance and accurately cut out the 8px x 8px pattern as shown in FIG. 12(a) and the like. An example of the correction method is the above-mentioned analysis of the ring-shaped small area 1701 (see FIG. 19(b)). After converting the marker detection area 1605 into the spatial frequency domain by FFT, the small area 1701 in which a peak is detected is determined by the above-mentioned analysis of the ring-shaped small area 1701. Then, based on the relationship between the imaging distance corresponding to the determined ring-shaped small area 1701 and the imaging distance as the reading reference, the enlargement / reduction ratio of the image is calculated, and the enlarged / reduction image is subjected to subsequent processing. The above correction may be performed as a preprocessing for the captured image. As another embodiment, it is also possible to enlarge or reduce the image of the marker detection area 1605 at a plurality of preset enlargement / reduction ratios and perform subsequent processing.

[0130] Next, a process is performed to correct the deviation in the coordinate position. One example of this method is to repeatedly cut out an 8px x 8px area from within the image and perform frequency analysis while shifting it by 1px each time in the horizontal and vertical directions (i.e., 8 times horizontally x 8 times vertically, for a total of 64 times), and to use the point with the strongest power spectrum as the reference position for cutting out.

[0131] After the detection of the reference position for the extraction is completed, marker detection is performed. According to the example of FIG. 13, in marker detection, first, the marker section 1301 of the unit block 1300 consisting of a specific binary pattern is detected. In the marker section 1301, a total of 100 patterns of 8px×8px as shown in FIG. 12(a) or 12(b) are arranged in 10 rows×10 columns according to a predetermined rule. Then, a check is performed to see whether this pattern exists in the marker detection area 1605 (i.e., whether a sequence of 100 bits of “0” or “1” can be obtained). Specifically, the above 100 pieces of data are extracted while shifting the position in the marker detection area 1605 by 8px, and a portion that matches the above pattern is detected. If the above-mentioned error detection or error correction data is included in the 100 bits, error detection or error correction is performed on the obtained data.

[0132] After the marker section 1301 is detected, the additional information is extracted (or may be expressed as separated) from the data section 1302 based on the position of the marker section 1301. In the above-mentioned unit block 1300, the additional information may be extracted only partially. In that case, the data of the other part that was not extracted may be supplemented with data of another block to complete the data. That is, the additional information may be extracted by extracting a first part of data in a first block, and extracting a second part of data other than the first part in a second block different from the first block. In the example of FIG. 13, when the data is completed, a sequence of "0" or "1" for 800 bits is obtained. If the data of the additional information includes error detection and error correction data like the marker section 1301, error detection and error correction are performed on the acquired data.

[0133] After the extraction of the additional information is completed as described above, data analysis is performed on the numeric string extracted as the additional information, and processing is performed to convert it into the format of the additional information when embedded. For example, the additional information to be embedded is previously converted into a numerical value in the character code "Shift JIS" as text document data.

[0134] In Shift JIS single-byte code (half-width characters), the combination of the upper 4 bits and the lower 4 bits can be used to convert to corresponding numbers and characters. For example, if the upper 4 bits are "0100" and the lower 4 bits are "0001", the character string is determined to be "A". In this way, the extracted numeric string can be converted to the corresponding character by performing a conversion process based on a conversion map that is stored or prepared in advance.

[0135] The numeric sequence extracted as additional information is temporarily stored in RAM 104, and the above conversion process can be realized relatively easily by referring to a "Shift JIS" conversion map stored in advance in secondary storage device 105 (see FIG. 1).

[0136] As an example, if the numeric string of additional information extracted by the additional information extraction unit 1702 is "0110100001100101011011000110110001101111", it is converted as follows according to the conversion map of this example: The upper 4 bits are "0110" and the lower 4 bits are "1000" to obtain the letter "h"; The upper 4 bits are "0110" and the lower 4 bits are "0101", giving the letter "e"; The upper 4 bits are "0110" and the lower 4 bits are "1100", giving the letter "l"; The four most significant bits are "0110" and the four least significant bits are "1100", giving the letter "l"; and The upper 4 bits are "0110" and the lower 4 bits are "1111", giving us the letter "o".

[0137] Therefore, the character string becomes "hello." The extracted character string is displayed on display 106 by display processing unit 401 (see FIG. 4). If the extracted character string is a URL, a screen at the URL destination may be displayed on display 106 by connecting to a network and using a specified browser. If the URL is a video site, the video may be played by display 106 and a speaker (not shown).

[0138] According to the above process, a pattern having a predetermined period (or a frequency corresponding thereto) is superimposed on a target image, so that additional information is embedded in the image. Then, a plurality of small areas 1701 in a captured image obtained by capturing the print 1601 are analyzed to calculate spatial frequency characteristics. From the spatial frequency characteristics of each small area 1701, the embedding position and embedded signal strength of the additional information in the captured image are specified or estimated. Based on the specified or estimated embedding position or embedded signal strength, the position of the marker detection area 1605 in the captured image is determined, and marker detection is performed in the determined marker detection area 1605. Then, by using the detected marker position as a reference, it becomes possible to acquire additional information in the captured image. This makes it possible to specify at which position in the captured image and with what level of signal strength the digital watermark information is embedded, and based on this, it becomes possible to control acquisition of the additional information. Therefore, according to this embodiment, it is possible to relatively easily acquire digital watermark information in various forms, which is advantageous for diversifying digital watermark reading.

[0139] <<Second embodiment>> In the first embodiment described above, an example was given of acquiring embedded additional information while changing the position of the marker detection area 1605. However, in a second embodiment, a predetermined notification may be given to a user operating the image processing device (mobile terminal) 100 via the display 106.

[0140] 21(a) shows, as an example of the notification, a display encouraging the user to move the mobile terminal 100. Here, the current marker detection area 1605 is displayed as a guide 2101, and a position where it is easier to read the additional information is visually displayed.

[0141] It is even better if a text notification such as "Please change the position over which you hold your mobile terminal" is given. The user who receives the notification can change the position of the mobile terminal 100 relative to the printed matter 1601 to read the additional information at a position where the embedded signal strength of the printed matter 1601 is high, thereby making it possible to appropriately read the additional information.

[0142] Generally, it may be difficult to adjust the focal length when capturing an image near an edge of the captured image, and the amount of light from the light source that the mobile terminal 100 typically has may not be sufficient when capturing an image. That is, even if the embedded signal strength is the same, it may be difficult to properly read the additional information in the above-mentioned cases. Therefore, if a small area 1701 with a high embedded signal strength exists outside the marker detection area 1605, capturing an image within the marker detection area 1701 where the additional information is easy to read makes it possible to further increase the embedded signal strength and read the additional information.

[0143] Fig. 21(b) shows an example of a visual notification to the user, in which the feature amount (see Fig. 17(b)) of each small area 1701 is superimposed on the captured image. In this case, it is preferable that the notification 2102 is visually displayed so that the user can understand the ease of reading the additional information.

[0144] Also, the embedded signal strength of the small area 1701 included in the marker detection area 1605 may be visually displayed. For example, the color of the frame of the guide 2101 or the color of the small area 1701 may be changed according to the embedded signal strength.

[0145] According to this embodiment, in addition to obtaining the effects described in the first embodiment, it is possible to more appropriately read additional information using the mobile terminal 100, which can be advantageous in improving the usability of the mobile terminal 100.

[0146] <<Third embodiment>> In the first embodiment described above, spatial frequency information obtained by frequency analysis is calculated as the image characteristics of the small areas 1701, and the embedded position and embedded signal strength of the additional information are specified or estimated using the image characteristics. However, this may be realized in other ways, and for example, color information in the captured image may be used.

[0147] 22 is a block diagram showing another example of the configuration of the image processing device 100 for performing the additional information reading process according to the third embodiment. The additional information reading unit 407 included in the application 400 of the image processing device 100 includes an image acquiring unit 2201, a signal intensity identifying unit 2202, a marker detection area determining unit 2203, and an additional information acquiring unit 2204.

[0148] The image acquisition unit 2201, like the image acquisition unit 1501 in the first embodiment, drives and controls the internal imaging device 110 to capture an image of the printed matter 1601 in which the additional information is embedded, and acquires image data. The image data acquired by the capture is sent to the signal intensity identification unit 2202.

[0149] The signal strength identification unit 2202 identifies or estimates the embedding position and embedded signal strength of the additional information based on the color information (RGB value) of the image data. Specifically, a reference table that allows reference to the embedded signal strength corresponding to the RGB value may be stored in advance in the application 400 of the image processing device 100, and the conversion to the embedded signal strength may be performed by referring to the reference table for each pixel of the image data. This conversion may be performed on the pixel value of each pixel of the image data, or may be performed on the representative value by defining a plurality of small areas 1701 and calculating a representative value of the RGB values ​​for each small area 1701 as in the first embodiment. Examples of the representative value here include the average value, the mode value, and the median value of the RGB values ​​of the small areas 1701.

[0150] Like the signal strength identifying unit 1503, the signal strength identifying unit 2202 may be expressed as a signal strength estimating unit 2202, or may simply be expressed as an identifying unit or an estimating unit.

[0151] 23 shows an embedded signal intensity conversion table 2301 as an example of the above-mentioned reference table. Details will be described later, but the table 2301 is a 3D lookup table (3D-LUT) in which combinations of RGB values ​​of colors to be embedded in additional information correspond to embedded signal intensities. Each of the R value, G value, and B value is one of 0, 32, 64, 96, 128, 160, 192, 224, and 255, and the table 2301 shows 9×9×9=729 pieces of data.

[0152] The table 2301 is obtained by measuring the embedded signal strength for RGB values ​​for each color in advance. For example, the additional information of the marker section 1301 and the data section 1302 illustrated in FIG. 13 is embedded in a patch image of (R, G, B)=(128, 128, 128) having a size of 240px×240px, and printed by the printer 112. The patch is imaged and the embedded signal strength is measured in the same procedure as in the first embodiment. By performing this measurement for each of the 729 colors in order, the lookup table 2301 can be created. Note that when imaging, it is preferable that the imaging conditions (e.g., the distance from the printed matter 1601 and the lighting, etc.) are fixed.

[0153] The embedding signal strength for any RGB value is calculated from the RGB value by, for example, tetrahedral interpolation of a 3D-LUT. Other examples of the interpolation method include known linear interpolation methods such as cubic interpolation and triangular prism interpolation.

[0154] The processing from the marker detection area determination unit 2203 onwards is similar to that in the first embodiment, and therefore a description thereof will be omitted here.

[0155] In the present embodiment, RGB values ​​are used as input values ​​of the conversion table 2301, but other color spaces may be used. For example, CIE-XYZ, YCbCr (YUV), CIE-Lab, etc. may be used.

[0156] Furthermore, the RGB values ​​used for embedding are usually treated as values ​​in the sRGB color space, but if the RGB values ​​of the captured image obtained from the internal imaging device 110 are device-specific RGB values ​​that are not color-managed, the values ​​will differ from the RGB values ​​of sRGB. In such a case, a 3D-LUT for performing device RGB-sRGB conversion may be stored in advance in the application 400 so that the difference between them can be corrected. In this case, the type of device is selected on the application UI when reading additional information, and color conversion is performed with reference to a 3D-LUT corresponding to the type of device, making it possible to identify or estimate the embedding signal strength corresponding to the type of device.

[0157] Furthermore, if the embedded signal strength varies depending on the type of paper on which the printed matter 1601 is printed, a 3D-LUT for each paper may be stored in advance in the application 400. In that case, when reading additional information, the type of paper on which the printed matter 1601 is printed is selected on the application UI, and the 3D-LUT corresponding to the paper type is referenced and converted, thereby making it possible to identify or estimate the embedded signal strength corresponding to the paper type.

[0158] According to this embodiment, it is possible to specify or estimate the embedding position and embedded signal strength of the additional information in a captured image based on color information in the captured image of the print 1601 in which the additional information is embedded. Therefore, the present embodiment also provides the effects described in the first embodiment.

[0159] <<Other embodiments>> In the above-described embodiments, the same additional information is repeatedly embedded in each unit block 1300 on the entire surface of the print image, but the present invention is not limited to this example. For example, different information may be embedded in each unit block 1300. Alternatively, the unit blocks 1300 may be classified into a plurality of groups, and first additional information may be embedded in blocks of a first group, and second additional information different from the first additional information may be embedded in blocks of a second group different from the first group. Note that the first group and the second group may be arranged in a biased manner in a predetermined region in the print image, or may be arranged in a dispersed manner. The number of groups may be three or more.

[0160] In addition, in each of the above-described embodiments, the area in which the additional information is embedded is a block of a predetermined size, but the present invention is not limited to this example. For example, the block does not have to be rectangular, and may be any other shape.

[0161] Furthermore, the additional information may be variable-length information, and information indicating the size of the additional information (the range in which the additional information is embedded) may be embedded in a predetermined location (e.g., the beginning) of the data portion 1302 of the unit block 1300. For example, in the case of 800 bits, some of the first bits (e.g., 80 bits) may indicate the range in which the additional information is embedded in the unit block 1300.

[0162] Furthermore, when there are multiple methods for embedding additional information, multiple rules corresponding to those methods may be predefined in the application 400, and information for identifying and selecting a corresponding rule may be embedded in, for example, a header portion at the beginning of the additional information. In this case, the additional information is extracted based on the identified rule.

[0163] In addition, in each of the above-described embodiments, the image processing device 100 and the printer 112 are connected via a network, and print data generated by the image processing device 100 is printed by the printer 112, but the present invention is not limited to this example. For example, the printer 112 may include the application 400, and the printer 112 may output a printed matter 1601 in which additional information is embedded, based on print data generated by the printer 112 itself. In other words, some of the functions of the application 400 described in the embodiments may be provided in the printer 112.

[0164] <<Program>> The present invention may be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. For example, the present invention may be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0165] <<Other>> In the above description, the printer 112 using the inkjet recording method is taken as an example, but the recording method is not limited to the above. The printer 112 may be a single-function printer having only a recording function, or a multifunction printer having multiple functions such as a recording function, a FAX function, and a scanner function. The printer 112 may also be a manufacturing device for manufacturing color filters, electronic devices, optical devices, microstructures, etc. using a predetermined recording method.

[0166] In addition, the term "recording" in this specification should be interpreted broadly. Therefore, the form of "recording" does not matter whether the object formed on the recording medium is significant information such as characters or figures, and also whether it is visible to humans or not.

[0167] In addition, the term "recording medium" should be interpreted broadly, just like the above-mentioned "recording." Therefore, the concept of a "recording medium" includes not only commonly used paper, but also any material capable of receiving ink, such as cloth, plastic film, metal plate, glass, ceramics, resin, wood, leather, etc.

[0168] Furthermore, "ink" should be interpreted broadly, just like the above-mentioned "recording." Therefore, the concept of "ink" includes not only liquid that forms an image, design, pattern, etc. by being applied onto a recording medium, but also incidental liquid that can be used for processing the recording medium, treating the ink (for example, solidifying or insolubilizing the coloring material in the ink applied to the recording medium), etc.

[0169] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0170] 110: internal imaging device (imaging unit), 1502: image characteristic calculation unit (calculation unit), 1503: signal strength determination unit (determination unit), 1504: marker detection area determination unit (determination unit), 1505: additional information acquisition unit (acquisition unit).

Claims

1. an imaging unit that captures an image of a print in which additional information is embedded by superimposing a pattern having a predetermined period or frequency on the image, and obtains the image thus obtained as a captured image; a calculation unit that analyzes a plurality of small areas in the captured image and calculates spatial frequency characteristics of each small area; a specifying unit that specifies an embedding position and an embedding signal intensity of the additional information in the captured image based on the spatial frequency characteristic; a determination unit that determines a position of a marker detection area in the captured image for detecting a marker that serves as a reference position for acquiring the additional information, based on the embedding position information and the embedded signal strength information; an acquisition unit that detects a marker within the marker detection area and acquires the additional information within the captured image based on the position of the detected marker.

2. A digital watermark analysis device comprising:

2. an imaging unit that captures an image of a print in which additional information is embedded by superimposing a pattern having a predetermined period or frequency on the image, and obtains the image thus obtained as a captured image; a determination unit that determines an embedding position and an embedding signal intensity of the additional information based on color information of pixels in the captured image and information that indicates a relationship between the color information and a signal intensity of a digital watermark that is stored in advance; a determination unit that determines a position of a marker detection area in the captured image for detecting a marker that serves as a reference position for acquiring the additional information, based on the embedding position information and the embedded signal strength information; an acquisition unit that detects a marker within the marker detection area and acquires the additional information within the captured image based on the position of the detected marker.

2. A digital watermark analysis device comprising:

3. The determining unit determines the position of the marker detection area so that an average value of the embedded signal strength of the small area included in the marker detection area becomes the largest.

2. The digital watermark analysis device according to claim 1.

4. The determination unit displays a guide corresponding to the embedding position and the embedded signal strength on a screen of a display device.

2. The digital watermark analysis device according to claim 1.

5. A step of capturing an image of a print in which additional information is embedded by superimposing a pattern having a predetermined period or frequency on the image, and acquiring the image obtained thereby as a captured image; A step of analyzing a plurality of small areas in the captured image and calculating spatial frequency characteristics of each small area; identifying an embedding position and an embedding signal intensity of the additional information in the captured image based on the spatial frequency characteristics; determining a position of a marker detection area in the captured image for detecting a marker serving as a reference position for acquiring the additional information based on the embedding position information and the embedded signal strength information; detecting a marker within the marker detection area, and acquiring the additional information within the captured image based on the position of the detected marker.

2. A digital watermark analysis method comprising the steps of:

6. A step of capturing an image of a print in which additional information is embedded by superimposing a pattern having a predetermined period or frequency on the image, and acquiring the image obtained thereby as a captured image; specifying an embedding position and an embedding signal intensity of the additional information based on color information of pixels in the captured image and information representing a relationship between the color information and a signal intensity of a digital watermark that is stored in advance; determining a position of a marker detection area in the captured image for detecting a marker serving as a reference position for acquiring the additional information based on the embedding position information and the embedded signal strength information; detecting a marker within the marker detection area, and acquiring the additional information within the captured image based on the position of the detected marker.

2. A digital watermark analysis method comprising the steps of:

Images