Information processing system and information processing method

Abstract

The present technology relates to an information processing system and an information processing method that make it possible to control an area for embedding a watermark in digital watermarking. The information processing system includes an encoder that is a trained model for generating, from an original image, message information on a message to be embedded in the original image, and embedded region information pertaining to a region where the message is embedded as a watermark, a watermarked image in which the message is embedded as a watermark. The encoder controls a region where the watermark is embedded in the original image on the basis of the embedded region information and generates the watermarked image. The present technology can be applied to, for a digital watermarking encoder.

Description

Information processing system and information processing method

[0001] The present technology relates to an information processing system and an information processing method, and more particularly to an information processing system and an information processing method that use digital watermarks.

[0002] Conventionally, a technique has been proposed in which a card management ID is assigned to a digital card in a digital trading card game to distinguish the card from other identical digital cards and increase added value (see, for example, Patent Document 1).

[0003] As a technique for adding unique information such as an ID to digital content such as a digital card, for example, digital watermarking may be used.

[0004] JP 2024-64018 A

[0005] However, with conventional digital watermarking techniques, it is difficult to control the area in which the watermark is embedded.

[0006] The present technology has been developed in light of these circumstances, and aims to enable control of the area in which a watermark is embedded in a digital watermark. Furthermore, by controlling the area in which the watermark is embedded, the noise resistance of the digital watermark is improved.

[0007] An information processing system according to a first aspect of the present technology includes an encoder, which is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about a message to be embedded in the original image, and embedding area information about an area in which the message is to be embedded as a watermark, and the encoder controls the area in which the watermark is to be embedded in the original image based on the embedding area information to generate the watermarked image.

[0008] In an information processing method according to a first aspect of the present technology, an encoder is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about a message to be embedded in the original image, and embedding area information about an area in which the message is to be embedded as a watermark, and controls the area in which the watermark is to be embedded in the original image based on the embedding area information to generate the watermarked image.

[0009] An information processing system according to a second aspect of the present technology includes a watermarked image in which a message is embedded as a watermark in an original image, and a decoder which is a trained model that restores the message from embedded region information regarding the region in which the watermark is embedded, and the decoder restores the message from the watermarked image using the embedded region information.

[0010] In a first aspect of the present technology, an encoder is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information regarding a message to be embedded in the original image, and embedding area information regarding an area in which the message is to be embedded as a watermark.The encoder controls the area in which the watermark is to be embedded in the original image based on the embedding area information, and generates the watermarked image.

[0011] In a second aspect of the present technology, a watermarked image in which a message is embedded as a watermark in an original image, and a decoder, which is a trained model that restores the message from embedded region information regarding the region in which the watermark is embedded, restores the message from the watermarked image using the embedded region information.

[0012] 1 is a diagram showing an example of a watermarked image for training a conventional digital watermark; FIG. 2 is a diagram showing an example of a watermarked image for training a conventional digital watermark; FIG. 3 is a diagram for explaining a problem with conventional digital watermarking; FIG. 4 is a diagram for explaining a problem with conventional digital watermarking; FIG. 5 is a block diagram showing an embodiment of an information processing system to which the present technology is applied; FIG. 6 is a block diagram showing an example of a functional configuration of a server; FIG. 7 is a block diagram showing an example of a functional configuration of an information processing terminal; FIG. 8 is a flowchart for explaining a training process; FIG. 9 is a diagram showing examples of an original image, a mask image, and a watermarked image; FIG. 10 is a diagram showing an example of mask processing; FIG. 11 is a diagram showing an example of mask processing; FIG. 12 is a diagram showing robustness of decoding; FIG. 13 is a flowchart for explaining encoding processing; FIG. 14 is a flowchart for explaining decoding processing; FIG. 15 is a diagram showing support for photographing an item; FIG. 16 is a diagram for explaining an example of applying the present technology to intermediate products of animation; FIG. 17 is a diagram showing an example of a collection account; FIG. 18 is a diagram for explaining an example of a countermeasure method against fraudulent registration to a collection account; FIG. 19 is a diagram showing an example of the configuration of a computer.

[0013] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 0. Background of the present technology 1. Embodiment 2. Application example 3. Modification 4. Other

[0014] <<0. Background of the Present Technology>> First, the background of the present technology will be described with reference to FIGS. 1 to 4. FIG.

[0015] The digital watermark algorithm mainly consists of two algorithms: an encoder and a decoder.

[0016] The encoder is an algorithm that encodes an image before embedding a watermark (hereinafter referred to as the original image). That is, the encoder receives the original image and message information about a message to be embedded in the original image, and embeds the message in the original image as a watermark.

[0017] The message is not limited to text, but may be an image such as a logo or symbol.

[0018] Hereinafter, embedding a message as a watermark in an original image may be simply referred to as embedding a message in the original image or embedding a watermark in the original image.

[0019] A decoder is an algorithm that decodes an image that has a watermark embedded in it (hereafter referred to as a watermarked image). That is, a decoder is an algorithm that takes a watermarked image as input and recovers the message that was embedded as a watermark in the watermarked image.

[0020] When learning a digital watermarking algorithm, the encoder and decoder are trained to be able to robustly restore the message even if various noises are superimposed on the watermarked image. For example, the decoder is trained using simulated images in which noises that may be superimposed in actual use are superimposed on the watermarked image.

[0021] However, conventional techniques have only considered rectangular regions or regions obtained by affine transformation of rectangular regions as regions into which watermarks are embedded. Furthermore, no particular attention has been paid to regions outside the region into which the watermark is embedded. For example, noise in regions outside the region into which the watermark is embedded is not particularly taken into account when training the encoder and decoder.

[0022] For example, Fig. 1 shows an example in which a rectangular image 11 with an embedded watermark is distorted by affine transformation to produce an image 12, which is used in the learning process. Note that the letters WM in Fig. 1 are a schematic representation of the area where the watermark is embedded, and are different from the actual watermark. This also applies to the subsequent drawings.

[0023] In this case, for example, a rectangular image 13 with white padding around the periphery of image 12 as shown in FIG. 1B, or a rectangular image 14 with black padding around image 12 as shown in FIG. 1C, is used for the learning process.

[0024] For example, FIG. 2 shows an example in which an image of an item 21 having a shape other than a rectangle, such as an acrylic stand with an embedded watermark, is used in the learning process.

[0025] In this case, for example, a rectangular image 22 with white padding around the item 21 as shown in FIG. 2A, or a rectangular image 23 with black padding around the item 21 as shown in FIG. 2B, is used for the learning process.

[0026] Therefore, conventional learning processes do not take into account the fact that various backgrounds may appear outside the area where the watermark is embedded. As a result, the background of the area where the watermark is embedded becomes noise that was not anticipated during learning, making it more likely that errors will occur during decoding.

[0027] For example, if the digital data of the watermarked image is not available, decoding is performed using an image taken of the watermarked image.

[0028] For example, A in Fig. 3 shows an example of a rectangular image 31 in which a watermark is embedded. For example, if image 31 is an image displayed on a display or a printed matter, and the digital data of image 31 is not available, image 32, which is a photograph of image 31, is used for decoding, as shown in B in Fig. 3. In other words, the digital data of image 32 is input to a decoder, and the watermark of image 31 in image 32 is restored.

[0029] In this case, the background may be reflected around the image 31 in the image 32, and this background may become noise, making it more likely that an error will occur during decoding.

[0030] For example, when restoring a watermark embedded in item 21 in FIG. 2, decoding is performed using an image of item 21.

[0031] In this case, as shown in A and B of Figure 4, various backgrounds may appear in the image of the item 21, and this background may become noise, making it more likely that errors will occur during decoding.

[0032] In contrast, the present technique improves the resistance to background noise outside the area where the watermark is embedded.

[0033] Furthermore, embedding a watermark in an image may result in a degradation of image quality. On the other hand, in images such as portraits and animations, there are cases where it is desirable to avoid degradation of image quality due to embedding a watermark. For example, embedding a watermark in a face image for authentication may result in degradation of image quality, which may reduce the accuracy of face authentication. Furthermore, there may be cases where there are areas in an image where degradation of image quality is undesirable, such as areas where animated characters are depicted.

[0034] However, conventional techniques do not allow explicit control of areas where the watermark is permitted to change an image and areas where it is prohibited, which can result in degradation of image quality in important areas of the original image.

[0035] In response to this, the present technology makes it possible to control the area in which a watermark is embedded. By controlling the area in which a watermark is embedded, it is possible to improve the noise resistance of the digital watermark and the image quality of the watermarked image.

[0036] In this technology, restoring a watermark includes not only restoring the content of a message embedded in an image as a watermark, but also detecting whether or not a watermark is embedded in an image.

[0037] <<1. Embodiment>> Next, an embodiment of the present technology will be described with reference to FIGS. 5 to 15 .

[0038] <Configuration Example of Information Processing System 101> FIG. 5 shows an embodiment of an information processing system 101 to which the present technology is applied.

[0039] The information processing system 101 is a system that executes processes related to digital watermarking. For example, the information processing system 101 executes processes such as embedding a watermark in an original image to generate a watermarked image, and restoring a message embedded as a watermark in the watermarked image.

[0040] The information processing system 101 includes a server 111 and information processing terminals 112-1 to 112-n, which are connected to one another via a network 121 such as the Internet.

[0041] Hereinafter, when there is no need to distinguish between the information processing terminals 112-1 to 112-n, they will be simply referred to as information processing terminals 112.

[0042] The server 111 performs a learning process for the encoder and decoder, and uses the trained encoder and trained decoder obtained by the learning process to encode and decode the watermarked image.

[0043] For example, the server 111 receives an original image, message information, and a mask image from the information processing terminal 112 via the network 121. The mask image is an image that includes embedding area information regarding an area in which a message is to be embedded as a watermark. The server 111 encodes the original image using the mask image to generate a watermarked image in which the message is embedded as a watermark in the original image. The server 111 transmits the watermarked image to the information processing terminal 112 via the network 121.

[0044] For example, server 111 receives a watermarked image and, if necessary, a mask image from information processing terminal 112 via network 121. Server 111 decodes the watermarked image using the mask image if necessary, and restores a message embedded as a watermark in the watermarked image. Server 111 transmits message information regarding the restored message to information processing terminal 112 via network 121.

[0045] The information processing terminal 112 is configured by an electronic device such as a PC (personal computer), a smartphone, a tablet terminal, a digital camera, etc. For example, the information processing terminal 112 transmits an original image, message information, and a mask image to the server 111 via the network 121, and receives a watermarked image in which the message is embedded in the original image as a watermark from the server 111. For example, the information processing terminal 112 transmits the watermarked image and, if necessary, a mask image to the server 111 via the network 121, and receives message information regarding the message restored from the watermarked image from the server 111.

[0046] <Configuration Example of Server 111> FIG. 6 shows a functional configuration example of the server 111.

[0047] The server 111 includes an input unit 151 , an information processing unit 152 , an output unit 153 , a communication unit 154 , and a storage unit 155 .

[0048] The input unit 151 includes, for example, various input devices, and is used by the user to operate the server 111 and input data necessary for processing by the server 111. The input unit 151 supplies the input data input by the user or the like to the information processing unit 152.

[0049] The information processing unit 152 includes a learning data generating unit 161 , a learning unit 162 , an encoding unit 163 , and a decoding unit 164 .

[0050] The training data generation unit 161 generates training data to be used for training the digital watermark encoder and decoder, based on input data and the like input via the input unit 151. The training data includes, for example, the original image before encoding, message information regarding a message to be embedded in the original image, and a mask image. The training data generation unit 161 supplies the generated training data to the training unit 162 and stores it in the storage unit 155.

[0051] The learning unit 162 executes a learning process for the encoder and decoder using the learning data supplied from the learning data generation unit 161 or the learning data stored in the storage unit 155. The learning unit 162 supplies the encoder, which is a trained model obtained by the learning process, to the encoding unit 163. The learning unit 162 supplies the decoder, which is a trained model obtained by the learning process, to the decoding unit 164.

[0052] For example, the encoding unit 163 acquires an original image, message information, and mask image input from the outside via the input unit 151. For example, the encoding unit 163 receives the original image, message information, and mask image from the information processing terminal 112 via the network 121 and the communication unit 154. The encoding unit 163 encodes the original image using the trained encoder and mask image. That is, the encoding unit 163 embeds the message as a watermark in the original image to generate a watermarked image. The encoding unit 163 outputs the watermarked image to the outside via the output unit 153, transmits it to the information processing terminal 112 via the communication unit 154 and the network 121, or stores it in the storage unit 155.

[0053] For example, the decoding unit 164 acquires a watermarked image and a masked image input from outside via the input unit 151. For example, the decoding unit 164 receives the watermarked image and the masked image via the network 121 and the communication unit 154. The decoding unit 164 decodes the watermarked image using a trained decoder and, if necessary, the masked image. That is, the decoding unit 164 restores the message embedded in the watermarked image as a watermark. The decoding unit 164 outputs message information relating to the restored message to the outside via the output unit 153, transmits the message information to the information processing terminal 112 via the communication unit 154 and the network 121, or stores the message information in the storage unit 155.

[0054] The output unit 153 includes an output device, such as a display device, that can output various types of information.

[0055] The communication unit 154 communicates with external devices such as the information processing terminal 112 via the network 121. The communication method of the communication unit 154 is not particularly limited, and may be either wired communication or wireless communication.

[0056] The storage unit 155 stores data necessary for processing by the server 111 .

[0057] In the following description, the phrase "via communication unit 154 and network 121" will be omitted when each unit of server 111 communicates via communication unit 154 and network 121. For example, when encoding unit 163 communicates with information processing terminal 112 via communication unit 154 and network 121, it will simply be stated that encoding unit 163 communicates with information processing terminal 112.

[0058] <Configuration Example of Information Processing Terminal 112> FIG. 7 shows a functional configuration example of the information processing terminal 112. As shown in FIG.

[0059] The information processing terminal 112 includes an input unit 201 , an image capturing unit 202 , an information processing unit 203 , an output unit 204 , a communication unit 205 , and a storage unit 206 .

[0060] The input unit 201 includes, for example, various input devices, and is used by the user to operate the information processing terminal 112 and input data necessary for processing by the information processing terminal 112. The input unit 201 supplies the input data input by the user or the like to the information processing terminal 112.

[0061] The photographing unit 202 includes a photographing device such as a camera, etc. The photographing unit 202 supplies the image obtained as a result of photographing to the information processing unit 203.

[0062] The information processing unit 203 includes an original image generating unit 211 , a mask image generating unit 212 , an encoding control unit 213 , a decoding control unit 214 , and a UI (user interface) control unit 215 .

[0063] The original image generation unit 211 generates an original image to be encoded, for example, using an image supplied from the input unit 201 or the imaging unit 202, or an image received from outside via the communication unit 205. The original image generation unit 211 supplies the generated original image to the mask image generation unit 212 and the encoding control unit 213, or stores the generated original image in the storage unit 206.

[0064] The mask image generation unit 212 generates a mask image using, for example, input data supplied from the input unit 201, an original image supplied from the original image generation unit 211, or data received from outside via the communication unit 205. The mask image generation unit 212 supplies the generated mask image to the encoding control unit 213 and the decoding control unit 214, or stores the generated mask image in the storage unit 206.

[0065] The encoding control unit 213 acquires an original image from the input unit 201, the original image generation unit 211, the communication unit 205 (or from the outside via the communication unit 205), or the storage unit 206. The encoding control unit 213 acquires a mask image from the input unit 201, the mask image generation unit 212, the communication unit 205 (or from the outside via the communication unit 205), or the storage unit 206. The encoding control unit 213 generates message information based on input data supplied from the input unit 201. The encoding control unit 213 transmits the original image, message information, and mask image to the server 111 via the communication unit 205 and the network 121, and requests the server 111 to encode the original image. The encoding control unit 213 receives a watermarked image generated by encoding the original image from the server 111 via the communication unit 205 and the network 121. The encoding control unit 213 supplies the watermarked image to the output unit 204, transmits it to the outside via the communication unit 205, or stores it in the storage unit 206.

[0066] The decode control unit 214 acquires a watermarked image from the input unit 201, the communication unit 205 (or from the outside via the communication unit 205), or the storage unit 206. The decode control unit 214 acquires a mask image from the input unit 201, the mask image generation unit 212, the communication unit 205 (or from the outside via the communication unit 205), or the storage unit 206. The decode control unit 214 transmits the watermarked image and, if necessary, the mask image to the server 111 via the communication unit 205 and the network 121, and requests the server 111 to decode the watermarked image. The decode control unit 214 receives, from the server 111, message information regarding a message restored by decoding the watermarked image via the communication unit 205 and the network 121. The decode control unit 214 supplies the message information to the output unit 204, transmits the message information to the outside via the communication unit 205, or stores the message information in the storage unit 206.

[0067] The UI control unit 215 controls the user interfaces of the input unit 201 , the image capturing unit 202 , and the output unit 204 .

[0068] The output unit 204 includes an output device, such as a display device, that can output various types of information.

[0069] The communication unit 205 communicates with external devices such as the server 111 and other information processing terminals 112 via the network 121. The communication method of the communication unit 205 is not particularly limited, and may be either wired communication or wireless communication.

[0070] The storage unit 206 stores data necessary for processing by the information processing terminal 112 .

[0071] In the following description, the phrase "via the communication unit 205 and the network 121" will be omitted when each unit of the information processing terminal 112 communicates via the communication unit 205 and the network 121. For example, when the encoding control unit 213 communicates with the server 111 via the communication unit 205 and the network 121, it will simply be described as "the encoding control unit 213 communicates with the server 111."

[0072] <Learning Process> Next, the learning process executed by the server 111 will be described with reference to the flowchart of FIG.

[0073] In step S1, the learning unit 162 acquires an original image, message information, and a mask image. That is, the learning unit 162 acquires the original image, message information, and mask image to be used for training the encoder and decoder of the watermarked image.

[0074] The learning data may be input from the outside via the input unit 151 or the communication unit 154, may be generated by the learning data generation unit 161, or may be stored in advance in the memory unit 155.

[0075] 9A shows an example of an original image 301. In the original image 301, a cat character 311 is placed.

[0076] The size of the original image 301 is, for example, H pixels (height) × W pixels (width) × 3 channels. Each channel is a color channel corresponding to, for example, the color space of R (red), G (green), and B (blue), or the color space of Y (luminance), U (chrominance), and V (chrominance). A pixel value within the range of 0 to 255 is set for each color channel of each pixel.

[0077] 9B and 9C show examples of mask images.

[0078] 9B has a size of, for example, H pixels (height) × W pixels (width) × 3 channels, similar to the original image 301. Each channel is a color channel corresponding to the same color space as the original image 301. A weight of 0 or 1 is set for each color channel of each pixel.

[0079] For example, in mask image 302A, the weight of each color channel of each pixel in the region corresponding to the foreground region in original image 301 where character 311 exists is set to 1, and this region is set as a non-masked region. On the other hand, in mask image 302A, the weight of each color channel of each pixel in the region corresponding to the background region other than the foreground region in original image 301 is set to 0, and this region is set as a masked region.

[0080] For example, by applying mask image 302A to original image 301, a watermark is embedded in the foreground areas of original image 301 corresponding to the unmasked areas of mask image 302A, and no watermark is embedded in the background areas of original image 301 corresponding to the masked areas of mask image 302A.

[0081] Hereinafter, an image in which the background region is masked (set in the mask region) like the mask image 302A will be referred to as a background mask image.

[0082] 9C is an image in which the masked and non-masked regions of the masked image 302A are inverted. That is, in the masked image 302B, the region corresponding to the foreground region of the original image 301 is set as the masked region, and the region corresponding to the background region of the original image 301 is set as the non-masked region.

[0083] For example, by applying mask image 302B to original image 301, a watermark is embedded in the background areas of original image 301 corresponding to the non-masked areas of mask image 302B, and no watermark is embedded in the foreground areas of original image 301 corresponding to the masked areas of mask image 302B.

[0084] Hereinafter, an image in which the foreground region is masked (set in the mask region), such as mask image 302B, will be referred to as a foreground mask image.

[0085] By making each pixel of the mask image three-channel, it is possible, for example, to embed a watermark in only some of the multiple color channels of the original image, or to embed a different watermark in each color channel.

[0086] For example, humans are generally less sensitive to changes in color difference than to changes in luminance. Therefore, for example, by masking only the Y channel and embedding a watermark only in the U and V channels, it is possible to generate a watermarked image in which changes in image quality as seen by the human eye are suppressed.

[0087] The weight of the mask region of the mask image may also be set to an intermediate value between 0 and 1. When the weight of the mask region is set to an intermediate value, the amount of watermark embedded in the mask region will be smaller than in the non-masked region in the watermarked image, and when the weight of the mask region is set to 0, no watermark will be embedded in the mask region in the watermarked image.

[0088] For simplicity of explanation, it is assumed that the weight of each channel is set to the same value for the same pixel in the mask image. It is also assumed that the weight of each channel of each pixel in the non-mask region is set to 1, and the weight of each channel of each pixel in the mask region is set to 0.

[0089] Furthermore, the mask image for learning may be prepared by the user, or may be automatically generated by the learning data generating unit 161 .

[0090] When the learning data generation unit 161 automatically generates a mask image, the mask image is generated based on the results of applying semantic segmentation, such as a segment anything model, to the original image. In this case, a masked region or a non-masked region may be set individually for each object region in the original image, or a masked region or a non-masked region may be set for a region that combines the regions of multiple objects. Furthermore, the masked region or the non-masked region does not necessarily have to exactly match the object region (foreground region), and may be shifted in position or shape from the object region. Furthermore, for example, one or more circular or polygonal masked or non-masked regions may be randomly arranged in the mask image.

[0091] Furthermore, this mask image may be generated based on the calculation results of a JND (Just Noticeable Difference) map, such as that shown in, for example, "Chun-Hsien Chou et al., 'A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile,' IEEE Transactions on Circuits and Systems for Video Technology, 5(6): pp. 467-476, 1995." A JND map is a map that represents the characteristics of the human eye, such as the fact that people generally have difficulty noticing slight pixel changes in areas with many edges, but are more likely to notice pixel changes in areas with few edges. For example, the learning data generation unit 161 may calculate a JND map of the original image and automatically generate a mask that applies a watermark only to areas where the calculation results of the map are below a certain level. Note that using the calculation results of the JND map as is may result in the area where the watermark is applied being discontinuous, resulting in reduced decoding performance. In response to this, for example, morphological operations such as blurring or dilation / erosion may be applied to the JND map to prevent unmasked regions from becoming too fragmented, thereby ensuring decoding performance.

[0092] In step S2, the learning unit 162 embeds a message in the original image to generate a watermarked image. Specifically, the learning unit 162 inputs the acquired original image and message information to the encoder being trained. The encoder being trained embeds the message as a watermark in the input original image, generates a watermarked image, and outputs it.

[0093] 9D shows a schematic example of a watermarked image 303. The watermarked image 303 is an image in which a watermark is embedded in the original image 301. The watermark is embedded in the watermarked image 303 without any particular area restriction. In other words, the watermark is embedded in both the foreground area where the character 311 is present and the background area where the character 311 is not present.

[0094] For example, at least one of a mask image 302A and a mask image 302B is input to an original image 301.

[0095] In step S3, the learning unit 162 determines whether or not to perform noise simulation. If it is determined that noise simulation is to be performed, the process proceeds to step S4.

[0096] In step S4, the learning unit 162 performs noise simulation on the watermarked image, that is, the learning unit 162 adds noise to the watermarked image.

[0097] Although the type of noise simulation is not particularly limited, it is desirable to perform a simulation of noise that may be added to the watermarked image during use. Specifically, for example, one or more of the following noise simulations are performed:

[0098] ・Digital editing such as flipping, cropping, resizing, blacking out, etc. ・Lossy compression such as JPEG (Joint Photographic Experts Group) ・Color conversion such as brightness, contrast, hue, sharpness, etc. ・Color restriction processing such as posterization and grayscale conversion ・Blurring such as Gaussian blur and median blur ・Noise addition such as Gaussian noise ・Noise addition using DNN (Deep Neural Network) such as CNN (Convolutional Neural Network) ・Viewpoint conversion such as rotation, translation, perspective warp, etc.

[0099] In this way, by training using an image in which noise has been added to a watermarked image, it becomes possible to decode the watermarked image robustly even if noise is added to the image during use.

[0100] Then, the process proceeds to step S5.

[0101] On the other hand, if it is determined in step S3 that noise simulation is not to be performed, the process of step S4 is skipped and the process proceeds to step S5.

[0102] In step S5, the learning unit 162 applies mask processing to the watermarked image. Specifically, the learning unit 162 updates the pixel value W of each channel of each pixel in the watermarked image using the following equation (1).

[0103] W←M×W+(1-M)×O...(1)

[0104] M indicates the weight of the mask image, and O indicates the pixel value of the original image.

[0105] In this way, pixel values ​​of the intermediate image, which is a watermarked image in which a watermark is embedded in an original image without specifying an area where the watermark is to be embedded, and pixel values ​​of the original image are weighted and added using the weight of the mask image for each color channel of each pixel. In this way, a part of the watermark is removed from the watermarked image using the mask image. In other words, the area masked by the mask image in the watermarked image is replaced with the original image, and the watermark is removed.

[0106] For example, as shown in FIG. 10, when a mask image 302A is applied to a watermarked image 303, a watermarked image 304A is generated in which the watermark in the background area other than the foreground area where the character 311 is present is removed.

[0107] For example, as shown in FIG. 11, when mask image 302B is applied to watermarked image 303, watermarked image 304B is generated in which the watermark in the foreground area where character 311 is present has been removed.

[0108] In this way, the learning unit 162 controls the area in which the watermark is embedded in the original image based on the mask image.

[0109] In step S6, the training unit 162 decodes the watermarked image. Specifically, the training unit 162 inputs the watermarked image, from which part of the watermark has been removed, to the decoder being trained. The decoder being trained restores and outputs the message embedded as a watermark in the input watermarked image.

[0110] In step S7, the learning unit 162 evaluates the model being learned. For example, the learning unit 162 calculates the objective function L of the following equation (2).

[0111] L=α×L1+(1-α)×L2...(2)

[0112] The objective function L is a function that performs weighted addition of the evaluation value L1 and the evaluation value L2 using a weight α.

[0113] The weight α takes a value within the range of 0 to 1 and is set by the user, for example.

[0114] The evaluation value L1 is an index for evaluating the image quality of the watermarked image after embedding the watermark. In other words, the evaluation value L1 is an index that indicates the degree of degradation of the watermarked image. For example, the evaluation value L1 is calculated based on the similarity between the original image and the watermarked image.

[0115] For example, MSE (Mean Squared Error) is used for the evaluation value L1. Specifically, the average value for each pixel of the squared difference between the pixel values ​​of each pixel of the original image and the watermarked image is used for the evaluation value L1. In this case, the higher the similarity between the original image and the watermarked image, the smaller the evaluation value L1.

[0116] For example, the inverse of SSIM (Structural Similarity Index Measure) is used as the evaluation value L1. SSIM evaluates the similarity of images based on values ​​calculated for each small region (window) within the image, and the value increases as the similarity of the images increases. In this case, the evaluation value L1 decreases as the similarity between the original image and the watermarked image increases.

[0117] For example, the inverse of LPIPS (Learned Perceptual Image Patch Similarity) is used as the evaluation value L1. LPIPS is one of the indexes of image similarity calculated based on the output of a DNN, and the higher the similarity of the images, the larger the value. In this case, the higher the similarity between the original image and the watermarked image, the smaller the evaluation value L1.

[0118] The evaluation value L2 is an index for evaluating the accuracy of the restored message, in other words, an index indicating the failure rate of decoding the watermarked image.

[0119] For example, the evaluation value L2 is calculated based on the distance (e.g., L2 norm, L1 norm, etc.) between a vector representing the original message embedded in the watermarked image and a vector representing the message restored from the watermarked image. The evaluation value L2 becomes smaller as the similarity between the original message and the restored message increases.

[0120] Here, there is a trade-off between evaluation value L1 and evaluation value L2. For example, if you try to suppress degradation of the image quality of the watermarked image, that is, if you try to make the watermarked image closer to the original image, the amount of information in the watermark will decrease, and the accuracy of restoring the message will decrease. On the other hand, if you increase the amount of information in the watermark, the accuracy of restoring the message will improve, but the watermarked image will change more from the original image, and the image quality of the watermarked image will decrease.

[0121] Therefore, increasing the weight α of the objective function L will result in training that emphasizes the image quality of the watermarked image, while decreasing the weight α of the objective function L will result in training that emphasizes the accuracy of message decoding.

[0122] In step S8, the learning unit 162 adjusts the parameters. Specifically, the learning unit 162 adjusts the parameters of the encoder and decoder under learning so as to minimize the objective function L.

[0123] In step S9, the learning unit 162 determines whether to continue learning. For example, if the learning unit 162 determines that the objective function L has not converged, it determines to continue learning, and the process returns to step S1.

[0124] Thereafter, the processes of steps S1 to S9 are repeatedly executed until it is determined in step S9 that learning should be stopped.

[0125] On the other hand, in step S9, for example, if the learning unit 162 determines that the objective function L has converged, it determines to stop learning, and the process proceeds to step S10.

[0126] In step S10, the learning unit 162 outputs a trained model. Specifically, the learning unit 162 supplies the trained encoder to the encoding unit 163 and supplies the trained decoder to the decoding unit 164.

[0127] The learning process then ends.

[0128] For example, if the encoder and decoder are trained using a background mask image, the encoder and decoder are trained to be able to robustly decode a message regardless of the background surrounding the content that constitutes the foreground region. For example, as shown in Figure 12, even if the background of the character 311 changes, the message can be robustly restored.

[0129] On the other hand, if the encoder and decoder are trained using a foreground mask image, the encoder and decoder are trained to robustly recover the message from a watermarked image in which the watermark is embedded only in the background region and not in the foreground region, which is the region where the image should not be altered. This allows robust recovery of the message while avoiding degradation of the image quality in the important foreground region of the watermarked image.

[0130] <Encoding Process> Next, the encoding process executed by the server 111 will be described with reference to the flowchart of FIG.

[0131] In step S101, the encoding unit 163 acquires the original image, message information, and mask image from the information processing terminal 112. For example, the encoding unit 163 receives the original image, message information, and mask image transmitted from the information processing terminal 112.

[0132] The original image may be, for example, one obtained or created by the user, or one generated by the original image generation unit 211 of the information processing terminal 112. The original image may also be, for example, the image itself that is to be watermarked, or may be a photograph or scan of the item that is to be watermarked.

[0133] The mask image may be, for example, one obtained or created by the user, or may be one generated by the mask image generation unit 212 of the information processing terminal 112. For example, the mask image generation unit 212 may automatically generate the mask image based on the original image, similar to the learning data generation unit 161 described above.

[0134] The message included in the message information may be created by the user, or may be automatically generated by the encoding control unit 213 of the information processing terminal 112, for example.

[0135] In step S102, the encoding unit 163 determines whether the mask image satisfies the condition. For example, if the proportion of pixels in the mask image that have a pixel value of 1 is less than a predetermined threshold, that is, if the proportion of non-mask regions is less than a predetermined threshold, the encoding unit 163 determines that the mask image does not satisfy the condition, and the process proceeds to step S103.

[0136] In step S103, the encoding unit 163 acquires a mask image from the information processing terminal 112. Specifically, the encoding unit 163 requests the information processing terminal 112 to regenerate a mask image, and receives the regenerated mask image from the information processing terminal 112.

[0137] For example, the mask image may be regenerated by the user or by the mask image generating unit 212. At this time, the mask image is regenerated so that the proportion of the non-masked region is increased.

[0138] Thereafter, the process returns to step S102, and steps S102 and S103 are repeatedly executed until it is determined in step S102 that the mask image satisfies the conditions.

[0139] On the other hand, in step S102, for example, if the proportion of pixels in the mask image with a pixel value of 1 is equal to or greater than a predetermined threshold, i.e., if the proportion of non-mask areas is equal to or greater than a predetermined threshold, the encoding unit 163 determines that the mask image satisfies the conditions, and processing proceeds to step S104.

[0140] In step S104, the encoding unit 163 embeds a message in the original image to generate a watermarked image. Specifically, the encoding unit 163 inputs the original image, message information, and mask image to an encoder. The encoder embeds the message as a watermark in the original image to generate a watermarked image.

[0141] In this case, the encoder may avoid the masked area that is masked by the mask image and focus on embedding the watermark in the unmasked area, or may embed the watermark in the original image on the assumption that the information in the masked area will disappear.

[0142] In step S105, the encoding unit 163 applies a mask process to the watermarked image. That is, similar to the process in step S5 of Fig. 8, the encoder applies a mask process to the watermarked image, replaces the area masked by the mask image with the original image, and removes the watermark.

[0143] For example, as described above, the weight of the mask area of ​​the mask image may be set to an intermediate value between 0 and 1, and the amount of watermark in the mask area may be reduced based on the weight of the mask image.

[0144] In this way, the encoding unit 163 controls the area in which the watermark is embedded in the original image based on the mask image.

[0145] In step S106, the encoding unit 163 outputs the watermarked image. Specifically, the encoder outputs the watermarked image. The encoding unit 163 transmits the watermarked image to the information processing terminal 112.

[0146] Then the encoding process ends.

[0147] <Decoding Process> Next, the decoding process executed by the server 111 will be described with reference to the flowchart of FIG.

[0148] In step S151 , the decoding unit 164 acquires the watermarked image from the information processing terminal 112 .

[0149] For example, if a user possesses digital data of a watermarked image to be decoded, the user uses the information processing terminal 112 to transmit the digital data of the watermarked image to the server 111 .

[0150] For example, when decoding a watermark embedded in an item, the user uses the information processing terminal 112 to photograph the item.

[0151] Possible target items include, for example, images displayed on a display screen, printed or scanned images such as posters and flyers, etc. Also, possible target items include, for example, objects on which two-dimensional images are printed or attached, such as acrylic stands and clear cards.

[0152] Furthermore, the target item may be, for example, a three-dimensional object such as a figurine. When the item is a three-dimensional object, for example, the watermark may be embedded in a part of the item that is close to a plane, or a planar part may be added to the item and the watermark may be embedded in that part.

[0153] In this case, for example, the output unit 153 of the information processing terminal 112 may display a list of items that can be candidates for decoding, as shown in FIG. 15A, under the control of the UI control unit 215.

[0154] In response, the user selects an item to be decoded from the displayed list of decoding candidates via the input unit 151 of the information processing terminal 112 .

[0155] In response to this, for example, the photographing unit 202 of the information processing terminal 112 may, under the control of the UI control unit 215, display a frame 351 indicating the outline of the selected item on the monitor of the photographing unit 202, as shown in B of Figure 15, and guide the user to photograph the item by aligning it with the frame 351.

[0156] In response to this, for example, the user photographs an item to be decoded using the photographing unit 202 of the information processing terminal 112. At this time, it is desirable that the user photograph the item by aligning the position of the item with the frame 351 displayed on the monitor of the photographing unit 202.

[0157] This allows the watermark to be decoded at a scale and position close to when the item was encoded.

[0158] It is not necessary to align the item position with the frame 351 exactly.

[0159] Furthermore, for example, if a watermark is embedded in a partial area of ​​an item, a frame indicating that area may be displayed on the monitor of the image capturing unit 202 .

[0160] The photographing unit 202 supplies digital data of the image of the item (watermarked image) obtained by photographing to the decoding control unit 214. The decoding control unit 214 transmits the watermarked image to the server 111.

[0161] If necessary, the decoding control unit 214 transmits a mask image corresponding to the watermarked image to the server 111 .

[0162] This mask image is, for example, an image in which the foreground region of the watermarked image to be decoded is set as a non-masked region or a masked region. The mask image may be prepared by the user, or may be automatically generated by the mask image generation unit 212, similar to the above-mentioned learning data generation unit 161. Furthermore, if the user owns a mask image that was used when the watermarked image was encoded, that mask image may be used.

[0163] In step S152, the decoding unit 164 determines whether or not a mask image has been acquired from the information processing terminal 112. If it is determined that a mask image has not been acquired, the process proceeds to step S153.

[0164] In step S153, the decoding unit 164 decodes the watermarked image. Specifically, the decoding unit 164 inputs the watermarked image to a decoder. The decoder restores the message embedded as a watermark in the watermarked image.

[0165] Thereafter, the process proceeds to step S155.

[0166] On the other hand, if it is determined in step S152 that a mask image has been acquired, the process proceeds to step S154.

[0167] In step S154, the decoding unit 164 decodes the watermarked image using the mask image. Specifically, the decoding unit 164 inputs the watermarked image and the mask image to the decoder. The decoder controls the area of ​​the watermarked image to be decoded based on the mask image. For example, the decoder may decode only the unmasked area of ​​the watermarked image indicated by the mask image, or may focus on decoding the unmasked area of ​​the watermarked image, or may prioritize decoding the unmasked area of ​​the watermarked image. The decoder then restores the message embedded as a watermark in the watermarked image.

[0168] Thereafter, the process proceeds to step S155.

[0169] In step S155, the decoding unit 164 outputs the restored message. Specifically, the decoder outputs the restored message. The encoding unit 163 transmits message information regarding the restored message to the information processing terminal 112.

[0170] After that, the decoding process ends.

[0171] As described above, a watermarked image can be generated by controlling the area in the original image where the watermark is embedded. This allows the area in which the watermark is embedded to be adjusted depending on the purpose of the watermarked image. For example, if it is desired to prevent deterioration of the image quality of the foreground area of ​​the original image, the watermark can be embedded in the background area. For example, if it is desired to prevent the foreground area of ​​the original image from being cut out, the watermark can be embedded in the foreground area.

[0172] Additionally, training the decoder using watermarked images with adjusted watermark embedding regions improves the decoder's noise robustness, i.e., it can accurately recover the message regardless of the background outside the watermarked region.

[0173] <<2. Application Examples>> Next, application examples of the present technology will be described.

[0174] <Application to Animation Intermediates> The present technology can be applied to animation intermediates such as storyboards, for example.

[0175] 16, for example, a message is embedded as a watermark in an original image 401, which is digital data, to generate a watermarked digital image 402. Here, a different ID is included in the message for each recipient, and a watermarked image 402 with a different message for each recipient is generated.

[0176] At this time, in order to prevent degradation of the image quality of the character 411 in the watermarked image 402, the watermark may be embedded in the background area around the character 411. Alternatively, for example, as a measure to prevent the character 411 from being cut out and leaked, the watermark may be embedded in the foreground area where the character 411 exists.

[0177] Next, each watermarked image 402 is printed on paper or the like to generate a plurality of distribution items 403. Each distribution item 403 is distributed to each recipient. At this time, the distribution source records and manages the message (ID) embedded in each distribution item 403 distributed to each recipient.

[0178] Next, assume that at least a portion of the distributed material 403 is leaked from the recipient as leaked material 404. At this time, noise may be added to or edited in the leaked material 404. In this example, editing has been performed to add a picture of a bell to the character 411.

[0179] Next, the leaked object 404 is scanned, a leaked image 405 is generated as digital data, and is made public or distributed on the Internet or the like.

[0180] In this case, by using this technology, even if noise or editing has been added to the leaked image 405 (or the leaked material 404 that is the source of the leaked image 405), the message can be accurately restored. This makes it possible to identify the leak route of the distributed material 403 based on the ID included in the restored message.

[0181] <Application to Real Goods> The present technology can be applied to real goods (hereinafter simply referred to as goods) that are sold or distributed for the purpose of fan engagement, for example.

[0182] For example, as the number of goods owned by a user increases, organizing the goods becomes cumbersome. For example, there is a need to easily view the goods owned. For example, there is a need to show the goods collection owned by a user to friends or to make it public on social media or the like. For example, there is a need for the management side that plans and sells goods to be able to recognize the goods collection owned by a user and to grant rewards such as awards and benefits depending on the level of the collection.

[0183] Alternatively, for example, a user may be able to take a photograph of a product and register the resulting image (hereinafter referred to as a product image) in his or her own collection account for management.

[0184] FIG. 17 shows an example of a collection account.

[0185] The collection account displays, for example, a list of merchandise images registered by the user. For example, it is also possible to select each merchandise image and display them individually.

[0186] This allows, for example, a user to easily view a collection of goods that the user owns. For example, the user can show the collection of goods that the user owns to friends or publish it on social media, etc. For example, the user can record information related to the registered goods (for example, the event at which the goods were purchased, etc.).

[0187] For example, the management side can provide or sell special content related to goods to users who have registered goods. Examples of such content include the following:

[0188] ・In the case of merchandise related to athletes, the story behind the athlete when the photo related to the merchandise was taken. ・The original artwork of the image used on the merchandise. ・In the case of merchandise related to social games, the costume of the character that the merchandise is based on.

[0189] For example, the management side can provide various rewards to users when the goods registered in each user's collection account meet certain conditions.

[0190] For example, the following are examples of achievement conditions:

[0191] ・Own a certain number of items of the target character. ・Own the target character's goods for a certain number of years. ・Own a combination of the target character's goods.

[0192] For example, the following are examples of rewards when conditions are met:

[0193] ・Images for SNS (Social Networking Service) icons ・Priority rights for the next match or event participation ・In the case of card games, the right to participate in tournaments for registered cards ・In the case of card games, the link between real and digital games

[0194] In this case, it is conceivable that, for example, a user who has not purchased merchandise legitimately may obtain merchandise images of other users registered in their collection accounts and register them in their own collection accounts without permission.

[0195] Alternatively, a message including an ID for individually identifying each item of merchandise may be embedded in the merchandise as a watermark.

[0196] Here, with reference to FIG. 18, an example of a countermeasure against fraudulent registration of a user who has not purchased goods legitimately to a collection account will be described.

[0197] First, for example, it is assumed that user A photographs a good 451 with the information processing terminal 112-1 and registers the obtained good image 452 in his / her collection account.

[0198] In response to this, the management side decodes the watermark embedded in the goods image 452 and restores the ID unique to the goods 451. The management side registers the goods image 452 together with the ID in the collection account of user A.

[0199] In response to this, for example, assume that user B obtains a digital copy of goods image 452 or takes a photograph of goods image 452 and registers the resulting copy image 453 in user B's collection account.

[0200] In response to this, the management side decodes the watermark embedded in the copied image 453 and restores the ID unique to the goods 451 shown in the copied image 453. Here, since the restored ID has already been registered in the collection account of user A, a message such as "This ID has already been registered" is displayed on user B's information processing terminal 112-2. Furthermore, a message such as "Do you want to request a transfer of rights to the current owner?" is displayed on the information processing terminal 112-2.

[0201] In response to this, if user B applies for a transfer of rights, a message notifying confirmation of the transfer of rights, stating "Mr. B is trying to register your item C. Do you want to allow this?", is displayed on user A's information processing terminal 112-1.

[0202] In response to this, user A determines whether or not to permit the right transfer and inputs the result of the determination into information processing terminal 112-1. For example, if user A does not permit the right transfer, user B cannot register copy image 453 in his collection account.

[0203] This prevents the goods 451 owned by user A from being registered in user B's collection account without permission.

[0204] On the other hand, for example, if user B legitimately receives goods 451 from user A and attempts to register an image of goods 451 in user B's collection account, registration can be easily made with user A's permission.

[0205] It is desirable to sell goods in such a way that the area where the watermark is embedded cannot be photographed when the goods are unopened, so that a malicious person cannot register an ID before the legitimate purchaser has registered the goods' ID.

[0206] Furthermore, for example, a QR code (registered trademark) for confirmation by the purchaser that the ID of the goods is unregistered before purchase may be printed on the package of the goods before sale.

[0207] As another method for assigning an ID to the goods 451 so that the user can register it, for example, a method using an NFC (Near Field Communication) tag or a QR code can be considered.

[0208] When an NFC tag is used, it is possible to prevent changes to the design of the goods 451. However, this incurs additional costs. Furthermore, if the information processing terminal 112 does not have an NFC reader, it is not possible to read the ID.

[0209] When a QR code is used, almost no additional cost is incurred. Also, if the information processing terminal 112 is equipped with a camera, the ID can be read. On the other hand, the design of the goods 451 needs to be changed more.

[0210] On the other hand, when a watermark is used, almost no additional cost is incurred. Also, if the information processing terminal 112 is equipped with a camera, the ID can be read. Furthermore, changes to the design of the goods 451 can be suppressed.

[0211] <Application to Facial Images for Authentication> For example, the present technology can be applied to facial images for authentication used in passports and the like.

[0212] In this case, for example, in order to improve the accuracy of authenticating a face image, a watermark can be embedded in the background area outside the face area of ​​the face image.

[0213] Furthermore, for example, in order to prevent the forgery of a face image, a watermark can be embedded in the face area of ​​the face image.

[0214] <<3. Modifications>> Modifications of the above-described embodiments of the present technology will now be described.

[0215] <Variations Regarding Sharing of Processing> The sharing of processing in the information processing system 101 described above can be changed as appropriate. For example, some or all of the processing of the server 111 may be performed by the information processing terminal 112, or some or all of the processing of the information processing terminal 112 may be performed by the server 111. For example, the processing of the server 111 may be shared and executed by a plurality of servers.

[0216] Specifically, for example, the information processing terminal 112 may perform encoding and decoding. In this case, the information processing terminal 112 can generate and decode a watermarked image without using the server 111.

[0217] For example, the information processing terminal 112 may perform encoding and the server 111 may perform decoding. In this case, the information processing terminal 112 can generate a watermarked image without using the server 111.

[0218] For example, the learning process for the encoder and decoder and the actual encoding and decoding may be performed by different servers.

[0219] For example, printing of a watermarked image for distribution may be performed by either the server 111 or the information processing terminal 112 .

[0220] <Other Variations> In the above explanation, an example has been shown in which a weight can be set for each color channel of each pixel of the mask image, but it is also possible to set a weight for each pixel of the mask image without distinguishing between color channels (common to all color channels).

[0221] <Application Examples of the Present Technology> The present technology can be applied to, for example, a system or device that executes a learning process for at least one of encoding and decoding a digital watermark. The present technology can be applied to, for example, a system or device that executes at least one of encoding and decoding a digital watermark.

[0222] This technology can be applied to both perceptible and imperceptible digital watermarks.

[0223] <<4. Others>> <Description of a Computer to which the Present Technology is Applied> The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0224] FIG. 19 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0225] In the computer, a processing circuit 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0226] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0227] The input unit 1006 may include physical or virtual operating means, such as a keyboard, mouse, or touch panel, that a user operates to input information, as well as means for the user to input information by voice, gaze, or the like. The input unit 1006 may also include sensors for inputting various physical quantities to the computer. For example, the input unit 1006 may include sensors, such as a camera or microphone, that acquire physical quantities such as light (including infrared light other than visible light) and sound. The input unit 1006 may also include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, and distance. The output unit 1007 may include means, such as a display, speaker, or haptic device, that present information to a user by stimulating the user's perception. The storage unit 1008 may be composed of a hard disk, non-volatile or volatile memory, or the like, and stores various information (including programs). The communication unit 1009 may be a network interface or the like, and may perform wired or wireless communication with the outside world. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0228] The processing circuit 1001 includes a processor that executes a program, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The processing circuit 1001 (the processor) performs the above-described series of processes by loading a program stored in a storage unit 1008 into a RAM 1003 via an input / output interface 1005 and a bus 1004 and executing the program. The processing circuit 1001 can output the processing results of the series of processes from an output unit 1007, for example, via the bus 1004 and the input / output interface 1005, as necessary. The processing circuit 1001 can also store the processing results in the storage unit 1008 or transmit them from a communication unit 1009.

[0229] The program executed by the computer (processing circuit 1001) can be provided by being recorded on removable media 1011 such as package media, for example. The program can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0230] In a computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 from another device such as a server via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0231] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0232] The processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0233] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0234] When the above-described series of processes are performed by a computer executing a program, for example, the input unit 1006 functions as the input unit 151 in Fig. 6 or the input unit 201 in Fig. 7, the output unit 1007 functions as the output unit 153 in Fig. 6 or the output unit 204 in Fig. 7, and the communication unit 1009 functions as the communication unit 154 in Fig. 6 or the communication unit 205 in Fig. 7. For example, the processing circuit 1001 (its processor) executes a program to function as the information processing unit 152 in Fig. 6 or the information processing unit 203 in Fig. 7.

[0235] In this specification, a system refers to one component or a collection of multiple components (devices, modules (components), etc.). Therefore, one or multiple components of a computer, for example, only a processor, or a combination of a processor and memory (for example, only the processing circuit 1001, or a combination of the processing circuit 1001 and the bus 1004, etc.), is a system. With respect to a collection of multiple components, it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, or a single device housed in a single housing with multiple modules, are both systems. Furthermore, for example, an entire computer or a combination of a computer and another device, such as a server (not shown), is also a system.

[0236] The components (blocks) of the devices illustrated in the present specification are conceptual blocks of functions, and the actual device does not necessarily have to have the configuration shown in the drawings. In other words, the device may have any configuration in which the functions of the illustrated components are divided and / or integrated in any unit, for example, a configuration having a single block in which the functions of all the components are integrated.

[0237] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0238] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0239] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0240] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0241] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0242] (1) An information processing system comprising an encoder which is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about a message to be embedded in the original image, and embedding region information about a region in which the message is to be embedded as the watermark, wherein the encoder controls a region in which the watermark is to be embedded in the original image based on the embedding region information to generate the watermarked image. (2) The information processing system described in (1), wherein the encoder embeds a smaller amount of the watermark in a first region of the original image indicated by the embedding region information than in a second region of the original image other than the first region. (3) The information processing system described in (2), wherein the embedding region information indicates a weight for each pixel of the original image, and the encoder generates the watermarked image by adding, for each pixel, pixel values ​​of the original image and pixel values ​​of an intermediate image in which the watermark is embedded in the original image without specifying a region in which the watermark is to be embedded, using the weight. (4) The information processing system according to (3), wherein the embedding region information indicates a weight for each color channel of each pixel of the original image, and the encoder generates the watermarked image by adding pixel values ​​of the original image and pixel values ​​of the intermediate image for each color channel of each pixel using the weight. (5) The information processing system according to any of (2) to (4), wherein the encoder embeds the watermark in the original image using the embedding region information if the proportion of the second region in the original image is equal to or greater than a predetermined threshold. (6) The information processing system according to any of (2) to (5), wherein the encoder embeds the watermark only in the second region. (7) The information processing system according to any of (2) to (6), wherein the first region is one of a foreground region and a background region of the original image, and the second region is the other of the foreground region and the background region of the original image.(8) The information processing system according to any one of (1) to (7), wherein the encoder embeds the watermark in only some of the multiple color channels of the original image. (9) The information processing system according to any one of (1) to (8), wherein the encoder is trained using training data including the original image, the message information, and the embedding region information. (10) The information processing system according to (9), wherein the encoder is trained together with a decoder that restores the message from the watermarked image. (11) The information processing system according to (10), wherein the encoder is trained using an objective function that performs a weighted addition of a first evaluation value based on the similarity between the watermarked image generated by the encoder and the original image and a second evaluation value based on the accuracy of the message restored by the decoder. (12) An information processing method, comprising: an encoder which is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about the message to be embedded in the original image, and embedding region information about a region in which the message is to be embedded as the watermark, and controls a region in which the watermark is to be embedded in the original image based on the embedding region information to generate the watermarked image. (13) An information processing system, comprising: a decoder which is a trained model that restores a watermarked image in which a message is embedded as a watermark in an original image and embedding region information about a region in which the watermark is to be embedded, and the decoder restores the message from the watermarked image using the embedding region information. (14) The information processing system according to (13), in which the decoder restores the message by controlling a region to be decoded in the watermarked image based on the embedding region information. (15) The information processing system according to (13) or (14), wherein the decoder is trained together with an encoder that generates the watermarked image from the original image using training data including the original image, message information related to the message, and the embedding region information.(16) The information processing system according to (15), wherein the decoder is trained using the watermarked image generated by the encoder by controlling the area where the watermark is embedded in the original image based on the embedding area information. (17) The information processing system according to (16), wherein the decoder is trained using an objective function that weights and adds a first evaluation value based on the similarity between the watermarked image generated by the encoder and the original image, and a second evaluation value based on the accuracy of the message restored by the decoder. (18) The information processing system according to any of (13) to (17), wherein the amount of the watermark embedded in a first area of ​​the watermarked image indicated by the embedding area information is smaller than that in a second area of ​​the watermarked image other than the first area. (19) The information processing system according to (18), wherein the watermark is embedded only in the second area of ​​the watermarked image. (20) The information processing system according to any one of (13) to (19), wherein the watermark is embedded in only some of the multiple color channels of the watermarked image.

[0243] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0244] 101 Information processing system, 111 Server, 112-1 to 112-n Information processing terminals, 152 Information processing unit, 161 Learning data generation unit, 162 Learning unit, 163 Encoding unit, 164 Decoding unit, 202 Imaging unit, 203 Information processing unit, 211 Original image generation unit, 212 Mask image generation unit, 213 Encoding control unit, 214 Decoding control unit, 215 UI control unit

Claims

1. An information processing system comprising an encoder which is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about a message to be embedded in the original image, and embedding area information about an area in which the message is to be embedded as a watermark, wherein the encoder controls the area in which the watermark is to be embedded in the original image based on the embedding area information to generate the watermarked image.

2. The information processing system of claim 1, wherein the encoder embeds a smaller amount of the watermark in a first region of the original image indicated by the embedding region information than in a second region of the original image other than the first region.

3. The information processing system of claim 2, wherein the embedding area information indicates a weight for each pixel of the original image, and the encoder generates the watermarked image by adding, for each pixel, the pixel values ​​of the original image and the pixel values ​​of an intermediate image in which the watermark is embedded in the original image without specifying an area in which the watermark is to be embedded, using the weight.

4. The information processing system of claim 3, wherein the embedded region information indicates a weight for each color channel of each pixel of the original image, and the encoder generates the watermarked image by adding the pixel values ​​of the original image and the pixel values ​​of the intermediate image using the weight for each color channel of each pixel.

5. The information processing system according to claim 2, wherein the encoder embeds the watermark in the original image using the embedding area information if the proportion of the second area relative to the original image is equal to or greater than a predetermined threshold.

6. The information processing system according to claim 2, wherein the encoder embeds the watermark only in the second region.

7. An information processing system as described in claim 2, wherein the first region is one of the foreground region and the background region of the original image, and the second region is the other of the foreground region and the background region of the original image.

8. The information processing system according to claim 1, wherein the encoder embeds the watermark in only some of the color channels of the original image.

9. The information processing system according to claim 1, wherein the encoder is trained using training data including the original image, the message information, and the embedded region information.

10. The information processing system of claim 9, wherein the encoder is trained with a decoder that recovers the message from the watermarked image.

11. The information processing system described in claim 10, wherein the encoder is trained using an objective function that weights and adds a first evaluation value based on the similarity between the watermarked image generated by the encoder and the original image, and a second evaluation value based on the accuracy of the message restored by the decoder.

12. An information processing method in which an encoder is a trained model that generates a watermarked image in which a message is embedded as a watermark from an original image, message information about a message to be embedded in the original image, and embedding area information about an area in which the message is to be embedded as a watermark, and controls the area in which the watermark is to be embedded in the original image based on the embedding area information to generate the watermarked image.

13. An information processing system comprising: a watermarked image in which a message is embedded as a watermark in an original image; and a decoder which is a trained model that restores the message from embedded region information regarding the region in which the watermark is embedded, wherein the decoder restores the message from the watermarked image using the embedded region information.

14. The information processing system according to claim 13, wherein the decoder controls the area to be decoded in the watermarked image based on the embedded area information to restore the message.

15. The information processing system of claim 13, wherein the decoder is trained together with an encoder that generates the watermarked image from the original image using training data including the original image, message information related to the message, and the embedded region information.

16. The information processing system according to claim 15, wherein the decoder is trained using the watermarked image generated by the encoder by controlling the area in which the watermark is embedded in the original image based on the embedding area information.

17. The information processing system described in claim 16, wherein the decoder is trained using an objective function that weights and adds a first evaluation value based on the similarity between the watermarked image generated by the encoder and the original image, and a second evaluation value based on the accuracy of the message restored by the decoder.

18. An information processing system as described in claim 13, wherein the amount of watermark embedded in a first region of the watermarked image indicated by the embedding region information is smaller than that in a second region of the watermarked image other than the first region.

19. The information processing system of claim 18, wherein the watermark is embedded only in the second region of the watermarked image.

20. The information processing system of claim 13, wherein the watermark is embedded in only some of the color channels of the watermarked image.

Images