End-to-end watermarking system

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for jointly training an encoder that generates a watermark and a decoder that decodes data items encoded within the watermark. The training includes obtaining a plurality of training images and data items. For each training image, a first watermark is generated using the encoder, and a subsequent second watermark is generated by tiling two or more of the first watermarks. The training image is watermarked using the second watermark to generate a first error value, and distortion is added to the watermarked image. A distortion detector predicts the distortion on which the modified distorted image is based. The modified image is decoded by the decoder to generate a predicted data item and a second error value. Training parameters of the encoder and the decoder are adjusted based on the first error value and the second error value.

Description

Technical Field

[0001] This specification generally relates to data processing and techniques for embedding watermarks in digital content and for recovering watermarks embedded in digital content. Background Technology

[0002] In a network environment such as the Internet, content providers can offer information for presentation in electronic documents, such as web pages or application interfaces. These documents may include first-party content provided by first-party content providers and third-party content provided by third-party content providers (e.g., content providers other than first-party content providers).

[0003] Various technologies can be used to add third-party content to electronic documents. For example, some documents include labels instructing client devices that render the document thereto request third-party content items directly from a third-party content provider (e.g., from a server in a different domain than the server providing the first-party content). Other documents include labels instructing client devices to invoke an intermediary service that works with multiple third-party content providers to return third-party content items selected from one or more of these providers. In some cases, third-party content items are dynamically selected for rendering in an electronic document, and the specific third-party content item selected for a given service of the document may differ from the third-party content items selected for other services of the same document. Summary of the Invention

[0004] Typically, a novel aspect of the subject matter described herein can be embodied in a method comprising jointly training an encoder machine learning model for generating a digital watermark embedded in an image and a decoder machine learning model for decoding a first data item encoded within the digital watermark embedded in the image, wherein the training comprises: obtaining a first plurality of training images and a plurality of data items, each of the plurality of data items being a data item to be encoded within a digital watermark to be embedded in the training images; for each of the first plurality of training images: obtaining a data item from the plurality of data items; generating a first digital watermark encoding the data item using the encoder machine learning model, with the data item provided as input to it; tiled two or more instances of the first digital watermark to generate a second digital watermark; and combining the second digital watermark with the training images to... The process involves: obtaining a watermarked training image; applying one or more distortions to the watermarked training image; using a distortion detector machine learning model to predict the one or more distortions present in the watermarked training image; modifying the watermarked training image based on the predicted one or more distortions while retaining the second digital watermark embedded in the distorted watermarked training image; decoding the modified watermarked training image using the decoder machine learning model to obtain decoded data items predicted to be embedded in the second digital watermark in the distorted watermarked training image; determining a first error value based on the watermarked training image and the training image; determining a second error value based on the decoded data items and the data items; and adjusting one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.

[0005] Other embodiments in this regard include corresponding methods, apparatus, and computer programs encoded on a computer storage device and configured to perform actions of the methods. These and other embodiments may each optionally include one or more of the following features.

[0006] The method may include determining the first error value based on the watermarked training image and the training image by calculating an L2 loss value based on the watermarked training image and the training image.

[0007] The method may include determining the second error value based on the decoded data item and the data item by calculating an S-shaped cross-entropy value based on the decoded data item and the data item.

[0008] The method may include image compression, Gaussian noise, or image scaling as one or more distortions.

[0009] The method may further include training the distortion detector machine learning model by: obtaining a second plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; fixing the weights of the encoder machine learning model; and for each of the second plurality of training images: using the encoder machine learning model to generate a watermarked training image; applying one or more distortions to the watermarked training image; using the distortion detector machine learning model to predict the one or more distortions present in the distorted watermarked training image; using the distortion detector machine learning model to predict the one or more distortions present in the watermarked training image; determining a third error value based on the predicted one or more distortions present in the distorted watermarked training image and the predicted one or more distortions present in the watermarked training image; and adjusting one or more training parameters of the distortion detector machine learning model to minimize the third error value.

[0010] The method may include determining the third error value by calculating an L2 loss value based on one or more predicted distortions present in the watermarked training image with the distortion and the predicted one or more distortions present in the watermarked training image.

[0011] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Visually imperceptible watermarks, also simply referred to as "watermarks" or "digital watermarks" for brevity, can be used to determine the source of third-party content presented alongside first-party content (e.g., on a website, in streaming video, or in a native application). These watermarks can be extracted and decoded in a more efficient manner than previously possible. For example, the watermark extraction and decoding techniques described in this specification implement encoder and decoder machine learning models trained together for encoding and decoding watermarks. This results in rapid watermark generation using a simple, lightweight encoder machine learning model and an efficient decoder specifically trained to decode watermarks generated by the encoder, thereby improving the overall robustness of the watermarking system.

[0012] The technique described herein includes an initial watermark detection process that detects the presence of a watermark in input digital content before attempting to decode it, which may be included in the input digital content (e.g., an image). This is motivated by considerations of the computing resources involved in decoding, which can be reduced by using a detection process (relative to the decoding process) to filter out the entire content or portions of the content that do not contain a watermark, thus saving both the time and computing resources required to process such input digital content through a more computationally expensive decoding process. In other words, instead of having to fully process the digital content and attempt to decode the watermark therein, the detection process can initially determine whether the image contains a watermark, using fewer computing resources and less time than required to perform a decoding process. In this way, using the detection process before initiating the decoding process saves computing resources, and by quickly filtering out all or part of the digital content that does not contain a watermark, it enables faster identification and analysis of digital content that actually contains a watermark, thereby reducing the computing resources otherwise required for such operations. In contrast, techniques that rely solely on the decoding process to detect and decode watermarked images, or processes that do not use a detection process as a filtering mechanism, are computationally more expensive.

[0013] The detection and decoding processes discussed in this paper are distortion-independent, meaning that watermarks can be detected and / or decoded regardless of distortion in the input image. These techniques use machine learning models to detect any distortion in the input image, which is then used to modify the input image to mitigate the distortion before decoding the watermark. This reduces any false predictions during decoding, resulting in a more robust and reliable watermarking system.

[0014] More specifically, the techniques discussed in this paper can be used to detect and decode watermarks in the reproduction of the original presented content (e.g., in an image or screenshot of the content), and the distortion of the original presented content captured will vary depending on the instance captured (e.g., varies depending on the image). Detection and / or decoding of watermarks in an input image (e.g., reproduction, such as an image of content presented at a client device) will require predicting one or more distortions only after a positive detection of the watermark. Therefore, the implementation of the disclosed methods is motivated by reducing the computational resources required to analyze images using different corresponding scaling levers to detect or decode watermarks.

[0015] Other advantages of the techniques discussed in this paper include the fact that the detection and decoding processes are data-independent of the digital content, meaning that watermarks can be encoded, detected, and decoded regardless of the data or context of the watermarked digital content. This allows for the pre-generation of watermarks and watermarked digital content, thus significantly reducing encoding time.

[0016] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0017] Figure 1 It is a block diagram of an environment used to transfer electronic documents with watermarked images to client devices.

[0018] Figure 2A is a depiction of an exemplary first watermark generated using an encoder machine learning model.

[0019] Figure 2B is a depiction of an exemplary second watermark generated using a watermark generated by an encoder machine learning model.

[0020] Figure 3A This is a block diagram of an image analysis and decoder device.

[0021] Figure 3B This is a block diagram of an exemplary convolutional neural network with the UNet architecture.

[0022] Figure 4 This is a block diagram of an exemplary process for jointly training an encoder and decoder machine learning model.

[0023] Figure 5A This is a flowchart illustrating an exemplary process for jointly training encoder and decoder machine learning models as part of an end-to-end learning pipeline.

[0024] Figure 5B This is a flowchart of an exemplary process for training a machine learning model for a distortion detector.

[0025] Figure 6 This is a flowchart of an exemplary process for adding a digital watermark to a source image.

[0026] Figure 7 This is a flowchart of an exemplary process for decoding the watermark of a watermarked image.

[0027] Figure 8 This is a block diagram of an exemplary computer system. Detailed Implementation

[0028] This specification describes systems, methods, apparatus, and techniques for detecting and decoding visually identifiable watermarks in the reproduction of captured content (e.g., digital photographs of content presented on client devices). While the following description pertains to watermark detection of visually identifiable watermarks, these techniques can also be applied to visually perceptible watermarks. For simplicity, a visually identifiable watermark, simply referred to as a "watermark," is semi-transparent and visually identifiable to human users under normal viewing conditions, allowing it to be embedded in content without degrading its visual quality. Watermarks can carry information such as an identifier of the source of the image in which the watermark is embedded. For example, in an Internet context, when a user accesses a publisher's property (e.g., a website, video stream, video game, or mobile application), the watermark can identify (among other things) the entity, server, or service that places the content on the publisher's property. Therefore, when a reproduction of content, such as that presented on the publisher's property (e.g., an image or screenshot of the content), is captured and submitted for verification, the watermark can be detected and decoded to verify whether the content was actually distributed by the appropriate entity, server, or service.

[0029] As discussed in detail below, the encoding, detection, and decoding of watermarks can be performed by a machine learning model trained to generate, detect, and decode watermarks regardless of any distortion in the captured image. To this end, the machine learning model is jointly trained so that it can detect and decode watermarks generated by the machine learning models involved during the training process.

[0030] Figure 1This is a block diagram of a computing environment 100 (or simply environment 100) for transmitting electronic documents and digital components with watermarked images to client devices. As shown, computing environment 100 includes a watermark generator 110 and an image analysis and decoder device 118. Environment 100 includes a server system 102, a client device 104, and one or more content providers 106a-n. Server system 102, client device 104, and content providers 106a-n are connected via one or more networks such as the Internet or a local area network (LAN). Typically, client device 104 is configured to generate a request for an electronic document and transmit the request to server system 102. Based on the request from client device 104, server system 102 generates a response (e.g., electronic document and digital component) to return to client device 104. A given response may include content configured to be displayed to a user of client device 104, such as source image 128a, where source image 128a is provided by one of content providers 106a-n. Server system 102 may enhance the response provided to client device 104 with a semi-transparent second watermark 126, which is arranged to be displayed on the source image 128a in the rendering of the response document at client device 104. For illustrative purposes, the following description is explained with reference to the source images 128a-n provided to client device 104; however, it should be understood that the second watermark 126 may be overlaid on various other types of visible content, including native application content, streaming video content, video game content, or other visible content. It should also be noted that instead of enhancing the response provided to client device 104 with the semi-transparent second watermark 126, server system 102 may deliver the watermark to one or more content providers 106a-n, whereby the content providers may generate the watermarked image before transmitting the content to client device 104 for rendering.

[0031] Client device 104 can be any type of computing device configured to present images and other content to one or more human users. Client device 104 may include an application, such as a web browser application, that makes requests to server system 102 and receives responses from said server system. The application may execute responses from server system 102, such as web page code or other types of document files, to present the responses to one or more users of client device 104. In some embodiments, client device 104 includes an electronic display device (e.g., an LCD or LED screen, a CRT monitor, a head-mounted virtual reality display, a head-mounted mixed reality display), or is coupled to an electronic display device that displays content from the rendered response to one or more users of client device 104. The displayed content may include a source image 128a and one or more second watermarks 126, which are displayed substantially transparently on top of the source image 128a, for example, by using a technique such as alpha blending as a process of merging two images. In some implementations, the client device 104 is a laptop, smartphone, tablet, desktop computer, game console, personal digital assistant, smart speaker (e.g., under voice control), smartwatch, or other wearable device.

[0032] In some implementations, the source image 128a provided in the response to client device 104 is, for example, a third-party content item not included in the content provided by the first-party content provider of the response. For example, if the response is a web page, the web page creator may include a slot in the web page configured to be filled with digital components (e.g., images) from a third-party content provider different from the web page creator (e.g., a provider of an image repository). In another example, the first-party content provider may directly link to the third-party source image 128a. Client device 104 may request the source image 128a directly from a corresponding computing system of one of the content providers 106a-n, or indirectly via an intermediary service such as a service provided by server system 102 or another server system. Server system 102 may be implemented as one or more computers located in one or more locations.

[0033] Server system 102 can be configured to communicate with the computing systems of content providers 106a-n, for example, to obtain source image 128a to be provided to client device 104. In such embodiments, server system 102 is configured to respond to requests from client device 104 with source image 128a and a semi-transparent watermark to be displayed on source image 128a in an electronic document. To generate the semi-transparent watermark, server system 102 may include watermark generator 110, which in turn may include encoder machine learning model 112, which includes multiple training parameters (see reference). Figure 4 (Figure 5 illustrates the training of the encoder machine learning model 112). The server system 102 can transmit the source image 128a and the semi-transparent watermark, as well as instructions for the application executed on the client device 104 to overlay the semi-transparent watermark onto the source image 128a after the semi-transparent watermark is generated.

[0034] In some implementations, server system 102 is configured to respond to a request from client device 104 with a watermarked source image 128a. In such implementations, instead of transmitting the source image 128a and the semi-transparent watermark to client device 104 (e.g., to enable client device 104 to overlay the semi-transparent watermark onto the source image 128a), server system 102 can generate a watermarked source image (also referred to as encoded image 130) by overlaying the semi-transparent watermark onto the source image 128a. After generating encoded image 130, encoded image 130 is transmitted to client device 104. In such implementations, watermark generator 110 can be implemented by server system 102 that generates the semi-transparent watermark, enabling server system 102 to access the semi-transparent watermark to generate encoded image 130 upon responding to a request from client device 104.

[0035] In another implementation, content providers 106a-n and server system 102 can communicate independently with client device 104 to transmit source image 128a and the semi-transparent watermark, respectively. In this type of implementation, content providers 106a-n and server system 102 can communicate with each other to verify simultaneous (or near-simultaneous) communication with client device 104 and the source image 128a to be watermarked at client device 104. Client device 104 then receives source image 128a and the semi-transparent watermark, along with instructions instructing an application executing on client device 104 to overlay the semi-transparent watermark onto source image 128a.

[0036] In another embodiment, content providers 106a-n can generate an encoded image 130 by overlaying a semi-transparent watermark onto the source image 128a. After generating the encoded image 130, the encoded image 130 is transmitted to the client device 104. In this type of embodiment, the watermark generator 110 can be implemented by the content providers 106a-n that generate the semi-transparent watermark, enabling the content providers 106a-106n to access the semi-transparent watermark to generate the encoded image 130 in response to a request from the client device 104.

[0037] The encoder machine learning model 112 is configured to operate during the training process (as referenced). Figure 4 (As further described in Figure 5) The encoder machine learning model 112 receives a data item (referred to as first data item 122) as input to generate a digital watermark (referred to as first watermark 124) encoded in the first data item 122. In some embodiments, the encoder machine learning model 112 may be a simple and lightweight model, such as a single fully connected convolutional neural network layer. However, it should be noted that the encoder machine learning model may include more than one convolutional, pooling, or fully connected layer. It should also be noted that the encoder machine learning model is not necessarily a neural network, but depending on the specific implementation, it can be any type of supervised, unsupervised, or reinforcement learning model.

[0038] In some implementations, the first data item 122 may be a unique identifier (which, for example, may be an alphanumeric value) that identifies a particular content provider 106. The first data item 122 may additionally or alternatively include a session identifier (which, for example, may be an alphanumeric value) that uniquely identifies a network session between the client device 104 and the server system 102 during which a response is provided to a request from the client device 104. The first data item 122 may include a reference to or information associated with a particular source image 128a provided to the client device 104 (e.g., information indicating which of the content providers 106a-n provided the particular source image 128a to the client device 104 and a timestamp indicating when the source image 128a was provided or requested).

[0039] In some implementations, server system 102 may further include a response record database that stores data relating to such information about source image 128a or responses provided to a specific request, enabling access to detailed information via a session identifier or other information represented by a first data item. The response record database may also associate session identifiers with image data, thereby making the image data accessible by querying the database using the session identifier represented by the first data item. A user of the server system can then use the session identifier of the first data item to identify, for example, which of the source images 128a-n was provided to client device 104 at what time and from which content provider 106a-n.

[0040] In some embodiments, the first watermark 124 is an image representing the first data item 122. The first watermark 124 can be a matrix barcode or any pattern that can encode the first data item 122. The first watermark 124 can have a predefined size in terms of the number of rows and columns of pixels. Each pixel in the first watermark 124 can encode multiple data bits, where the values ​​of the multiple bits are represented by different colors. For example, a pixel encoding the binary value '00' can be black, while a pixel encoding the binary value '11' can be white. Similarly, a pixel encoding the binary value '01' can be a lighter shade of black (e.g., dark gray), while a pixel encoding the binary value '10' can be a lighter shade of black (e.g., light gray). In some embodiments, the smallest encoding unit of the first watermark can actually be larger than a single pixel. However, for the purposes of the examples described herein, it is assumed that the smallest encoding unit is a single pixel. However, it should be appreciated that the techniques described herein can be extended to embodiments where the smallest encoding unit is a set of multiple pixels (e.g., a 2×2 or 3×3 pixel set). Referring to Figure 2A, an exemplary first watermark 124 generated by encoder machine learning model 112 using first data item 122 is depicted and described.

[0041] Figure 2A depicts an exemplary watermark pattern 200 that can be used as a first watermark 124 for the purposes of the techniques described herein. In some embodiments, the watermark 200 has a fixed size, for example, 32 × 64 pixels in this example, but watermarks with other predefined sizes can also be used. A watermark pattern 200 can be generated using the first data item 122 as shown in the reference. Figure 1 Description (and as referenced) Figure 6(Further description) Watermark 200. A distinguishing feature of watermark pattern 200 is that each pixel or group of pixels can be a different color and a corresponding shade of that color. For example, watermark pattern 200 may include white or black pixels or groups of pixels, where different pixels or groups of pixels can have different shades of white or black. This feature allows for the provision of a greater number of unique patterns using a set number of pixels or watermark size (relative to other watermark patterns, such as QR codes).

[0042] Continue to refer to Figure 1 In the discussion, after generating the first watermark 124, the server system 102 uses the watermarking device 114 to combine multiple instances of the first watermark 124 to generate a second watermark 126. For example, the watermarking device 114 can generate the second watermark 126 by placing two or more instances of the first watermark 124 side by side. The exemplary second watermark 126 is further explained with reference to FIG2B.

[0043] Figure 2B depicts an exemplary watermark 250 that can be used as a second watermark for the purposes of the techniques described in this specification. Watermark 250 has a size of 64 × 128 pixels and is generated by watermarking device 114 by placing four first watermarks adjacent to each other. For example, watermark 250 includes four instances (255 to 258) of first watermark 124.

[0044] Return to Figure 1In some implementations, server system 102 generates a response to be returned to client device 104 as a reply to the client's request for an electronic document. The response may include one or more content items, including first-party and third-party content items, which together form an electronic document such as a webpage, application interface, PDF, presentation slide set, or spreadsheet. In some implementations, the response includes a main document specifying how the various content items are arranged and displayed. The main document, such as a Hypertext Markup Language (HTML) page, can refer to the first-party and third-party content items to be displayed in the presentation of the document. In some implementations, server system 102 is configured to add computer code to the main document that instructs client device 104, when executing the response, to display one or more instances of a second watermark 126 on the source image 128a, for example, to add a watermark that is substantially recognizable to a human user to the source image 128a. The application of rendering an electronic document at client device 104 can use alpha blending technology to overlay a second watermark 126 onto a source image 128a with a specified transparency, which specifies the opacity level of the second watermark 126 when overlaid on the source image 128a. For example, server system 102 can add code that instructs client device 104 to display the source image 128a as a background image in a third-party content slot within the electronic document and to display one or more instances of the second watermark 126 as a foreground image on image 128a. In some implementations, when server system 102 is configured to respond to a request from client device 104 with a watermarked image 130, server 102 performs the alpha blending technology to overlay the second watermark 126 onto the source image 128a. Similarly, if any other entity (e.g., content provider 106) is configured to respond to a request from client device 104 with a watermarked image 130, that entity performs the alpha blending technique to overlay the second watermark 126 onto the source image 128a.

[0045] In some implementations, an entity such as client device 104 that generates the watermarked image 130 applies an S-shaped function to each pixel intensity value of the second watermark 126 to constrain the intensity values ​​to [0, 1] before overlaying the second watermark onto the source image 128a. This can be expressed by the following formula:

[0046] I m =sigmoid(W e M0+b e )

[0047] Where I m It's the second watermark 126, M0 is the data item, W e and b eThese are the weights and biases of encoder machine learning model 112.

[0048] In some implementations, if the second watermark 126 is larger than the size of the source image 128a, the second watermark 126 is cropped based on predefined rules. For example, a predefined rule could state that the second watermark 126 can be cropped from the bottom right to adjust the size of the second watermark 128 to match the size of the source image 128a.

[0049] In some implementations, to reduce the file size of the second watermark 126, it can also be based on a constant color vector c∈R 3 Adjust each pixel of the second watermark 126 to obtain the adjusted watermark I. m This can be represented as ′。

[0050] I' m =Repeat(I m .c)

[0051] In some implementations, the alpha mixing technique that overlays the second watermark 126 onto the source image 128a according to a specified transparency can be represented as:

[0052] I w = (1-α)*I o +α*I′ m

[0053] Where I w It is the encoded image 130, and α is a measure of the opacity of the second watermark 126 when it is superimposed on the source image 128a.

[0054] In an environment where millions of images (and other visual content) may be distributed to many different client devices 104, there may be situations where the server system 102 needs to determine the provider or source of an image (or other visual content), other characteristics of the image (or other visual content), or the context of a particular impression (e.g., presentation) of the image (or other visual content).

[0055] For example, in response to a request for an electronic document, a user of client device 104 may receive an inappropriate or irrelevant image 128a from one of content providers 106a-n. The user may capture a screenshot of the encoded image 130 (e.g., a reproduction of an image or other content presented at client device 104) and transmit the screenshot to server system 102 for analysis, such as querying the source of the original image 128a. Because the screenshot shows the original image 128a overlaid with the watermarked image 126, server system 102 can process the screenshot to recover a first data item from the digital watermark included in the image. System 102 can then use the recovered first data item for various purposes, such as querying a response record database to find detailed information about image 128a and its source, or other information about a specific client session in which the source image 128a was provided to client device 104.

[0056] In some embodiments, server system 102 may include image analysis and decoder device 118 to detect and decode the encoded representation of first data item 122 from encoded source image 130. As described above, in some embodiments, encoded source image 130 is an image obtained by client device 104 rendering second watermark 126 onto source image 128a. Even if second watermark 126 is separated from source image 128a, encoded source image 130 processed by image analysis and decoder device 118 may also be a merged image showing the second watermark 126 mixed on source image 128a. Encoded source image 130 may be input to image analysis and decoder device 118, which detects and / or decodes the watermark present in encoded source image 130. The encoded source image 130 input to the image analysis and decoder device 118 may be the actual encoded source image provided at the client device 104, or it may be a reproduction (e.g., a screenshot or other digital capture) of the image (as described above, the image is generated by merging / blending the second watermark 126 with the source image 128a). Therefore, the original source image 128a and the original second watermark 126 may not be submitted to the image analysis and decoder device 118 for analysis.

[0057] In some cases, server system 102, including image analysis and decoder device 118, can receive requests to analyze images that may be encoded / watermarked. As used herein, the term "may" refers to a condition of an item that may be attributed to the item, but which remains unknown to the processing entity processing the item (e.g., server system 102). That is, a possible condition of an item is a candidate condition of an item whose authenticity is unknown to the processing entity. The processing entity can perform processing to identify possible (candidate) conditions of an item, to predict the authenticity of possible (candidate) conditions, and / or to identify possible (candidate) items that present a particular condition. For example, a possible encoded source image is a source image that may be watermarked, but server system 102 initially does not know whether the image has actually been watermarked. Therefore, a potentially watermarked encoded source image 130 is a candidate condition of the encoded source image 130, and the encoded source image 130 is a candidate for presenting a watermarked encoded candidate condition. It is possible that the encoded image is obtained by the user capturing a screenshot of the image (or another digital reproduction, such as a digital photograph) and providing the captured image to the server system 102 for analysis, but there is no further information indicating whether the image has been encoded / watermarked.

[0058] In these cases where server system 102 receives a request to analyze a potentially encoded (watermarked) source image, image analysis and decoder device 118 uses watermark and distortion detection device 132 to analyze the received image. Watermark and distortion detection device 132 can implement one or more machine learning models, such as a watermark detector machine learning model 134a for detecting whether a potentially encoded source image might contain a watermark, and a distortion detector machine learning model 134b for detecting potential distortions in the potentially encoded source image when compared to the encoded source image 130 provided to client device 104. (See reference...) Figure 3A Each of these machine learning models will be further described. For the sake of brevity, the source image may also be referred to as the encoded image.

[0059] If the watermark and distortion detection device 132 detects a visually discernible watermark and one or more distortions in a portion of a potentially encoded source image, the image analysis and decoder device 118 can modify that portion of the potentially encoded source image to remove any distortions. After distortion removal, the watermark decoder 134, implemented within the image analysis and decoder device 118, attempts to decode the portion / region of the potentially encoded image in which the digital watermark was detected. As explained in further detail with reference to other figures, the watermark decoder 134 can implement one or more machine learning models (referred to as decoder machine learning models) configured to process the potentially encoded regions and features of the potentially encoded image to predict the watermark state of the potentially encoded image. The image analysis and decoder device 118 may also include a scaling device 138 and a verification device 140, which will be discussed in more detail below. The image analysis and decoder device 118 and any subsystems can be implemented on one or more computers in one or more locations where the server system 102 is implemented.

[0060] The watermark generator 110, watermark and distortion detection device 132, and watermark decoder 134 can be implemented by a single entity or different entities. For example, client device 104 may include watermark and distortion detection device 132, such that client device 102 can detect the presence of watermarks and / or distortions in the captured possible encoded image before generating and transmitting a request for analysis of the possible encoded image. In another example, client device 104 may include both watermark and distortion detection device 132 and watermark decoder 134, such that client device 104 can detect and decode watermarks present in the possible encoded image. In another example, watermark generator 110 may be implemented by content providers 106a-n, such that content providers 106a-n can generate encoded image 130 in response to a request from client device 104.

[0061] Figure 3A This is a block diagram 300 of an exemplary image analysis and decoder device 118, which detects and decodes a possible encoded image 302 provided as input to the image analysis and decoder device 118 to obtain a first data item of prediction encoded within a digital watermark included in the possible encoded image 302.

[0062] The encoded image 302 may take the form of a screen capture or digital photograph of an image presented at a client device. For example, the encoded image 302 may be a screen capture of an image presented on a publisher's website. More specifically, the encoded image 302 may have been captured by a user visiting the publisher's website and then submitted by the user to report the presentation of the image (e.g., as inappropriate). The image analysis and decoder device 118 may include one or more of a watermark and distortion detection device 132, a watermark decoder 134, and a verification device 140.

[0063] In some embodiments, the watermark and distortion detection device 132 may implement a watermark detector machine learning model 132a, configured to process a potentially encoded image 302 and generate an indication as output whether a portion of the potentially encoded image 302 includes one or more watermarks. The watermark detector machine learning model 132a can be any model considered suitable for a particular embodiment, such as decision trees, artificial neural networks, genetic programming, logic programming, support vector machines, clustering, reinforcement learning, Bayesian inference, etc. The machine learning model may also include computer vision and image processing methods, algorithms, and techniques for analyzing images. In such embodiments, the indication of whether the potentially encoded image 302 includes a portion of a watermark or one or more watermarks can take the form of classification or numbers such as scores or probabilities. For example, the watermark detector machine learning model 132a can be implemented as a classification model that processes the potentially encoded image 302 to classify the image as an image including a watermark or an image not including a watermark. In another example, the watermark detector machine learning model 132a can process the potentially encoded image 302 to generate scores, such as scores indicating the likelihood that the potentially encoded image 302 includes a watermark.

[0064] In some implementations, the watermark and distortion detection device 132 may implement a watermark detector machine learning model 132a to perform semantic image segmentation and generate a segmentation mask that identifies the set of coded pixels that have been watermarked. Semantic image segmentation is the process of classifying each pixel of an image into one or more classes. For example, the watermark detector machine learning model 132a may process a potentially coded image 302 to classify each pixel of the potentially coded image 302 into multiple classes (e.g., a first class and a second class). In the implementation of classifying each pixel into a first class and a second class, the first class corresponds to pixels of image 302 that have been mixed with the second watermark 126, and the second class corresponds to pixels of image 302 that have not been mixed with the second watermark 126. The watermark detector machine learning model 132a classifies pixels based on pixel characteristics of the potentially coded image 302. For example, pixels classified as the first class (i.e., encoded with the second watermark) may be distinguishable to the watermark detector machine learning model 132a even if they are visually indistinguishable to the human eye. For example, a 32-bit RGB pixel includes 8 bits for each color channel (e.g., red (R), green (G), and blue (B)) and an "alpha" channel for transparency. This format can support 4,294,967,296 color combinations that can be recognized by a computing system, even if some of these combinations are indistinguishable to the human eye.

[0065] Based on the classified pixels, the watermark detector machine learning model 132a generates a segmentation mask as output. The segmentation mask identifies the set of encoded pixels that have been watermarked (e.g., a set of pixels classified into a first class corresponding to pixels that include a portion of the watermark / encode a portion of the watermark). For example, after classifying the pixels of the potentially encoded image 302 into a first class and a second class, the watermark detector machine learning model 132a can generate a segmentation mask by assigning labels to pixels belonging to the class to which the pixels are assigned. For example, the watermark detector machine learning model 132a receives a potentially encoded image 302 of size 1000×1000×3 (e.g., a screenshot from client device 104) as input, where size refers to the length, width, and number of channels of the potentially encoded source image 302. The watermark detector machine learning model 132a generates a segmentation mask of size 1000×1000×1 as output, where each value of the segmentation mask corresponds to a label assigned to the corresponding pixel of the potentially encoded image 302. For example, if a pixel of the potentially encoded image 302 is classified as a first class, it can be assigned the label "1", and if a pixel is classified as a second class, it can be assigned the label "0". In this example, by processing the potentially encoded image 302, a segmentation mask 310 is generated by the watermark detector machine learning model 132a. Figure 3AAs shown, the segmentation mask 310 includes two parts 310a and 310b containing pixels classified as the first class, and a third part 310c containing pixels classified as the second class. Figure 3A As shown, the possible encoded image 302 includes two watermarks 126a and 126b in two different regions. Using the possible encoded image 302 as input, the watermark detector machine learning model 132a outputs a segmentation mask 310, which identifies the portions of the possible encoded image 302 that include watermarks 126a and 126b. After the watermarks are detected, the possible encoded image 302 can be processed by the watermark decoder 134, as discussed in detail below.

[0066] In another example, the watermark detector machine learning model 132a can generate a segmentation mask for each class of the watermark detector machine learning model 132a. For example, the watermark detector machine learning model 132a can generate a segmentation mask of size 1000×1000×NumClass, where NumClass = 2 is the number of classes of the watermark detector machine learning model 132a. In this example, the segmentation mask can be interpreted as two 1000×1000 matrices, where the first matrix identifies pixels of the possible encoded image 302 belonging to the first class, and the second matrix identifies pixels of the possible encoded image 302 belonging to the second class. In this case, labels “0” and “1” are used to indicate whether a pixel belongs to a specific class. For example, the values ​​of the first matrix whose corresponding pixels of the possible encoded image 302 are classified as belonging to the first class have the label “1”, and the elements whose corresponding pixels are classified as belonging to the second class have the label “0”. Similarly, the values ​​of the elements of the second matrix whose corresponding pixels of the possible encoded image 302 are classified as belonging to the second class have the label “1”, and the elements whose corresponding pixels are classified as belonging to the first class have the label “0”. refer to Figure 3B Further explanation of the deep convolutional neural network (CNN) with UNet architecture, which can be used as a machine learning model for watermark detectors 132a.

[0067] Figure 3BThis is a block diagram of the example architecture watermark detector machine learning model 350. The watermark detector machine learning model 350 is a CNN with a UNet architecture. The watermark detector machine learning model 350 includes encoder blocks 360, 365, and 370, and decoder blocks 375, 380, and 385. It should be noted that encoder blocks 360, 365, and 370, and decoder blocks 375, 380, and 385, are distinct from the encoder and decoder machine learning models. The encoder blocks 360, 365, and 370 of CNN 350 include convolutional layers, followed by one or more max-pooling layers. For example, the encoder blocks may include convolutional layers performing 3×3 convolutions, followed by max-pooling layers performing 2×2 max-pooling operations. In some implementations, the encoder blocks may be a pre-trained classification network, such as a VGG network. Decoder blocks 375, 380, and 385 may include convolutional layers, followed by upsampling layers. For example, a decoder block may include convolutional layers that perform 3×3 convolutions, followed by upsampling layers, after which the input of each block is appended with feature maps from the encoder block.

[0068] CNN 350 is configured to receive an image, such as a possible source image 302, as input and generate a segmentation mask as output, which is trained on CNN 350 to classify different image segments. For example, CNN 350 generates a segmentation mask 390 of size 1000×1000×1 as output, where each value of the segmentation mask corresponds to a label assigned to the corresponding pixel of the possible source image 302. For example, if a pixel of the possible source image 302 is classified as a first class, it can be assigned the label "1", and if a pixel is classified as a second class, it can be assigned the label "0". Figure 3A As shown, the segmentation mask 310 includes two parts 310a and 310b containing pixels classified as the first class, and a third part 310c containing pixels classified as the second class.

[0069] In some implementations, a training process is used to train a watermark detector machine learning model 132a on a training dataset (referred to as the detector model training dataset). This training process can adjust multiple training parameters to generate an indication of whether a potentially encoded image 302 includes one or more watermarks. The detector model training dataset may include multiple training samples, each of which includes a watermarked training image and a target that identifies pixels in the watermark-encoded training image. For example, the training image may be an image similar to a screenshot from a client device 104, which includes a watermark in one or more regions of the training image. The target corresponding to the training image may include a segmentation mask that identifies either watermarked or unwatermarked pixels, or in some cases, both watermarked and unwatermarked pixels.

[0070] To enhance the generalization potential of the watermark detector machine learning model 132a, the training process can use distortion devices to augment the detector model training dataset. These distortion devices can generate new distorted training samples, for example, using existing training samples from the detector model training set. To generate new training samples, the training process can distort images in the training image set to create distorted images. In some implementations, distorted images can be generated by applying visual perturbations widely found in real-world visual data, such as horizontal and vertical flipping, translation, rotation, cropping, scaling, color distortion, adding random noise, horizontal and vertical scaling, and stitching images with other background images. The training process can also generate new training samples by encoding the training images into different file formats using lossy compression or transformation techniques. For example, the training process can use JPEG compression to introduce small artifacts into the training images, and the training images generated after compression can be used to augment the detector model training dataset.

[0071] During training, the training process can use loss functions such as cross-entropy loss to adjust various parameters of the watermark detector machine learning model 132a. For example, pixel-wise cross-entropy loss can examine each pixel individually to compare the class prediction with the target class of the pixel and adjust the parameters of the watermark detector machine learning model 132a accordingly. The training process can be iterative in nature, such that during each iteration, the training process aims to minimize the cross-entropy loss, for example, until the loss is less than a specified threshold or until the training process has performed a specified number of iterations. The cross-entropy loss can take the following form:

[0072] L=-(y log log(p)+(1-y)log(1-p))

[0073] Where y is the target label of the pixel, and p is the predicted probability that the pixel belongs to the first class. Examples of other loss functions may include weighted cross-entropy loss, focal loss, sensitivity-specific loss, dice loss, boundary loss, Hausdorff distance loss, or a composite loss that can be calculated as the average of two or more different types of losses.

[0074] In some implementations, the watermarking and distortion detection device 132 may implement a distortion detector machine learning model 132b, which may be configured to process the potentially encoded image 302 to generate an indication as output of one or more distortions that the potentially encoded image 302 has undergone relative to the source image 128a. For example, by processing the potentially encoded image 302, the distortion detector machine learning model 132b may generate an indication as output of vertical scaling, horizontal scaling, and image offset. Vertical and horizontal scaling are distortions that respectively indicate changes in the length and width of the potentially encoded image 302 relative to the source image 128a. Other types of distortion, such as scaling, may be generated from the predicted horizontal and vertical scaling.

[0075] In some embodiments, the watermark and distortion detection device 132 may implement a distortion machine learning model 132b, which may be configured to process only the portion of the possibly encoded image 302 that includes one or more watermarks, to generate an indication as output of one or more distortions that the portion of the possibly encoded image 302 has undergone relative to a corresponding portion of the source image 128a. For example, by processing the portion of the possibly encoded image 302, the distortion detector machine learning model 132b may also generate an indication as output of vertical and horizontal scaling, where vertical and horizontal scaling are distortions that respectively indicate changes in the length and width of the portion of the possibly encoded image 302 relative to a portion of the source image 128a.

[0076] In some implementations, the distortion detector machine learning model 132b may be a CNN with a UNet architecture, trained to process portions of the image 302 that may encode them, to generate an indication of one or more distortions of those portions as output. The distortion machine learning model 132b is trained on a training dataset (referred to as the distortion model training dataset) using a training process that may adjust multiple training parameters to generate indications of one or more distortions of those portions of the image 302. The distortion model training dataset may include multiple training samples, each including a watermarked training image. For example, the watermarked training image may be an image similar to the watermarked image 130 generated by overlaying a second watermark 126 onto the source image 128a.

[0077] In some implementations, the distortion detector machine learning model 132b can be trained to detect distortion in the potentially encoded image 302 or distortion in a portion of the potentially encoded image 302 specifically encoded by the trained encoder machine learning model 112. In other words, the distortion detector machine learning model 132b is fine-tuned to detect distortion in images encoded using a specific encoder machine learning model 112. In such implementations, the detector model training dataset may include unwatermarked training images. After training the encoder machine learning model 112, the parameters of the encoder machine learning module 112 are fixed and then used to watermark each training image in the detector model training dataset to generate a corresponding watermarked training image.

[0078] In some implementations, when training the distortion machine learning model 132b, the training process can distort watermarked training images from the distortion model training dataset to generate distorted watermarked training images. For example, during each iteration of the training process, the watermarked training images from the distortion model training dataset can be distorted based on a random horizontal scaling factor, a random vertical scaling factor, and a random image offset. The training process then feeds the distorted watermarked training images and the watermarked training images as input to the distortion detector machine learning model 132b to generate one or more outputs indicating one or more distortions in the distorted watermarked training images and the watermarked training images. For example, after generating distorted watermarked training images from the watermarked training images, the training process can feed the watermarked training images as input to the distortion detector machine learning model 132b to generate a pattern (called a general pattern). Similarly, the training process can feed the distorted watermarked training images as input to the distortion detector machine learning model 132b and generate another pattern (called a transformed pattern) as output.

[0079] In some implementations, the general pattern and the transformed pattern can be a grid pattern generated using a pair of periodic signals, which further generate a pair of horizontal and vertical lines on both the watermarked training image and the distorted watermarked training image. In such implementations, when the second watermark 126 is superimposed on the source image 128a, the peak of the signal corresponds to the x and y coordinates of the center of the second watermark 126.

[0080] After generating the general pattern and the transformed pattern, the training process compares the two patterns to calculate a third error value using a loss function (e.g., L2 loss). It should be noted that the third error value is a predicted measurement of the distortion added to the watermarked training image. The third error value can sometimes be expressed in the form ||T(U0)-U1|| 2Here, T refers to the transformation applied to the watermarked training image by adding one or more distortions, U0 is the general pattern, and U1 is the transformation pattern. The training process can then use a third error value to adjust various parameters of the distortion detector machine learning model 132b. The training process can be iterative in nature, such that during each iteration, the training process aims to minimize the L2 loss, e.g., until the loss is less than a specified threshold or until the training process has performed a specified number of iterations.

[0081] In some implementations, the watermark detector machine learning model 132a and the distortion machine learning model 132b can be implemented as a single machine learning model. In one such exemplary implementation, the single machine learning model can process information in two phases, such that during the first phase, the single machine learning module can process the potentially encoded image 302 to determine if a portion of the potentially encoded image 302 includes one or more watermarks. For example, by processing the potentially encoded image 302 and generating a corresponding segmentation mask that identifies the portions of the potentially encoded image 302 that include one or more watermarks. During the second phase, the single machine learning model can process the portions of the potentially encoded image 302 that include one or more watermarks to generate an indication of the distortion that these portions of the potentially encoded image 302 have undergone.

[0082] In another exemplary embodiment where the watermark detector machine learning model 132a and the distortion machine learning model 132b can be implemented as a single machine learning model, the single machine learning model can be configured to process the possibly encoded image 302 and generate three outputs, wherein the first output is a segmentation mask that identifies a portion of the possibly encoded image 302, the second output is a predicted vertical scaling, and the third output is a predicted horizontal scaling.

[0083] In some implementations, the image analysis and decoder device 118 may generate a scaled version of the possibly encoded image 302 in response to the watermark detection device 132 failing to detect and / or extract the entire watermarked region of the possibly encoded image 302. For example, suppose the segmentation mask only generates a portion of the watermarked region. In this case, the watermarked decoder 134 will be unable to decode the watermark due to incomplete information. In this case, the image analysis and decoder device 118 may generate a scaled version of the possibly encoded image 302 and check whether the entire watermarked region of the possibly encoded image 302 can be identified before decoding.

[0084] In some implementations, the watermark and distortion detection device 132 can process a portion of the potentially encoded image 302 after the detection machine learning model 132a has successfully determined the presence of a watermark in the potentially encoded image 302.

[0085] In some implementations, after detecting and determining that a portion of the potentially encoded image 302 includes one or more watermarks, the image analysis and decoder device 118 can modify that portion of the image based on distortion predicted by a distortion detector machine learning model to generate a modified portion of the potentially encoded image 302 that is similar to or nearly similar to the source image 130. For example, after determining the presence of a watermark on the potentially encoded image 302 using a watermark detector machine learning model 132a, the image analysis and decoder device 118 can obtain a portion of the potentially encoded image 302 that includes one or more watermarks. In response to a definitive determination of the presence of one or more watermarks, the image analysis and decoder device can also generate one or more predictions indicating different distortions experienced by the potentially encoded image 302. For example, suppose the distortion detector machine learning model 132b predicts that the potentially encoded image 302 has undergone a 2x vertical scaling. In response to this prediction, the image analysis and decoder device 118 can modify a portion of the potentially encoded image to generate a modified version with a 1 / 2 vertical scaling factor, thereby mitigating any distortion experienced by the potentially encoded image 302.

[0086] Similarly, if the distortion detector machine learning model 132b predicts that the potentially encoded image 302 has undergone vertical and / or horizontal scaling (identified using vertical and horizontal scaling factors), the image analysis and decoder device 118 can modify portions of the potentially encoded image 302 by scaling those portions to generate a modified version. The modified version is scaled by the same vertical and / or horizontal scaling factors, thereby mitigating any vertical and / or horizontal distortion experienced by the potentially encoded image 302.

[0087] To decode a watermark detected in a potentially encoded image 302, the image analysis and decoder apparatus includes a watermark decoder 134. In some embodiments, the watermark decoder 134 may implement a decoder machine learning model 134a, configured to process modified portions of the potentially encoded image 302 and generate a predicted first data item as output. The decoder machine learning model 134a can be any model considered suitable for a particular embodiment, such as decision trees, artificial neural networks, genetic programming, logic programming, support vector machines, clustering, reinforcement learning, Bayesian inference, etc. The machine learning model may also include computer vision and image processing methods, algorithms, and techniques for analyzing images. In some embodiments, the decoder machine learning model 134a may be a deep convolutional neural network (CNN) with a UNet architecture, trained to predict the predicted first data item. The decoder machine learning model 134a may include multiple training parameters that can be tuned to generate a prediction (e.g., the predicted first data item).

[0088] In some implementations, after generating a predicted first data item by processing the potentially encoded image 302, the image analysis and decoder device 118 can use the predicted first data item to verify the authenticity (or source) of the potentially encoded image 302. To verify authenticity (or source), a verification device 140 implemented within the server system 102 can compare the predicted first data item with a first data item stored in the response record database 120. If a match is found (e.g., an exact match), the verification device 140 can conclude that the source image 128a presented on the client device 104 was actually provided by the server system 102 or content providers 106a-b. If no match is found, the verification device 140 can conclude that the source image 128a presented on the client device 104 was not provided by the server system 102 or content providers 106a-b.

[0089] Figure 4 This is a block diagram of training process 400, as part of an end-to-end learning pipeline. Training process 400 jointly trains an encoder machine learning model to generate a digital watermark to be included in a digital component and a decoder machine learning model to decode the digital watermark of the digital component to obtain the data item encoded in the digital watermark. The operation of the training process is schematically controlled by a system (e.g., Figure 1 The system 400 is executed by an encoder machine learning model 112 and a decoder machine learning model 134a. The operation of process 400 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of process 400. The training process 400 is an iterative process, wherein each iteration of process 400 is interpreted using steps A through E. The training process terminates after reaching the termination criteria described below.

[0090] In some implementations, a training process is used to train encoder and decoder machine learning models on a training dataset (referred to as an end-to-end training dataset). This training process can adjust multiple training parameters of the encoder and decoder machine learning models by processing the watermarked digital component (e.g., possibly encoding image 302) to generate a predicted first data item, where the watermark overlaid on the watermarked image is encoded using the first data item. In other words, the joint training process aims to enable the encoder machine learning model to encode the first data item into a digital watermark pattern, then overlay it onto the digital component, and enable the decoder machine learning module to decode the watermarked digital component to output the same predicted first data item as the first data item.

[0091] The end-to-end training dataset may include multiple training images (or other types of digital components) and multiple first data items. For example, the training images may be images similar to source images 128a-n of third-party content provided to client device 104, and the first data items may be first data items processed by encoder machine learning model 112 to generate a second watermark for watermarking the training images.

[0092] During training, the encoder machine learning model encodes each of the first data items from multiple data items into a digital watermark, and then the digital watermark is superimposed onto a specific training image (from multiple training images) to obtain the corresponding watermarked training image. Figure 4 The purpose of this training is to generate a first data item (also referred to as the watermarked image) and a first error value (referred to as Loss1 425). These watermarked images are processed by the decoder machine learning model to generate a first data item for the prediction of each corresponding watermarked image and a corresponding second error value (referred to as Loss2 460). It should be noted that each corresponding watermarked image has a corresponding first data item for generating the watermark of the corresponding watermarked image. Depending on Loss1 425 and Loss2 460, the learnable parameters of the encoder and decoder machine learning models are adjusted. The training process 400 is further described below, and for simplicity and ease of explanation, the training process 400 is explained with reference to the training image 420 and the first data item 410 for generating the digital watermark.

[0093] During step A of a specific iteration of the training process 400, the encoder machine learning model 112 processes the first data item 410 to generate the first watermark. Although Figure 4 Not shown, but the watermarking device 114 can use the first watermark to generate a second watermark, the second watermark being, for example, a tiled version of the first watermark (as shown and described with reference to Figures 2A and 2B). In some embodiments, the second watermark may undergo additional processing, such as, for example, cropping the second watermark to have the same size as the training image 420 (i.e., the same dimensions). The second watermark may also undergo processes such as those described in reference... Figure 3A The processing of pixel intensity and transparency adjustments discussed. Once the second watermark is completed, the watermarked training image 430 is generated by overlaying the second watermark onto the training image 420 (e.g., using alpha mixing as described above).

[0094] During step B, Loss1 425 is calculated based on training image 420 and watermarked training image 430. Loss1 425 indicates the difference between training image 420 and watermarked training image 430. For example, a per-pixel loss function such as an absolute error function can be used to calculate the difference between images 420 and 430 at the pixel level. Other error functions may include perceptual loss functions such as mean squared error (L2).

[0095] During step C, distortion device 440 (e.g., the distortion device described with reference to FIG. 3) can process the watermarked training image 430 to generate one or more distorted images 450. The distorted images 450 are generated by adding one or more distortions such as vertical and horizontal scaling, cropping, to simulate real-world image changes that the encoded image 302 might undergo. For example, distortion device 440 can distort the watermarked training image 430 by applying random horizontal and vertical distortion factors.

[0096] Although not used as an example, it should be noted that the distortion device 440 can generate multiple different distorted versions of the same image, which can be used to decode the watermark in the distorted version of the image, thus improving the versatility of the decoder machine learning model 134a. For example, given a specific watermarked training image 430, the distortion device 440 can generate multiple different versions of the distorted image 450, which the decoder machine learning model 134a can later use to improve its versatility across different types of distortion.

[0097] During step D, a portion of the distorted image 450 is provided as input to the decoder machine learning model 134a. In some implementations, although Figure 4 However, before providing the distorted image 450 to the decoder machine learning model 134a, the distorted image 450 can be processed using a watermarking and distortion detection device 132, as shown in the reference. Figure 3A As explained. In such embodiments, the watermark and distortion detection device 132 can process the distorted image 450 to identify a portion of the distorted image 450 including a watermark (as referenced). Figure 3A (As described). In some embodiments, the identified portion of the distorted image 450 may be further processed to generate a modified portion of the distorted image 450, wherein the processing may include mitigating any distortion experienced by the watermarked training image 430. The decoder machine learning model 134a processes the identified portion or the modified portion of the distorted image 450 to generate a first data item 460 of predictions to be included in the image.

[0098] During step E, a second error value (referred to as Loss2 460) is calculated based on the predicted first data item 460 and the target first data item 410. The second error value indicates the difference between the predicted and actual values ​​of the first data item used for watermarking the image. For example, Loss2 could be an sigmoid cross-entropy loss.

[0099] After calculating Loss1 425 and Loss2 470, the learnable parameters of the encoder machine learning model 112 and the decoder machine learning model 134a can be adjusted to minimize the total loss (i.e., loss1 + loss2) or the individual values ​​of Loss1 and Loss2. The total loss can be expressed as follows:

[0100] Total loss = ||I w -I o || 2 + Cross-entropy (M) d M o )

[0101] Where I w It is the watermarked training image 430, I o The training image is 420, M d The first predicted data item is 460, and M o The target first data item is 410. That is, the magnitude of the loss value indicates how far the prediction is from the actual value (e.g., the difference between the predicted first data item 460 and the target first data item 410), and the sign of the loss value indicates the direction in which the learnable parameters must be adjusted. It should be noted that Loss1 425 and Loss2 470 can be viewed as two competing objectives. For example, the objective of Loss1 425 is to change the training image as little as possible, while the objective of Loss2 470 is to make the decoding as accurate as possible. The same training image balancing the two loss functions is used to train both the encoder machine learning model and the decoder machine learning model.

[0102] As previously described, training process 400 is an iterative process that iterates over training samples in an end-to-end training dataset. Training process 400 terminates when a termination criterion is met. For example, training process 400 may terminate when the loss value calculated during steps B and E falls below a specified threshold. For instance, if the specified threshold for the total error is set at 0.1, the training process will continue iterating over the training images until the value of loss1 + loss2 > 0.1. In another example, training process 400 may terminate after a specified number of iterations (e.g., 10,000 iterations).

[0103] Figure 5AThis is a flowchart illustrating an exemplary process for jointly training an encoder machine learning model and a decoder machine learning model as part of an end-to-end learning pipeline. The operation of process 500 can be implemented, for example, by a server system 102 including image analysis and decoder devices 118. The operation of process 500 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of process 500.

[0104] Training process 500 iterates over the training samples in the end-to-end training dataset. Training process 500 terminates when a termination criterion is met. For example, training process 500 may terminate when the total loss falls below a specified threshold. For instance, if the specified threshold for the total loss is set at 0.1, the training process will continue iterating over the training images until the total loss value is <= 0.1. In another example, training process 500 may terminate after a specified number of iterations (e.g., 10,000 iterations).

[0105] Server system 102 obtains multiple training images and multiple data items (505). For example, an end-to-end training dataset can be used to train encoder and decoder machine learning models. The end-to-end training dataset may include multiple training images and multiple first data items. For example, the training images may be images similar to source images 128a-n of third-party content provided to client device 104, and the first data items may be first data items processed by encoder machine learning model 112 to generate a second watermark for watermarking the training images.

[0106] Server system 102 uses an encoder machine learning model to generate a first digital watermark (510). For example, an encoder machine learning model 112 implemented in the watermark generator 110 of server system 102 encodes a first data item 410 to generate the first watermark (as shown and described with reference to FIG2A).

[0107] Server system 102 uses a tile-based device to generate a second digital watermark (515). For example, after generating a first watermark, server system 102 uses watermark tile-based device 114 to combine multiple instances of the first watermark to generate a second watermark, for example, a tile-based version of the first watermark (as shown and described with reference to FIG. 2B). The second watermark may also undergo processes such as cropping (as discussed with reference to FIG. 2B and 3A).

[0108] Server system 102 combines the second digital watermark with the training image to obtain a watermarked image (520). See reference... Figure 4As described, server system 102 can use, for example, alpha mixing techniques to combine the second watermark and training image 420, thereby watermarking the training image 420 to generate a watermarked training image 430.

[0109] Server system 102 applies distortion to the watermarked image (525). See reference... Figure 4 As described, the distortion device 440 can process the watermarked training image 430 to generate one or more distorted images 450. The distorted images 450 are generated by adding one or more distortions such as vertical and horizontal scaling, stitching with other background images, JPEG compression, and cropping to simulate real-world image changes that the possibly coded image 302 in FIG. 3 might undergo. For example, the distortion device 440 can distort the watermarked training image 430 based on a random vertical scaling factor.

[0110] Server system 102 uses a distortion detector machine learning model to predict distortion (530). See reference... Figure 4 As described, the distortion detector machine learning model 132b processes the distorted watermarked training image 430 to generate one or more predicted distortions. For example, by processing the distorted watermarked training image 430, the distortion detector machine learning model 132b can generate a predicted vertical scaling factor as output, indicating the estimated level of vertical scaling of the distorted watermarked training image 430 relative to the training image 420.

[0111] The server system modifies the distorted watermarked training image based on one or more predicted distortions (535). See reference. Figure 4 As described, after predicting one or more distortions in the watermarked training image 430, the image analysis and decoder device 118 can modify portions of the distorted watermarked training image 430 to generate a modified portion of the distorted watermarked training image 430. For example, suppose the distortion detector machine learning model 132b predicts that the watermarked training image 430 has undergone a 2x vertical scaling. In response to this prediction, the image analysis and decoder device 118 can modify portions of the distorted watermarked training image 430 to generate a modified version with a 1 / 2 vertical scaling factor, thereby mitigating any distortion experienced by the watermarked training image 430.

[0112] Similarly, if the distortion detector machine learning model 132b predicts that the watermarked training image 430 has undergone horizontal scaling, the image analysis and decoder device 118 can modify a portion of the distorted watermarked training image 430 by scaling that portion to generate a modified version.

[0113] Server system 102 decodes the watermark to generate the first predicted data item (540). See reference... Figure 4 As described, in order to decode the watermark detected in the watermarked training image, the decoder machine learning model 134a processes the modified portion of the distorted watermarked training image to generate a first predicted data item as output.

[0114] Server system 102 determines a first error value (545). For example, Loss1 425 is calculated based on training image 420 and watermarked training image 430, where Loss1 425 indicates the difference between training image 420 and watermarked training image 430. For example, a per-pixel loss function such as an absolute error function can be used to calculate the difference between images 420 and 430 at the pixel level.

[0115] Server system 102 determines a second error value (550). For example, the second error value (referred to as Loss2 460) is calculated based on the predicted first data item 460 and the first data item 410. The second error value indicates the difference between the predicted value and the actual value of the first data item used to watermark the image. For example, loss2 could be an sigmoid cross-entropy loss.

[0116] Server system 102 adjusts the parameters (555) of the encoder and decoder machine learning models. After calculating Loss1425 and Loss2470, the learnable parameters of encoder machine learning model 112 and decoder machine learning model 134a can be adjusted to minimize the total loss (i.e., Loss1 + loss2) or the individual values ​​of loss1 and loss2. For example, the magnitude of the loss value indicates how far the prediction is from the actual value, and the sign of the loss value indicates the direction in which the learnable parameters must be adjusted. The total loss can be expressed as follows:

[0117] Total loss = ||I w -I o || 2 + Cross-entropy (M) d M o )

[0118] Where I w It is the watermarked training image 430, I o The training image is 420, M d The first predicted data item is 460, and M o The target is the first data item, 410. In other words, the magnitude of the loss value indicates how far the prediction is from the actual value, and the sign of the loss value indicates the direction in which the learnable parameters must be adjusted.

[0119] Figure 5BThis is a flowchart of an exemplary process 560 for training a machine learning model for a distortion detector. The operation of process 560 can be implemented, for example, by a server system 102 including image analysis and decoder devices 118. The operation of process 550 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of process 560.

[0120] The training process 560 iterates over the training samples in the end-to-end training dataset. Training process 560 terminates when a termination criterion is met. For example, training process 560 may terminate when the total loss falls below a specified threshold. For instance, if the specified threshold for the total loss is set at 0.1, the training process will continue iterating over the training images until the total loss value is <= 0.1. In another example, training process 560 may terminate after a specified number of iterations (e.g., 10,000 iterations).

[0121] Server system 102 obtains multiple training images and multiple data items (565). For example, a distortion detector machine learning model can be trained using a distortion model training dataset. The distortion model training dataset may include multiple training images and multiple first data items. For example, the training images may be images similar to source images 128a-n of third-party content provided to client device 104, and the first data items may be first data items processed by encoder machine learning model 112 to generate a second watermark for watermarking the training images.

[0122] Server system 102 fixes the weights of the encoder machine learning model (570). In order to detect distortion in the possible encoded image 302 or distortion in a portion of the possible encoded image 302 specifically encoded by the trained encoder machine learning model 112, training process 560 may fix the parameters of encoder machine learning model 112 so that each training image in the distortion model training dataset is watermarked to generate a corresponding watermarked training image.

[0123] Server system 102 uses an encoder machine learning model to generate watermarked training images (575). For example, an encoder machine learning model 112 implemented within the watermark generator 110 of server system 102 encodes a first data item to generate a first watermark (as shown and described with reference to FIG. 2A). After generating the first watermark, server system 102 uses a watermark tileization device 114 to combine multiple instances of the first watermark to generate a second watermark, for example, a tiled version of the first watermark (as shown and described with reference to FIG. 2B). The second watermark may also undergo processing such as cropping (as discussed with reference to FIG. 2B and 3A). Server system 102 combines the second digital watermark with the training image to obtain a watermarked image. Figure 4 As described, server system 102 can use techniques such as alpha mixing to combine a second watermark and a training image, thereby watermarking the training image to generate a watermarked training image.

[0124] Server system 102 applies distortion to the watermarked image (580). See reference... Figure 4 The distortion device can process watermarked training images to generate one or more distorted watermarked images. Distorted watermarked images are generated by adding one or more distortions such as vertical and horizontal scaling, stitching with other background images, JPEG compression, and cropping to simulate real-world image changes that the possible coded image 302 in Figure 3 might undergo. For example, the distortion device can distort watermarked training images from a distortion model training dataset to generate distorted watermarked training images.

[0125] Server system 102 uses a distortion detector machine learning model to predict distortion (585). See reference... Figure 4 As described, the distortion detector machine learning model 132b processes the distorted watermarked training image to generate one or more predicted distortions. For example, by processing the distorted watermarked training image, the distortion detector machine learning model 132b can generate a predicted vertical scaling factor as output, indicating the estimated level of vertical scaling of the distorted watermarked training image relative to the training image.

[0126] To generate output, the training process 560 can provide the distorted watermarked training image and the watermarked training image as input to the distortion detector machine learning model 132b to generate one or more outputs indicating one or more distortions in the distorted watermarked training image and the watermarked training image. For example, after generating the distorted watermarked training image from the watermarked training image, the training process can provide the watermarked training image as input to the distortion detector machine learning model 132b to generate a pattern (called a general pattern). Similarly, the training process can provide the distorted watermarked training image as input to the distortion detector machine learning model 132b and generate another pattern (called a transformed pattern) as output.

[0127] Server system 102 determines a third error value (590). For example, after generating a general pattern and a transformed pattern, the training process compares the two patterns to calculate the third error value using a loss function (e.g., L2 loss). The third error value can sometimes be expressed in the form ||T(U0)-U1|| 2 , where T refers to the transformation applied to the watermarked training image by adding one or more distortions, U0 is the general pattern and U1 is the transformation pattern.

[0128] Server system 102 adjusts the parameters of the encoder and decoder machine learning models (595). For example, training process 560 can use a third error value to adjust various parameters of the distortion detector machine learning model 132b. The training process can be iterative in nature, such that during each iteration, the training process aims to minimize the L2 loss, for example, until the loss is less than a specified threshold or until the training process has performed a specified number of iterations.

[0129] Figure 6 This is a flowchart of an exemplary process 600 for adding a digital watermark to a source image. The operation of process 600 can be, for example, by... Figure 1 The system implementation shown includes a server system 102, which in turn includes a watermark generator 110. The operation of process 600 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of process 600.

[0130] Following the training of the end-to-end learning pipeline, a watermark generator 110, comprising an encoder machine learning model 112 and a watermark tileization device 114, is deployed by the entity providing the digital watermark. For example, if server system 102 is configured to communicate with the computing systems of content providers 106a-n, for example, to obtain a source image 128a for client device 104, server system 102 may include a watermark generator 110 that can be used to generate a digital watermark. Server system 102 may transmit the source image 128a and the semi-transparent watermark, along with instructions instructing an application executing on client device 104 to overlay the semi-transparent watermark onto the source image 128a after generating a semi-transparent watermark. If content providers 106a-n are configured to communicate independently with client device 104, content providers 106a-106n may include a watermark generator 110 that can be used to generate a digital watermark.

[0131] Server system 102 obtains the source image (610). For example, client device 104 may request the source image 128a directly from the corresponding computing system of one of the content providers 106a-n, or indirectly via an intermediary service such as a service provided by server system 102 or another server system. Server system 102 may be configured to communicate with the computing systems of content providers 106a-n, for example, to obtain the source image 128a to be provided to client device 104.

[0132] Server system 102 receives a first data item (620). For example, server system 102 may be configured to respond to a request from client device 104 with an electronic document and a semi-transparent second watermark 126 to be displayed on source image 128a in the electronic document. To generate the semi-transparent watermark, server system 102 may include a watermark generator 110, which may further include an encoder machine learning model 112 that can generate the first watermark by processing the first data item 122. For example, the first data item 122 may be a unique identifier identifying content providers 106a-n. The first data item 122 may also include a session identifier that uniquely identifies a network session between client device 104 and server system 102 during which a response is provided to a request from client device 104. The first data item 122 may also include or reference image data that identifies a specific source image 128a provided to the client device 104 or information associated with the source image 128a (e.g., information indicating which of the content providers 106a-n provided the specific source image 128a to the client device 104 and a timestamp indicating when the source image 128a was provided or requested).

[0133] Server system 102 generates the first digital watermark (630). See reference... Figure 1 The encoder machine learning model 112, implemented within the watermark generator 110 of server system 102, is configured to receive a first data item 122 as input to generate a first watermark 124, which encodes the first data item 124 into the first watermark 124. In some embodiments, the first watermark 124 may be a matrix barcode representing the first data item 122 depicted in FIG. 2. The first watermark 124 may have a predefined size in terms of the number of rows and columns of pixels. Each pixel in the first watermark 124 may encode multiple data bits, wherein the values ​​of the multiple bits are represented by different colors. For example, a pixel encoding the binary value '00' may be black, while a pixel encoding the binary value '11' may be white. Similarly, a pixel encoding the binary value '01' may be a lighter shade of black (e.g., dark gray), while a pixel encoding the binary value '10' may be a lighter shade of black (e.g., light gray). In some embodiments, the smallest encoding unit of the first watermark may actually be larger than a single pixel. However, for the purposes of the examples described herein, it is assumed that the smallest coding unit is a single pixel. It should be recognized, however, that the techniques described herein can be extended to implementations where the smallest coding unit is a set of multiple pixels (e.g., a 2×2 or 3×3 pixel set).

[0134] The exemplary first watermark 124 is further explained with reference to Figure 2A, which depicts the exemplary watermark 200. In this example, the watermark 200 has a fixed size of 32 × 64 pixels, but watermarks with other predefined sizes would also be suitable. A distinguishing feature of the watermark 200 is that each pixel can employ different color shading values, including white or black.

[0135] Server system 102 generates a second digital watermark (640). See reference... Figure 1As described, after generating a first watermark 124, server system 102 uses watermarking device 114 to combine multiple instances of the first watermark 124 to generate a second watermark 126. For example, watermarking device 114 can generate the second watermark 126 by placing two or more instances of the first watermark 124 side by side. An exemplary third watermark 126 is further explained with reference to FIG. 2B, which depicts an exemplary second watermark 250. In the example of FIG. 2B, the second watermark 250 has a size of 64 × 128 pixels and is generated by watermarking device 114 by placing four first watermarks in a 2x2 array. For example, watermark 250 includes four instances (255 to 258) of the first watermark 124. After generating the second watermark 126, the second watermark 128 can be updated such that the size of the second watermark is no larger than the size of the source image 128a. For example, if the size of the second watermark 126 is larger than the size of the source image 128a, the second watermark 128 can be cropped to match the size of the source image 128a.

[0136] The second digital watermark 250 is combined with the source image 128a to obtain the watermarked image 130 (650). See reference... Figure 1 As described, when server system 102 generates a response to return to client device 104 as a reply to the client's request for an electronic document, the response may include computer code instructing client device 104 to display one or more instances of the second watermark 126 on the source image 128a when executing the response, for example, to add a watermark to the source image 128a that is substantially imperceptible to a human user. The application rendering the electronic document at client device 104 may perform alpha blending to overlay the second watermark 126 onto the source image 128a according to a specified transparency, which specifies the opacity of the second watermark 126 when overlaid on the source image 128a. For example, server system 102 may add code that instructs client device 104 to display the source image 128a as a background image in a third-party content slot in the electronic document and to display one or more instances of the second watermark 126 as a foreground image on image 128a. The alpha blending technique, which overlays the second watermark 126 onto the source image 128a, can also be performed by other entities such as server system 102 or content providers 106a-n. For example, if server system 102 or content providers 106a-n are configured to transmit the watermarked image 130 to client device 104, the corresponding entity can perform the alpha blending technique to generate the watermarked image 130 and then transmit the watermarked image 130 to client device 104.

[0137] Figure 7This is a flowchart of an exemplary process 700 for detecting whether an image contains one or more digital watermarks and decoding the one or more digital watermarks. The operation of process 700 can be implemented, for example, by a server system 102 including an image analysis and decoder device 118. The operation of process 700 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing devices can cause the one or more data processing devices to perform the operation of process 700.

[0138] Process 700 has been explained with reference to the assumption that the image analysis and decoder device 118 is implemented in server system 102. However, it should be understood that the image analysis and decoder device 118 may be implemented by other entities such as content providers 106a-n.

[0139] Server system 102 obtains the image (710). In some implementations, and as referenced... Figure 1 and 3A As described, the potentially encoded image 302 is obtained by server system 102, which includes an image analysis and decoder device 118, which further includes a watermark and distortion detection device 132 and a watermark decoder 134. For example, a user of client device 104 may receive inappropriate or irrelevant content (e.g., violence, gore, hate speech, or any content that raises concerns about the source of the content) in response to a request for an electronic document. A user may capture a screenshot of the content (referred to as a potentially encoded image or candidate image) and transmit the screenshot to image analysis and decoder device 118 for analysis, for example, to inquire about the source of the content presented to the user and depicted by the potentially encoded image. While image analysis and decoder device 118 receives multiple images, they do not need to be received simultaneously. For example, images may be acquired over a period of time because the images are submitted by a user presenting content on the publisher's property.

[0140] Server system 102 determines that the digital watermark is embedded in a portion of a potentially encoded image (720). See reference... Figure 3AAs described, before any processing of the potentially encoded image 302 to check for any distortion or decoding by the watermark decoder 134, the watermark and distortion detection device 132 performs the determination of whether the potentially encoded image includes a visually identifiable watermark. Using the detection machine learning algorithm 132a to determine whether the potentially encoded image 302 includes a watermark before any further processing of the image provides a more efficient computational system. For example, the UNet-based detection machine learning model 132a can be used to detect the presence of a watermark in a received image before the distortion detector machine learning model 132b and the decoder machine learning model, which require more computationally intensive processing, process the potentially encoded image 302. In this way, the system can ignore any images where no watermark is detected, without wasting resources required to perform further computation. For example, if no watermark is detected in the potentially encoded image 302, the server system 102 can employ other techniques (outside the scope of this document) to verify the presence of a watermark in the potentially encoded image 302.

[0141] The watermark detector machine learning model 132a is configured to process the potentially encoded image 302 and generate an indication as output whether the potentially encoded image 302 includes a portion of a watermark or one or more watermarks. For example, the watermark detector machine learning model can be implemented as a classification model that can process the potentially encoded image 302 to classify the image as an image that includes a watermark or does not include a watermark.

[0142] The watermark detector machine learning model 132 can be configured to perform semantic image segmentation to determine the watermark-included portion of the possibly encoded image 302.

[0143] Server system 102 uses a distortion detector machine learning model to predict one or more distortions in a portion of a potentially encoded image (730). See reference... Figure 3A As described, the watermark and distortion detection device 132 can implement a distortion detector machine learning model 132b, which can be configured to process the potentially encoded image 302 or a portion of the potentially encoded image 302 including one or more watermarks (obtained from step 620 of process 600) to generate an indication as output of the distortion that the potentially encoded image 302 has undergone relative to the source image 128a. For example, the distortion detector machine learning model can generate an indication of vertical and horizontal scaling as output, where vertical and horizontal scaling are distortions along the length and width of the potentially encoded image 302 or a portion of the potentially encoded image 302. It should be noted that the watermark and distortion detection device 132 can process the potentially encoded image 302 or a portion of the potentially encoded image 302 after the detection machine learning model 132a has successfully determined the presence of a watermark in the potentially encoded image 302.

[0144] Server system 102 modifies portions (740) of a potentially encoded image based on one or more predicted distortions. For example, after detecting and determining that a portion of a potentially encoded image 302 includes one or more watermarks, image analysis and decoder device 118 can modify portions of the image based on distortions predicted by distortion detector machine learning model 132b to generate modified portions of the potentially encoded image 302 that are similar to or nearly similar to the source image 130. For example, after determining the presence of a watermark on the potentially encoded image 302 using watermark detector machine learning model 132a, image analysis and decoder device 118 can obtain portions of the potentially encoded image 302 that include one or more watermarks. In response to the determination of the presence of one or more watermarks, image analysis and decoder device 118 can also generate one or more predictions indicating different distortions experienced by the potentially encoded image 302. In response to predicted distortions, image analysis and decoder device 118 can modify portions of the potentially encoded image 302 to mitigate any distortions experienced by the potentially encoded image 302. For example, if the distortion detector machine learning model 132b predicts that the potentially encoded image 302 has undergone vertical and / or horizontal scaling (identified using vertical and horizontal scaling factors), the image analysis and decoder device 118 can modify a portion of the potentially encoded image 302 by scaling that portion to generate a modified version. The modified version is then inversely scaled with the same vertical and / or horizontal scaling factors predicted by the distortion detector machine learning model 132b, thereby mitigating any vertical and / or horizontal distortion experienced by the potentially encoded image 302. For example, if the distortion detector machine learning model predicts that the watermarked portion of the potentially encoded image 303 has undergone a horizontal scaling of 2 and a vertical scaling of 3, a modified portion of the potentially encoded image 302 will be generated by performing a horizontal scaling of 1 / 2 and a vertical scaling of 1 / 3 on a portion of the potentially encoded image 130.

[0145] Server system 102 decodes the watermark included in the modified portion of the image (750). See reference... Figure 3A As described, for example, in order to decode a watermark detected in a possibly encoded image 302, the image analysis and decoder apparatus includes a watermark decoder 134. In some embodiments, the watermark decoder 134 may implement a decoder machine learning model 134a, which is configured to decode a modified portion of the possibly encoded image 302 to generate a predicted first data item as output, the predicted first data item being predicted to be encoded within the watermark included in the image.

[0146] Server system 102 verifies the predicted first data item (760). For example, after generating the predicted first data item by processing the possibly encoded image 302, image analysis and decoder device 118 can use the predicted first data item to verify the authenticity (or source) of the possibly encoded image 302. To verify authenticity (or source), verification device 140 implemented within server system 102 can compare the predicted first data item with a first data item stored in response record database 120. If a match is found, verification device 140 can conclude that the source image 128a presented on client device 104 was actually provided by server system 102 or content provider 106a-b. If no match is found, verification device 140 can conclude that the source image 128a presented on client device 104 was not provided by server system 102 or content provider 106a-b.

[0147] Figure 8 This is a block diagram of an exemplary computer system 800 that can be used to perform the operations described above. System 800 includes a processor 810, memory 820, storage device 830, and input / output device 840. Each of components 810, 820, 830, and 840 may be interconnected, for example, using a system bus 850. Processor 810 is capable of processing instructions for execution within system 800. In some embodiments, processor 810 is a single-threaded processor. In another embodiment, processor 810 is a multi-threaded processor. Processor 810 is capable of processing instructions stored in memory 820 or on storage device 830.

[0148] Memory 820 stores information within system 800. In one embodiment, memory 820 is a computer-readable medium. In some embodiments, memory 820 is a volatile memory cell. In another embodiment, memory 820 is a non-volatile memory cell.

[0149] Storage device 830 provides high-capacity storage for system 800. In some embodiments, storage device 830 is a computer-readable medium. In various different embodiments, storage device 830 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices (e.g., cloud storage devices) over a network, or some other high-capacity storage device.

[0150] Input / output device 840 provides input / output operations for system 700. In some embodiments, input / output device 840 may include one or more network interface devices, such as Ethernet cards, serial communication devices (e.g., RS-232 ports), and / or wireless interface devices, such as 802.11 cards. In another embodiment, input / output device may include driver devices configured to receive input data and send output data to external device 860, such as keyboards, printers, and display devices. However, other embodiments, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc., may also be used.

[0151] Despite Figures 1 to 6 An example processing system is described herein, but the implementation of the subject matter and functional operations described herein may also be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware that includes the structures disclosed herein and their equivalents, or in a combination of one or more of them.

[0152] The embodiments of the subject matter and operation described in this specification can be implemented in digital electronic circuits, or in computer software, firmware, or hardware that includes the structures disclosed in this specification and their equivalents, or in a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more computer storage media (or media) for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these, or included therein. Furthermore, although the computer storage medium is not a propagating signal, it can be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices) or included therein.

[0153] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0154] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, systems-on-a-chip, and multiple or combinations thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0155] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected through a communications network.

[0156] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuits, and the devices can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0157] For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices (e.g., hard disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices; hard disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. Processors and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

[0158] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from devices used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0159] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0160] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The relationship between the client and server arises from computer programs running on their respective computers, and a client-server relationship exists between them. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from the user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) may be received at the server from the client device.

[0161] Although this specification includes numerous specific implementation details, these details should not be construed as limiting any invention or the scope of any possible claims, but rather as descriptions of specific features of particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations, and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations thereof.

[0162] Similarly, although the operations are depicted in a specific order in the figures, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential manner, or performing all the illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0163] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for jointly training an encoder machine learning model for generating a digital watermark embedded in an image and a decoder machine learning model for decoding a first data item encoded within the digital watermark embedded in the image, wherein the training comprises: Obtain a first plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; For each of the first plurality of training images: Obtain data items from the plurality of data items; The encoder machine learning model, which is provided with the data item as input, is used to generate a first digital watermark that encodes the data item; Two or more instances of the first digital watermark are tiled to generate a second digital watermark; The second digital watermark is combined with the training image to obtain a watermarked training image; Apply one or more distortions to the watermarked training images; A distortion detector machine learning model is used to predict the one or more distortions present in the watermarked training images; The watermarked training image is modified based on one or more predicted distortions, while the second digital watermark embedded in the distorted watermarked training image is preserved. as well as The modified watermarked training image is decoded using the decoder machine learning model to obtain a decoded data item to be embedded in the second digital watermark in the distorted watermarked training image. A first error value is determined based on the watermarked training image and the training image; A second error value is determined based on the decoded data item and the data item; as well as Adjust one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.

2. The method of claim 1, wherein determining the first error value based on the watermarked training image and the training image comprises: The L2 loss value is calculated based on the watermarked training image and the training image.

3. The method of claim 1, wherein determining the second error value based on the decoded data item and the data item comprises: The S-shaped cross-entropy value is calculated based on the decoded data item and the data item.

4. The method of claim 1, wherein the one or more distortions include image compression, Gaussian noise, or image scaling.

5. The method of claim 1, further comprising training the machine learning model of the distortion detector, including: Obtain a second plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; The weights of the encoder's machine learning model are fixed; as well as For each of the second plurality of training images: The encoder machine learning model is used to generate watermarked training images; Apply one or more distortions to the watermarked training images; The distortion detector machine learning model is used to predict the presence of one or more distortions in the training images with distorted watermarks; The distortion detector machine learning model is used to predict one or more distortions present in the watermarked training images; A third error value is determined based on one or more predicted distortions present in the distorted watermarked training image and the predicted one or more distortions present in the watermarked training image; and Adjust one or more training parameters of the distortion detector machine learning model to minimize the third error value.

6. The method of claim 5, wherein determining the third error value comprises: The L2 loss value is calculated based on one or more predicted distortions present in the watermarked training image with the distortion.

7. A system for jointly training an encoder machine learning model for generating a digital watermark embedded in an image and a decoder machine learning model for decoding a first data item encoded within the digital watermark embedded in the image, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the training, wherein the training includes: Obtain a first plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; For each of the first plurality of training images: Obtain data items from the plurality of data items; The encoder machine learning model, which is provided with the data item as input, is used to generate a first digital watermark that encodes the data item; Two or more instances of the first digital watermark are tiled to generate a second digital watermark; The second digital watermark is combined with the training image to obtain a watermarked training image; Apply one or more distortions to the watermarked training images; A distortion detector machine learning model is used to predict the one or more distortions present in the watermarked training images; The watermarked training image is modified based on one or more predicted distortions, while the second digital watermark embedded in the distorted watermarked training image is preserved. as well as The modified watermarked training image is decoded using the decoder machine learning model to obtain a decoded data item to be embedded in the second digital watermark in the distorted watermarked training image. A first error value is determined based on the watermarked training image and the training image; A second error value is determined based on the decoded data item and the data item; as well as Adjust one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.

8. The system of claim 7, wherein determining the first error value based on the watermarked training image and the training image comprises: The L2 loss value is calculated based on the watermarked training image and the training image.

9. The system of claim 7, wherein determining the second error value based on the decoded data item and the data item comprises: The S-shaped cross-entropy value is calculated based on the decoded data item and the data item.

10. The system of claim 7, wherein the one or more distortions include image compression, Gaussian noise, or image scaling.

11. The system of claim 7, further comprising training the machine learning model of the distortion detector, including: Obtain a second plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; The weights of the encoder's machine learning model are fixed; as well as For each of the second plurality of training images: The encoder machine learning model is used to generate watermarked training images; Apply one or more distortions to the watermarked training images; The distortion detector machine learning model is used to predict the presence of one or more distortions in the training images with distorted watermarks; The distortion detector machine learning model is used to predict one or more distortions present in the watermarked training images; A third error value is determined based on one or more predicted distortions present in the watermarked training image with the distortion; and Adjust one or more training parameters of the distortion detector machine learning model to minimize the third error value.

12. The system of claim 11, wherein determining the third error value comprises: The L2 loss value is calculated based on one or more predicted distortions present in the watermarked training image with the distortion.

13. A non-transitory computer-readable medium storing instructions that, when executed by one or more data processing means, cause the one or more data processing means to perform operations, said operations for jointly training an encoder machine learning model for generating a digital watermark embedded in an image and a decoder machine learning model for decoding a first data item encoded within the digital watermark embedded in the image, wherein said training includes: Obtain a first plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; For each of the first plurality of training images: Obtain data items from the plurality of data items; The encoder machine learning model, which is provided with the data item as input, is used to generate a first digital watermark that encodes the data item; Two or more instances of the first digital watermark are tiled to generate a second digital watermark; The second digital watermark is combined with the training image to obtain a watermarked training image; Apply one or more distortions to the watermarked training images; A distortion detector machine learning model is used to predict the one or more distortions present in the watermarked training images; The watermarked training image is modified based on one or more predicted distortions, while the second digital watermark embedded in the distorted watermarked training image is preserved. as well as The modified watermarked training image is decoded using the decoder machine learning model to obtain a decoded data item to be embedded in the second digital watermark in the distorted watermarked training image. A first error value is determined based on the watermarked training image and the training image; A second error value is determined based on the decoded data item and the data item; as well as Adjust one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.

14. The non-transitory computer-readable medium of claim 13, wherein determining the first error value based on the watermarked training image and the training image comprises: The L2 loss value is calculated based on the watermarked training image and the training image.

15. The non-transitory computer-readable medium of claim 13, wherein determining the second error value based on the decoded data item and the data item comprises: The S-shaped cross-entropy value is calculated based on the decoded data item and the data item.

16. The non-transitory computer-readable medium of claim 13, wherein the one or more distortions include image compression, Gaussian noise, or image scaling.

17. The non-transitory computer-readable medium of claim 13, further comprising training the machine learning model of the distortion detector, including: Obtain a second plurality of training images and a plurality of data items, wherein each of the plurality of data items is a data item to be encoded within a digital watermark to be embedded in the training images; The weights of the encoder's machine learning model are fixed; as well as For each of the second plurality of training images: The encoder machine learning model is used to generate watermarked training images; Apply one or more distortions to the watermarked training images; The distortion detector machine learning model is used to predict the presence of one or more distortions in the training images with distorted watermarks; The distortion detector machine learning model is used to predict one or more distortions present in the watermarked training images; A third error value is determined based on one or more predicted distortions present in the watermarked training image with the distortion; and Adjust one or more training parameters of the distortion detector machine learning model to minimize the third error value.

18. The non-transitory computer-readable medium of claim 17, wherein determining the third error value comprises: The L2 loss value is calculated based on one or more predicted distortions present in the watermarked training image with the distortion.

Images