Verification of the provenance of digital objects using watermarking and embedding.

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for verifying the origin of digital objects generated by a neural network, such as images or audio objects. Also, methods, systems, and apparatus, including computer programs for training watermarking neural networks and watermarking-decoding neural networks. The techniques described are efficient in their use of computing resources and robust against attacks.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Application No. 18 / 510,537, filed on 15 November 2023, which claims the benefit of the filing date of European Patent Application EP23167380.7, filed on 11 April 2023, which is incorporated herein by reference in its entirety. [Background technology]

[0002] This specification relates to verifying the origin of digital objects, such as digital objects generated using machine learning models.

[0003] Machine learning models can be trained to generate parts of text or digital objects such as images. Some machine learning models are parametric models, which produce outputs based on the values ​​of the model's parameters. Neural networks are machine learning models that use one or more layers of nonlinear units. Deep neural networks include one or more hidden layers in addition to the output layer. Each layer of the network produces an output according to the current values ​​of its respective set of parameters. [Overview of the project]

[0004] This specification describes a method and a corresponding system for verifying the provenance of digital objects, which can be implemented as a computer program on one or more computers in one or more locations. Digital objects may be, for example, still images or videos, digital audio objects representing audio waveforms, or multimedia objects, or combinations thereof. Also described are a method and a corresponding system for training a system for watermarking digital objects.

[0005] In one embodiment, a computer-implemented method for training a watermarking system including a watermark-generating neural network and a watermark-decrypting neural network is described. Generally, the watermark-decrypting neural network is configured to distinguish between the presence and absence of a watermark. In some embodiments, the watermark-decrypting neural network can decrypt information from the watermark (but does not have to).

[0006] One embodiment of the method is used specifically for audio data objects. Generally, audio data objects include a representation of an audio signal, more specifically, a digital representation of an audio signal in the time domain. An audio object may be a time slice of an augmented audio signal, e.g., a moving time slice, or a discrete object. The training method is performed using a plurality of such audio data objects. Broadly speaking, the method involves generating a spectrogram of the audio data object and processing this spectrogram using a watermark-generating neural network to generate a watermark. Adversarial transformations are applied to clean audio data objects and watermarked objects, and a watermark-decoding neural network is used to process these and generate watermarked signals. The system is trained with the objective of accurately classifying each watermarked signal, using in the form of cross-entropy loss in the embodiment.

[0007] In other embodiments, a computer-implemented method is described for verifying the origin of a digital object, particularly to predict (indicate) whether the digital object was created by a generative neural network (or, if the digital object was created by a generative neural network, the generative neural network used to generate the digital object). The method includes maintaining an object verification system, in embodiments the object generation and verification system includes a first interface for receiving a digital object, or a request to generate or verify a digital object; optionally, a generative neural network configured to process a request to generate a digital object according to the request; a watermarking neural network configured to process a digital object to generate a watermarked digital object; and a second interface (which may be the same as or different from the first interface) for providing the watermarked digital object for use. The object generation and verification system also includes an embedding neural network configured to process a digital object to generate an embedding of the digital object; and an object verification database configured to store at least the embedding of the digital object.

[0008] As used herein, the “embedding” of a digital object may refer to a representation of the object as an ordered set of numbers, for example, as a vector or matrix of numbers. Entity embeddings can be generated as the output of a neural network that processes data characterizing the entity.

[0009] The method also includes receiving a query digital object for verification, processing the query digital object using a watermark-decoding neural network to generate a watermark signal for the query digital object, and processing the query digital object again using an embedding neural network to generate a query embedding for the query digital object. The object verification database is queried using query embeddings to determine one or more sets of similarity scores against one or more stored embeddings of digital objects similar to the corresponding query digital object. The origin of the query digital object is then verified based on the combination of the watermark signal and the one or more sets of similarity scores. Generally, the watermark signal can indicate whether or not the query digital object is watermarked.

[0010] In some embodiments, the method also includes receiving, for example, a request from a user to generate a digital object, and processing the request using a generative neural network to generate the digital object. The digital object is then processed using a watermarking neural network to generate a watermarked digital object, which is then provided, for example, for use by the user. The digital object is also processed using an embedding neural network to generate an embedding of the digital object, which is stored in an object validation database. The origin of the query digital object can then be validated as generated by the object generation and validation system, provided that the query embedding matches an embedding of the digital object stored in the object validation database (i.e., only if a match is found).

[0011] In some embodiments, the watermarking neural network and the watermark decoding neural network are jointly trained (end-to-end) to generate watermarked signals under adversarial transformations of watermarked digital objects, i.e., transformations or perturbations that increase the likelihood that the watermarked signal will incorrectly indicate that the watermarked digital object is not watermarked.

[0012] In a related embodiment, a computer-implemented method for generating a digital object having a verifiable provenance is described. The method includes maintaining an object generation and verification system as described above. The method also includes receiving a request to generate a digital object, processing the request using a generation neural network to generate a digital object, processing the digital object using a watermarking neural network to generate a watermarked digital object, and providing the watermarked digital object for use.

[0013] The method further includes processing digital objects using an embedded neural network to generate embeddings for the digital objects, and storing the embeddings for the digital objects in an object validation database. Thus, the origin of a digital object is queryed using an embedded neural network to generate query embeddings for the digital object, Verification is possible by querying an object validation database using query embeddings to determine one or more sets of similarity scores for one or more stored embeddings of digital objects similar to the corresponding query digital object, and by verifying the origin of the query digital object based on a combination of a watermark signal indicating the watermark of the query digital object and one or more sets of similarity scores.

[0014] In a further embodiment, a computer-implemented method for training a watermarking system for watermarking digital objects is described. This method may be used to train the watermarking neural network and watermark decoding neural network described above. Digital objects may include image objects, i.e., still images or moving images, or audio data objects, or both.

[0015] Some of the training methods described herein are particularly well-suited for training a watermarking system to watermark images, for example, due to the adversarial transformations (maybe multiple) used by those methods. Some are particularly well-suited for training a watermarking system to watermark audio data objects. In some embodiments, these approaches, for example, different types of adversarial transformations, can be combined to train a watermarking system suitable for watermarking multimedia objects containing two or more elements, such as text, images, and audio, for example, data objects combining audio and images.

[0016] In one embodiment, the watermarking system includes a watermarking neural network configured to process a digital object according to watermarking neural network parameters in order to generate a watermarked digital object, and a watermark decoding neural network configured to process the watermarked digital object according to watermark decoding neural network parameters in order to generate a watermarked signal.

[0017] In this embodiment, the method is performed for each of a plurality of training objects. The method processes the training objects using a watermarking neural network to generate watermarked training objects. The method then applies a differentiable adversarial transformation to the watermarked training objects to generate alternative training objects. Generally, an adversarial transformation is a transformation that reduces the accuracy of the watermarking signal when identifying a watermarked training object as watermarked. The method processes the alternative training objects using a watermark-decoder neural network to generate watermarking signals for the alternative training objects.

[0018] This method backpropagates the gradient of the objective function, and the gradient is obtained with respect to the watermark decoding neural network parameters and the watermarking neural network parameters. The gradient is backpropagated through the watermark decoding neural network, a differentiable adversarial transformation, and the watermarking neural network to update the watermark decoding neural network parameters and the watermarking neural network parameters. Any suitable gradient descent optimization algorithm, such as Adam or AdamW, or other optimization algorithms can be used for this. The objective function measures the accuracy of the watermarked signal when identifying watermarked training objects as watermarked. In the backpropagation method, the watermark decoding neural network and the watermarking neural network are trained together to optimize the objective function.

[0019] In some further embodiments, this specification describes watermarking neural networks and watermark decoding neural networks trained in this manner.

[0020] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages:

[0021] In general, there is a need for improved systems to identify the origin of digital objects generated using machine learning models. The increasing proliferation of such objects and their size, for example, the size / resolution of generated images, can lead to computational bottlenecks. Therefore, there is a general need for techniques to address this while maintaining robustness.

[0022] The system implementation is faster than several other approaches and can provide precedence verification with fewer computations required. The training method described can also reduce the computations required and facilitates the use of asymmetric architectures that allow the watermarking neural network to be shallower than the watermark decoding neural network, i.e., have fewer parameters (e.g., weights). Thus, watermarks can be generated quickly, but these watermarks can still be reliably detected using a deeper watermark decoding neural network.

[0023] In the system's embodiment, the object verification database stores the embeddings of the original digital objects generated by the generative neural network, rather than the embeddings of watermarked digital objects. Nevertheless, it is not necessary to remove the watermark from the query digital object before querying the object verification database to generate the query embeddings used to determine the similarity score. This approach helps provide robustness when verifying the provenance of digital objects. Optionally, the object verification database may also store the initially generated digital objects to facilitate further checks.

[0024] Embodiments of the system are resistant to adversarial perturbations, such as attempts to modify a watermarked digital object, e.g., a watermarked image, so that the watermark is not recognized. Thus, a trained watermark decoding neural network can detect the presence of a watermark even under various transformations such as standard image editing conversions. In an embodiment, the embedding of digital objects is also resistant to adversarial perturbations. As described herein, embedding a watermark in an audio data object based on a spectrogram can facilitate hiding the watermarking information and make it impossible to hear the watermark, and can also enhance the robustness of the watermark against conversions such as such audio compression.

[0025] In an embodiment, the generated watermarks are relatively diverse, reducing the overall impact on the content of the digital object and making it difficult for malicious entities to detect the watermark. Further, by querying an object verification database using query embedding based on the watermarked digital object, the search performance, e.g., the true positive rate of the search, can be improved.

[0026] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the specification, the drawings, and the claims.

Brief Description of the Drawings

[0027] [Figure 1] An example of an object generation and verification system is shown. [Figure 2] It is a flowchart of an exemplary process for generating a digital object with a verifiable history. [Figure 3] It is a flowchart of an exemplary process for verifying the history of a digital object. [Figure 4] An example of a watermark embedding system is shown. [Figure 5]This is a flowchart illustrating an example of a process for jointly training a watermarking neural network and a watermarking decoding neural network. [Figure 6] This is a flowchart illustrating another example process for jointly training a watermarking neural network and a watermark decoding neural network. [Figure 7A] This is an illustrative flowchart of a process for watermarking an audio signal. [Figure 7B] This demonstrates the use of a trained watermarking neural network to add watermarks to digital objects. [Figure 8A] This is an illustrative flowchart of a process for identifying whether or not an audio signal is watermarked. [Figure 8B] This shows a trained watermark decoding neural network that predicts whether or not a digital object has a watermark. [Modes for carrying out the invention]

[0028] Similar reference numbers and names in various drawings refer to the same elements.

[0029] Figure 1 shows an example of an object generation and verification system 100 that can be used to verify the provenance of a digital object. The system 100 may be implemented as one or more computer programs on one or more computers in one or more locations.

[0030] System 100 includes a first interface 104 that receives requests 103 to generate a digital object. The first interface 104 may include, for example, an API (Application Programming Interface), a web page, a digital assistant, or any other type of machine or human interface. Requests may originate from a human or machine user 102 and may include information characterizing the digital object to be generated by the system, such as a text prompt characterizing a still image or video to be generated by the system. In some embodiments, the first interface 104 also provides the user 102 with a watermarked version of the digital object 113. In some embodiments, the watermarked digital object 113 is provided via a second interface (not shown) which may differ from the first interface. System 100 may include a request filter 106 to filter requests, such as text prompts, and retrieve filtered requests. Such filtering may be used to remove unwanted requests or to remove unwanted information from requests, i.e., to suppress requests that should not be used to generate a digital object.

[0031] The (filtered) request 103 is provided to the generative neural network 108, which is configured to process the request and generate a digital object 109 according to the request. In one embodiment, the generative neural network 108 may be configured to generate a still image or a moving image from a text prompt, including a depiction of the text prompt. In another example, the generative neural network 108 may be configured to generate a digital object representing audio data, for example, for speech described by a text prompt, utterances representing text within a text prompt, and / or utterances for which the request identifies a desired speaker of the utterance.

[0032] In general, a digital object can be any type of object, such as an image, audio, or multimedia object. The generated digital object may include multiple elements that define the digital object, such as continuous value elements. For example, if the generated digital object includes an image, the elements may include values ​​such as pixel intensity values. If the generated digital object includes audio, such as speech, the elements may include values ​​that characterize the audio waveform, such as instantaneous values ​​of the waveform or values ​​for the time-frequency representation of the waveform. In some embodiments, the generated digital object 109 is filtered by an object filter 110 to remove undesirable digital objects, such as undesirable images.

[0033] The (filtered) digital object 109 is provided to a watermarking neural network 112 configured to process the digital object 109 and generate a watermarked digital object 113. The watermarked digital object 113 is provided to an interface 104, which can then provide it to the user 102. The watermarking neural network 112 can be any type of neural network configured to add watermarks to digital objects, for example, a watermarking neural network trained as described below. For example, the watermarking neural network 112 can be based on a convolutional neural network or a neural network that implements a self-attention mechanism such as a transformer-based neural network (Vaswani et al., “Attention Is All You Need”), which can implement zero-bit or multi-bit watermarking techniques.

[0034] In some embodiments, the watermarking neural network 112 may be randomly selected from a group of watermarking neural networks, or the watermarking neural network 112 may include an ensemble of watermarking neural networks having outputs that are combined to add watermarks to digital objects. As another example, an ensemble of paired watermarking and watermark decoding neural networks can be used during training, for example, during training described later, to provide a unique watermark decoding neural network in the sense that the watermarks generated by the paired watermark decoding neural network cannot be detected (decoded) by the other watermark decoding neural networks in the ensemble. Such an approach can increase the robustness of the system against adversarial attacks.

[0035] The digital object 109 is also provided to an embedded neural network 114 configured to process the digital object and generate an embedding 115 of the digital object 109. The embedded neural network 114 can have any suitable neural network architecture and can include any suitable number (e.g., 5, 10, or more) of any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) and can be connected in any suitable configuration (e.g., as a linear array of layers).

[0036] In some embodiments, the embedded neural network 114 is trained using a contrasting loss. Such a contrasting loss may be one that encourages embeddings generated from transformed versions of the same training digital object to be more similar, and embeddings generated from different (i.e., not transformed versions of the same training digital object) training digital objects to be more different.

[0037] The embedding 115 of the digital object 109 is stored in an object validation database 116, such as a relational database, such as Spanner (Corbett et al., “Spanner: Google's Globally-Distributed Database”, Proceedings of OSDI 2012). In some embodiments, the original, i.e., unwatermarked, digital object 109 is also stored in the database 116, for example, in relation to the embedding 115. Storing and searching digital objects based on embedding facilitates matching between digital objects even if one of the digital objects is affected by watermarking or malicious transformation.

[0038] System 100 also includes a query interface 118 for receiving query digital objects 117 for verification from a human or machine user 124, for example, as described above with respect to interface 104. User 124 may be the same as or different from user 102.

[0039] In some embodiments, the query interface 118 can rate limit queries. For example, it can limit the rate at which a particular user can submit queries, or specify the maximum number of queries a particular user can submit within a given time interval. This can increase the system's robustness against attacks by malicious users. Such an approach can be particularly useful when combined with the watermarking neural network training scheme described later, which can result in individual watermarks that are difficult to attack.

[0040] The query interface 118 provides the query digital object 117 to a watermark-decoder neural network 120 configured to generate a watermark signal 121 for the query digital object 117. The watermark-decoder neural network 120 may be any neural network suitable for decoding the watermark applied by the watermark neural network 112, for example, a watermark-decoder neural network trained as described below. In some embodiments, the watermark neural network and the watermark-decoder neural network are jointly trained (end-to-end) to generate a watermark signal under adversarial transformation of the watermarked digital object.

[0041] The watermarking signal 121 may relate to so-called zero-bit watermarking techniques, meaning that the watermarking signal 121 may simply indicate whether or not the query digital object 117 has a watermark. For example, the watermarking signal may indicate the likelihood of a watermark being present, for example, as a score or probability.

[0042] In some embodiments, the watermark of a watermarked digital object is a so-called multi-bit watermarking technique that encodes a message containing one or more bits, such as a random number or a secret key. In this case, the watermarked signal 121 may include one component or value indicating whether or not a watermark is present, and other components or values ​​that provide information for decoding the message.

[0043] The query interface 118 provides the query digital object 117 to the embedded neural network 114, which processes the query digital object to generate a query embedding 119 for the query digital object. The query embedding 119 is used by the similarity search service 122 to query the object validation database 116 to determine one or more sets of similarity scores 123 for one or more sets of stored embeddings of corresponding digital objects similar to the query digital object 117. For example, the similarity search service 122 processes the query embedding 119 and one or more sets of stored embeddings to determine one or more sets of similarity scores 123 according to any appropriate similarity metric, such as cosine or dot product similarity or distance metrics.

[0044] One or more similarity scores 123 may be combined with a watermark signal 121, for example, by the query interface 118, to generate a verification output 125 that verifies the origin of the query digital object 117. For example, the verification output 125 may include a verification decision that identifies whether the query digital object 117 was generated by the generative neural network 108 with a probability, for example, higher than a threshold probability.

[0045] Each similarity score 123 can be combined with a watermark signal 121 in any convenient way. For example, a weighted combination of a watermark signal and a similarity score can be determined. The weights may be determined empirically as system hyperparameters based on a specific true or false positive rate or precision, for example, AUC (area under the receiver's operating characteristic curve). As another example, the determination that a query digital object 117 was generated by a generative neural network 108 can be made conditional on both a watermark signal indicating that the query object is watermarked with a probability, for example, higher than a threshold probability, and one of the similarity scores indicating that the query embedding 119 matches an embedding of a digital object stored in an object verification database with a probability, for example, higher than a threshold probability.

[0046] If the original, i.e., non-watermarked digital object 109 is also stored in the database 116, this can be used for further checks, such as tamper detection to determine whether the query digital object 117 has been modified after it was generated and provided to user 102. For example, the stored original digital object can be used to detect attempts to remove the watermark from the query digital object. Access to the original digital object also facilitates human operators to perform comparisons between the query digital object 117 and the initially generated digital object.

[0047] As mentioned above, in the embodiment, the embedded neural network 114 is trained using contrast loss. For example, the embedded neural network 114 can be trained using the approach described in SimCLR (Chen at al., “A Simple Framework for Contrastive Learning of Visual Representations”, arXiv:2002.05709). In such an approach, positive pairs of examples are generated by applying different random transformations to the same training digital object, for example, by applying cropping or color distortion to an image object, or by changing the amplitude or frequency characteristics of an audio object by filtering. Negative pairs of examples may include, for example, different digital objects. These examples can be processed by the embedded neural network 114 and subsequently by a projection neural network (which may have any suitable architecture) to project the embedding into a lower-dimensional projection embedding.

[0048] Next, the training method can determine the similarity measure between projected embeddings using similarity operations, such as a cosine similarity measure. The contrasting loss is the similarity measure between projected embedding pairs arising from the same digital object. ij While maximizing the similarity of, for example

number

[0049] In one trial, training the embedded neural network 114 using the contrast loss described above resulted in an improvement in the true positive rate (TPR) at a 0.1% false positive rate (FPR) from less than 40% to over 95%. In another trial, the improvement was from less than 30% to over 95%.

[0050] Figure 2 is a flowchart illustrating an exemplary process for generating a digital object with a verifiable provenance. The process in Figure 2 can be performed by one or more computer systems located in one or more locations. In this example, the process is described as being performed by the object generation and verification system 100 of Figure 1.

[0051] In step 202, request 103 is received, for example, at the first interface 104 of the object generation and verification system 100, to generate a digital object. The request is processed using the generation neural network 108 to generate a digital object 109 (step 204). The digital object 109 is processed using the watermarking neural network 112 to generate a watermarked version 113 of the digital object (step 206), which is provided, for example, for use by user 102 via a second interface (which may be the same as the first interface) (step 208). The digital object 109 is also processed using the embedding neural network 114 to generate an embedding 115 of the digital object (step 210), which is stored in the object verification database 116 (step 212).

[0052] If a digital object includes a digital audio object, generating a watermarked digital object may involve processing the digital audio object to generate a spectrogram of the digital audio object, and processing the spectrogram using a watermarking neural network to generate a watermark for the digital audio object. The watermark and spectrogram can then be combined to obtain a watermarked spectrogram, and the watermarked spectrogram can be converted into a watermarked digital object. This will be explained in more detail below.

[0053] Figure 3 is a flowchart illustrating an exemplary process for verifying the provenance of a digital object. The process in Figure 3 can be performed by one or more computer systems located in one or more locations, and is described, for example, as being performed by the object generation and verification system 100 in Figure 1.

[0054] In step 302, the query digital object 117 is received for verification. The query digital object is processed using the watermark-decoding neural network 120 to generate a watermark signal 121 for the query digital object (step 304), the watermark signal 121 indicating whether a watermark is expected to be present. The query digital object 117 is processed using the embedding neural network 114 to generate a query embedding 119 for the query digital object (step 306). The object verification database 116 is queried using the query embedding 119 to determine a set of one or more similarity scores 123 against a set of one or more stored embeddings of digital objects similar to the corresponding query digital object (step 308). The watermark signal 121 and the set of one or more similarity scores 123 are then combined to generate a verification output 125 (step 310). The verification output 125 identifies whether the query digital object 117 was generated by the generative neural network 108, and thus verifies the origin of the query digital object.

[0055] If the query object contains a digital audio object, generating a watermark signal may involve processing the digital audio object to generate a spectrogram of the digital audio object, and then processing the spectrogram using a watermark-decoding neural network to generate a watermark signal for the query digital object. This will be discussed further later.

[0056] Figure 4 shows an embodiment of a watermarking system 400 that can be used to train watermarking neural networks and watermarking decoding neural networks, such as the watermarking neural network 112 and the watermarking decoding neural network 120 described above. The system 400 may be implemented as one or more computer programs on one or more computers in one or more locations.

[0057] System 400 includes a watermarked neural network 404 configured to process a digital object, e.g., a training object 402, according to trainable parameters of the watermarked neural network, e.g., weights, in order to generate a watermarked (digital) training object 408. In one embodiment, the watermarked training object 408 is obtained by combining the output of the watermarked neural network 404 with the training object 402, e.g., by summing them element by element 406.

[0058] In some embodiments, but not necessarily, the watermarked neural network 404 includes a neural network with a U-Net architecture. Generally, the U-Net architecture is characterized by having a series of neural network layers. These layers first degrade the resolution of the representation of the digital object, for example, using pooling operations, and then improve the resolution of the representation of the digital object, for example, using upsampling operations. The neural network layers may include, for example, convolutional neural network layers or self-attention neural network layers (i.e., neural network layers incorporating a self-attention mechanism), or both. For example, skip connections may be included between layers of corresponding resolutions.

[0059] In some embodiments, the watermarked neural network 404 is configured to encode a message, such as a random number or a secret key. This may be done by combining the message, or the encoding of the message, with inputs to one or more intermediate layers of the watermarked neural network 404, for example, by summing or concatenating.

[0060] System 400 also includes a watermark-decoder neural network 420 configured to process digital objects according to watermark-decoder neural network trainable parameters so as to generate a watermark signal 422.

[0061] Generally, the watermarking signal 422 indicates whether a watermark exists in a digital object, i.e., whether the object is likely to be watermarked. The watermarking signal 422 can be as described above. In some examples, the watermarking signal 422 may include a binary signal indicating whether a watermark was detected, or a score indicating the likelihood of a watermark being present (which can be compared to a threshold to determine whether a watermark is present or not). If the watermark is a multi-bit watermark, the watermarking signal 422 may include a decoded version of the message encoded with the watermark when the watermark is present.

[0062] In some embodiments, but not necessarily, the watermark decoding neural network 420 includes a neural network with a convolutional neural network architecture.

[0063] Some watermarking approaches are not sufficiently scalable when generating a large number of digital objects. The training architecture in Figure 4 allows the embodiment of system 400 to have more watermark-decode neural network parameters than watermarking neural network parameters. Such an asymmetric architecture is advantageous because, after training, it facilitates the rapid generation of watermarked digital objects with relatively little computation. This architecture also promotes rapid training.

[0064] System 400 is configured to apply a differentiable adversarial transformation 410 to a watermarked training object 408 to generate an alternative training object that provides a watermarked digital object processed by the watermark-decoding neural network 420. An example of applying such a differentiable adversarial transformation will be described later.

[0065] System 400 includes a training engine 430 configured to jointly train a watermarking neural network 404 and a watermark decoding neural network 420.

[0066] System 400 is trained using multiple training digital objects 402. Generally, System 400 can be trained using watermarked training objects and the original training objects from which the watermarked training objects were generated.

[0067] Figure 5 is a flowchart illustrating an example of a process for jointly training the watermarking neural network 404 and the watermarking-decoding neural network 420 shown in Figure 4. The process in Figure 5 can be executed by one or more computer systems located in one or more locations. The process in Figure 5 can be executed for each of the multiple training objects 402.

[0068] In step 502, the watermarking neural network 404 processes the training object 402, x so as to generate a watermark that is applied to the training object in order to generate the watermarked training object 408, x'. For example, the watermarked digital object 408 may be generated as x' = x + g(x), where g(x) is a function applied to x by the watermarking neural network 404.

[0069] Step 504 involves alternative training objects.

number

[0070] Alternative training objects

number

number

[0071] Furthermore, the (original) training object 402 is processed using a differentiable adversarial transformation to generate a watermark signal 422 for the original training object 402, and then processed using the watermark-decode neural network 420 (step 508, performed before or after step 502). The watermark signal 422 for the original training object 402 may be denoted by d(T(x)). In some embodiments, the same transformation T(·) is applied to both the original training object and the watermarked training object. In some other embodiments, a different transformation T(·) is used.

[0072] Next, the watermarked neural network 404 and the watermarked-decoded neural network 420 are jointly trained (end-to-end) by backpropagating the gradient of the training objective function through the watermarked-decoded neural network, a differentiable adversarial transformation, and the watermarked neural network to update the watermarked-decoded neural network parameters and the watermarked neural network parameters (step 510). The gradient is obtained with respect to the watermarked-decoded neural network parameters and the watermarked neural network parameters. The watermarked-decoded neural network parameters and the watermarked neural network parameters include the layers of the neural network, such as the kernel of any convolutional layer in the network, the weight matrix of any fully connected layer in the network, and possibly the biases. The training engine 430 can determine the gradient of the objective function using the backpropagation technique, and can use the gradient to update the parameter values, for example, using any suitable gradient descent optimization algorithm, such as Adam.

[0073] Backpropagating the gradient trains the watermarking-decoding neural network and the watermarking neural network together to optimize the objective function, which measures the accuracy of the watermark signal when identifying watermarked training objects as watermarked. Training the system in this way using differentiable adversarial transformations helps the watermarking neural network learn to apply watermarks that can be robustly detected by the watermarking-decoding neural network under adversarial attack.

[0074] In some embodiments, the objective function includes a first loss term and a second loss term. The first loss term has a value that depends on i) a watermarking signal for an alternative training object and ii) a first training watermarking signal indicating that a watermarked training object is watermarked. The second loss term has a value that depends on i) a watermarking signal for the (original) training object and ii) a second training watermarking signal indicating that the training object is not watermarked. The values ​​of the first and second loss terms can be combined to obtain a value of the objective function in order to backpropagate the gradient of the objective function.

[0075] In an exemplary embodiment, the objective function includes the sum of a first term d(T(x+g(x))) that depends on the watermarking signal 422 for the alternative training object and a second term d(T(x)) that depends on the watermarking signal 422 for the original training object 402. For example, the objective function may include a loss defined as follows: l(d(T(x+g(x))),y)+l(d(T(x)),y′) In the formula, y and y' represent labels indicating the ground truth watermarking signal for the alternative training object and the ground truth watermarking signal for the original training object, respectively, where l(·) represents the mismatch between the watermarking signal 422 and the respective ground truth watermarking signals. For example, l(·) may include the cross-entropy loss. In some embodiments, y and y' may have different binary values ​​1 and 0, e.g., y=1 and y'=0. One or more other terms may also be included. For example, the objective function may include a term that encourages the watermarked training object to be similar to the training object, e.g., a loss term. Such a term may depend on a metric of similarity or difference between the training object and the watermarked training object (either before or after adversarial perturbation). In general, any suitable metric can be used. Just as an example, if the object includes images, such a term may be based on a structural similarity index (SSIM) measurement.

[0076] In some embodiments, the differentiable adversarial transformation T(·) is a random transformation. In some embodiments, applying the differentiable adversarial transformation T(·) involves applying one or more perturbations to the values ​​of the watermarked training object elements. These perturbations can be configured to change the value of the objective function so as to reduce the accuracy of the watermark signal when identifying the watermarked training object as watermarked.

[0077] As a first specific example, applying a differentiable adversarial transformation T(·) may include determining a perturbation to each value of a watermarked training object element by adjusting the value of each element based on the gradient of the objective function with respect to the watermarked training object element at each of one or more iteration steps. The same perturbation T(·) may be applied to the elements of the original training object. In a variant form, the perturbation T(·) of the adversarial transformation may instead be derived from the original training object. As an example, the perturbation applied to x′ in the i-th iteration is δ i+1 =δ i +α∇ x′ It can be obtained as l(d(T(x′)),y), where δ if necessary. i+1 This is then projected back onto the support set of elements in x'.

[0078] As a second specific example, the adversarial transformation T(·) can be parameterized by a set of one or more parameters μ. The applied adversarial transformation may be denoted as T(x, μ), where μ defines the behavior of the transformation, i.e., one or more parameters μ adjust (characterize) the transformation. The value of one or more parameters μ can be adjusted based on the gradient of the objective function with respect to the parameter(s). Thus, the differentiable adversarial transformation T(x, μ) can be determined by, at each of one or more iteration steps, determining a perturbation of each value of one or more parameters characterizing the differentiable adversarial transformation T(x, μ) by adjusting the value of each parameter based on the gradient of the objective function with respect to the parameter. For example, the perturbation applied to μ at the i-th iteration can be obtained as μ i+1 = μ i + α∇ μ l(d(T(x, μ)), y), where, if necessary, μ i+1 is projected back onto a support set of the parameter(s) μ, e.g., a “reasonable range” of the relevant parameter(s) of the transformation.

[0079] As a mere illustration of this second specific example, consider the case where the digital object includes an image and the adversarial perturbation for which the system should be robust includes changes in the image's luminance. In this case, the transformation T(x, μ) can be defined, for example, as min(max(image + μ, 0), 1). In the formula, image represents the pixel value of the image, which in this example is within the range [0, 1], and μ is a scalar value. Other random transformations can be applied in a corresponding manner, as described above.

[0080] In general, the techniques described herein are not limited to digital objects of a specific size. If the digital object includes an image, the image may be resized to a target size, for example, a size suitable for the system, for example, an image size to which the system was trained, watermarked, and then resized to its original size. The watermarking techniques described herein are robust against such transformations.

[0081] As mentioned earlier, to encourage the watermark to be imperceptible, a term can be included in the objective function, which may be, for example, a term based on a metric of the difference between a watermarked version and an unwatermarked version of a digital object. Alternatively, in some embodiments, such as when the digital object is an audio data object, the degree of adversarial perturbation (maximum allowable) can be limited to restrict the perceptibility of the watermark.

[0082] In embodiments where the training object includes an audio data object, the processing of the training object to generate a watermarked training object may include processing the training object to generate a spectrogram of the training object, processing the spectrogram using a watermarking neural network to generate a watermark for the training object, and combining the watermark and the spectrogram to obtain a watermarked spectrogram. The watermarked training object may then include a watermarked spectrogram and / or a watermarked version of the training object obtained from the watermarked spectrogram. Adversarial transformations may be applied to the watermarked spectrogram or the watermarked version of the training object obtained from the watermarked spectrogram. Optionally, the spectrogram may be adapted to the sampling rate of the audio signal from which the (digital) audio data object is derived. This will be discussed further below.

[0083] In tests involving watermarking images, training the watermarking system as described above resulted in improved watermark detection compared to previous techniques, with TPR increasing from less than 50% at 0.1% FPR to over 95%.

[0084] Figure 6 is a flowchart of another exemplary process for jointly training the watermarking neural network 404 (i.e., the watermark generating neural network) and the watermark decoding neural network 420 of the watermarking system 400 of Figure 4. The process in Figure 6 can be executed by one or more computer systems located in one or more locations. The process in Figure 6 can be executed for each of the multiple training objects 402 and is adapted for use when the training objects include audio data objects. The steps in Figure 6 do not all need to be performed in the order shown. Some steps can be performed in parallel.

[0085] The process in Figure 6 is performed on multiple audio data objects, each containing a representation of an audio signal. This process specifically involves processing the audio data objects to generate a spectrogram of the audio data objects by performing a time-frequency domain transformation on the audio signals to generate a frequency domain representation of the audio signals over a certain range of frequencies (step 602).

[0086] For example, the time-frequency domain transformation could be the Short-Time Fourier Transform (STFT). Other time-frequency domain transformations may also be used.

[0087] In general, a spectrogram can be an image representing a time-frequency transformation. More specifically, a spectrogram can include a representation of an audio data item with time on one axis, e.g., the horizontal axis, and frequency on the other axis, e.g., the vertical axis. Pixel positions in the image along the time axis can represent time positions within the audio data object. Positions along the frequency axis can represent the frequency at that point in time. Pixel values, e.g., brightness or color values, can represent components of the audio signal at that point in time and frequency, e.g., magnitude and / or phase of the audio signal. In some time-frequency transformations, the components of the audio signal are represented by complex numbers. Therefore, in general, a spectrogram can represent the changing spectrum of an audio signal as it changes over time.

[0088] The spectrogram (image) is processed using a watermark-generating neural network, i.e., a watermark-imprinting neural network 404, to generate a watermark for the audio data object (step 604), and in embodiments, the watermark is also in the form of an image.

[0089] The watermark (image) is combined with the spectrogram (image), for example, by adding it to the spectrogram (image) to obtain a watermarked spectrogram (image) (step 606).

[0090] The transformation, particularly the adversarial transformation, is applied to one or both of the watermarked spectrogram and the watermarked spectrogram, i) a watermarked version of the audio data object obtained by transforming the watermarked spectrogram into a watermarked version of the audio data object, and ii) a watermarked spectrogram, so as to perturb the watermark and generate a perturbed watermarked data object (step 608). That is, this method may include, but is not required, transforming the watermarked spectrogram into a watermarked version of the audio data object (because the watermark-decoding neural network processes the spectrogram).

[0091] Generally, converting a spectrogram, such as a watermarked spectrogram, to (time-domain) audio data, such as a watermarked version of an audio data object, involves applying the inverse transform of the time-frequency-domain transform, i.e., a frequency-time-domain transform. The specific inverse transform depends on which time-frequency-domain transform was used. For example, an inverse STFT may be performed.

[0092] Adversarial transformations can be applied in the time domain to a watermarked version of an audio data object, or in the frequency domain to a watermarked spectrum, or both. In general, an adversarial transformation can be any perturbation that alters the audio data object, making watermark detection particularly more difficult. If the adversarial transformation is differentiable, it can be useful for training. Some examples of adversarial transformations of audio data objects are discussed later.

[0093] Adversarial transformations are also applied to one or both of the audio data object and the spectrogram of the audio data object to generate a perturbed data object (step 610). In this case as well, adversarial transformations can be applied to the time domain or the frequency domain.

[0094] The perturbed watermarked data object is processed by a watermark decoding neural network to generate a first (trained) watermark signal indicating whether or not the perturbed watermarked data object is predicted to be watermarked (step 612). For example, the first watermark signal can predict (identify) when the perturbed watermarked data object was watermarked.

[0095] The perturbation data object is processed by a watermark decoding neural network to generate a second (trained) watermark signal indicating whether or not the perturbation data object is predicted to be watermarked (step 614). For example, the second watermark signal can predict (identify) when the perturbation data object was watermarked.

[0096] The watermark decoding neural network and the watermark generating neural network are trained together (end-to-end) using the first watermarking signal and the second watermarking signal (step 616). Specifically, the watermark decoding neural network and the watermark generating neural network are trained to distinguish between perturbed watermarked data objects and perturbed data objects.

[0097] In this embodiment, jointly training the watermark decoding neural network and the watermark generating neural network involves backpropagating the gradient of the classification-based objective function to both the watermark decoding neural network and the watermark generating neural network.

[0098] In one embodiment, the objective function has a value that depends on classifying a first watermark signal as indicating that a perturbed watermarked data object is watermarked, and classifying a second watermark signal as indicating that a perturbed data object is not watermarked. In one embodiment, the objective function includes a cross-entropy loss for accurately classifying each signal.

[0099] In general, adversarial transformations can be selected as transformations for which watermarking should be robust. In a typical embodiment, multiple different adversarial transformations may be applied, for example, by using different adversarial transformations for different audio data objects.

[0100] For audio data objects, exemplary adversarial transformations may include modifications to pitch / time / speed or frequency, e.g., filtering or masking; modifications to amplitude; modifications to noise levels, e.g., noise addition or signal removal; and the application of audio compression / decompression. Adversarial transformations can be applied to the audio data object and / or the spectrogram of the audio data object.

[0101] As described above, the watermarking neural network 404, or watermarking neural network, can include a U-net. This makes it easier to apply the watermarking neural network to a continuous audio signal, in which case the watermarking neural network slides or steps along the spectrogram of the audio signal.

[0102] As mentioned above, the watermark decoding neural network 420 may include a convolutional neural network.

[0103] Such embodiments facilitate processing each time step of an audio signal using a watermark decoding neural network to generate an output that predicts whether or not a watermark is present at each time step.

[0104] In one embodiment, each audio data object includes a digital representation of an audio signal obtained by sampling the audio signal in the time domain at a signal sampling rate. The method may then include fitting the spectrograms to the sampling rates of the audio signals so as to compensate for the different sampling rates of different audio data objects.

[0105] In this embodiment, a time-frequency domain transformation is performed on a series of frames of an audio data object to generate a spectrogram. Each such frame defines a time window for the audio data object, containing multiple audio signal samples. Fitting the spectrogram to the sampling rate of the audio signal then may involve varying the number of audio signal samples within a frame so that each frame has the same duration for different sampling rates. Details of a specific exemplary process for this will be described later.

[0106] Figure 7A is a flowchart illustrating an exemplary process for watermarking an audio signal. The process in Figure 7A can be performed by one or more computer systems located in one or more locations, for example, by a trained watermarking neural network such as the watermarking neural network 404 in Figure 4, after training as described above.

[0107] The audio signal is processed to generate a spectrogram of the audio signal (step 702), specifically by performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal over a certain range of frequencies.

[0108] The spectrogram is processed using a trained watermarking neural network, for example, a watermarking neural network trained as described above, to generate a watermark for the audio signal (step 704).

[0109] Next, the watermark and spectrogram are combined, for example, by addition to obtain a watermarked spectrogram (step 706), and the watermarked spectrogram is converted into a watermarked version of the audio signal (step 708).

[0110] Generally, an audio signal includes a digital representation of an audio signal obtained by sampling the audio signal in the time domain at the signal sampling rate. Optionally, the method may include fitting a spectrogram to the sampling rate of the audio signal (step 704).

[0111] In this embodiment, a time-frequency domain transformation is performed on a series of frames of an audio signal to generate a spectrogram, where each frame defines a time window on the audio signal containing multiple audio signal samples. Fitting the spectrogram to the sampling rate of the audio signal may involve varying the number of audio signal samples within a frame so that each frame has the same duration for different sampling rates.

[0112] In this embodiment, the watermark generating neural network is trained at a sampling rate (or maximum sampling rate) called the reference sampling rate. Generally, when a time-frequency transformation is performed on an audio signal, a frequency-domain representation of the audio signal is generated for a certain range of frequencies. If the sampling rate of the audio signal is higher than the reference sampling rate, a portion of the frequency-domain representation corresponding to frequencies up to the reference maximum frequency is selected, where the reference maximum frequency represents the reference sampling rate, i.e., corresponds to the reference sampling rate. For example, the reference maximum frequency could be the highest frequency that can be represented when sampling at the reference sampling rate, for example, according to the Nyquist criterion. Because the sampling rate of the audio signal is higher than the reference sampling rate, the reference maximum frequency is within the frequency range of the frequency-domain representation of the audio signal.

[0113] This process can generate a spectrogram and a watermark of an audio signal from a time-frequency domain transformation (only) for frequencies up to a reference maximum frequency. The watermark and spectrogram can be combined to obtain a watermarked spectrogram, and the watermarked spectrogram and a portion of the frequency domain representation of frequencies above the reference maximum frequency can be combined to determine a combined spectrogram that includes frequencies not represented in the watermarked spectrogram. The relevant portion of the frequency domain representation includes frequencies above the reference maximum frequency within the range of frequencies from the time-frequency domain transformation. The combined spectrogram can then be converted to time-domain audio to obtain a watermarked version of the audio signal.

[0114] If the sampling rate is lower than the reference sampling rate on which the watermarking neural network was trained, missing frequencies, i.e., can be represented by a watermarked spectrogram, but frequencies that are not available from the frequency-domain audio signal may be represented by zero.

[0115] For example, a time-frequency domain conversion of an audio signal can generate a frequency domain representation of the audio signal for a certain range of frequencies. This frequency domain representation can then be padded with zeros up to a reference maximum frequency for frequencies exceeding the maximum frequency defined by the sampling rate, thereby generating a padded frequency domain representation of the audio signal. Subsequently, the spectrogram and watermark of the audio signal can be generated from the padded frequency domain representation of the audio signal.

[0116] Figure 7B illustrates the use of a watermarking neural network 404 trained to watermark a digital object, such as a digitized audio signal. The digital object 702 is provided as input to the trained watermarking neural network 404, which processes the digital object 702 to generate a watermark 704 of the digital object, and then combines the watermark with the digital object 702 to obtain a watermarked version 706 of the digital object.

[0117] Figure 8A is a flowchart illustrating an exemplary process for identifying the presence or absence of a watermark in an audio signal. The process in Figure 8A can be performed by one or more computer systems located in one or more locations, for example, by the watermark decoding neural network 420 in Figure 4 after training as described above.

[0118] The audio signal is processed to generate a spectrogram of the audio signal (step 802), specifically by performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal for a certain range of frequencies. The spectrogram is then processed using a trained watermark decoding (watermark identification) neural network to generate a watermark signal for the audio signal (step 804). The watermark signal indicates whether or not the audio signal is predicted to be watermarked.

[0119] Optionally, if the sampling rate is higher than the reference sampling rate above, the process may include selecting a portion of the frequency domain representation for frequencies up to the reference maximum frequency within the frequency range representing the reference sampling rate above. The spectrogram of the audio signal (for processing using a watermark-decoded neural network) can be generated from the time-frequency transformation for frequencies up to the reference maximum frequency (only).

[0120] If the sampling rate is lower than the reference sampling rate described above, the process may include padding the frequency domain representation of the audio signal with zeros for frequencies exceeding the maximum frequency defined by the sampling rate up to the reference maximum frequency representing the reference sampling rate, in order to generate a padded frequency domain representation of the audio signal. The spectrogram of the audio signal and a watermark of the audio signal can then be generated from the padded frequency domain representation of the audio signal.

[0121] Figure 8B illustrates the use of a trained watermark decoding neural network 420 to predict whether a digital object, such as a digitized audio signal, is watermarked. The digital object 802 is provided as input to the trained watermark decoding neural network 420, which processes the digital object 802 to generate a watermark signal 804 for the digital object, which predicts whether the digital object 802 is watermarked, specifically as described above.

[0122] Next, we describe one exemplary technique for fitting a spectrogram to the sampling rate of an audio signal. Here, the time-frequency domain transformation is the Short-Time Fourier Transform (STFT). The STFT transform has a frame length W that defines the number of samples in a frame and a frame step S that corresponds to the number of samples to shift between frames. For example, an STFT with parameters W=1024 and S=512 results in a spectrogram containing the calculated frequency response for samples 0 to 1024 in the first frame (t=0), and for samples 512 to 1576 in the second frame (t=1), and so on.

[0123] The STFT transforms a time-domain signal s, a tensor of size T, into a time-frequency representation, i.e., a spectrogram I, a tensor of size [t,F]. The second dimension of the spectrogram corresponds to frequency. In a given frame (fixed t), the content of the spectrogram I[t,F] is defined by the convolution between a window of selected samples, i.e., the frame, and frequency-dependent filters. These frequency-dependent filters are placed in increasing frequency order, for example, a constant, one period across the window, two periods across the window, and so on, up to a maximum of (n / 2) periods across the window, where n is the number of samples in the frame.

[0124] It is useful for audio signal processing to be invariant with respect to the sampling rate. For example, it is useful if the same spectrogram Ia is obtained from an audio signal encoded at a 24kHz sample rate (sample_rate_a), and the same spectrogram Ib is obtained from an audio signal encoded at a 48kHz sample rate (sample_rate_b), i.e., Ia = Ib.

[0125] This can be achieved by modifying the spectrogram parameters. A given frame of spectrogram Ia contains W samples corresponding to a duration of (W / sample_rate_a) seconds. If this period is encoded at a sample rate of sample_rate_b, it represents sample_rate_b*(W / sample_rate_a) samples. Therefore, to obtain the equivalent W' required to have frames of matching length, the frame length can be multiplied by the ratio of the sampling rates. Regardless of the sampling rate, the same logic is applied to frame steps, S, to obtain the same time difference between adjacent frames.

[0126] The available frequencies in the spectrogram depend on the number of points within the window, and optionally, the process described above can be adapted to these frequencies. We consider how the system model adapts to different sample rates, assuming a trained reference sample rate. If the sample rate is higher than the reference sample rate, there are more points within the window than necessary; that is, it is possible to compute all the frequencies that exist when using the reference sample rate, plus some additional frequencies. In this case, the process can compute all possible frequencies and divide them into two parts: one is the frequencies the model can use, which are watermarked and passed to the decoder, etc., and the other is the frequencies the model cannot process, which are left as they are. When the (time-domain) audio signal is reconstructed using frequency-time-domain transformation, the two sets of frequencies can be reconnected to avoid the loss of any portion of the signal encoded at higher frequencies.

[0127] If the sample rate is higher than the reference sample rate, the number of points in the window will be less than necessary, and the spectrogram can only be calculated for the reduced set of frequencies. In this case, the unavailable frequencies can be represented by zeros.

[0128] In a test of watermarking audio data objects, training the watermarking system for audio data objects as described above can provide imperceptible watermarking with over 98% TPR at 0.1% FPR, even when adversarial perturbations are applied to the audio.

[0129] The training data for the systems and neural networks described herein can simply include a set of digital objects of the desired type to be watermarked, such as text, images, audio, or a combination thereof. Numerous publicly available datasets exist for images, audio, etc., such as ImageNet for images, AudioSet for audio, or versions of the Common Crawl dataset for text. The trainable parameters of the watermarking neural network (watermark generating neural network) and the watermark decoding neural network, such as the number of weights, may vary depending, for example, the size of the digital objects being processed, and their architecture may also vary. Techniques for determining the number of training data items to use are well known, for example, based on the system's performance against a given set of data items and / or based on monitoring a desired value, such as the loss function.

[0130] This specification uses the term “configured” in relation to systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action.

[0131] The subject matter and functional embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium, which are executed by or control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded in artificially generated propagating signals, such as machine-generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information to be transmitted to a suitable receiving device for execution by a data processing device.

[0132] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or multiple computers. A device may be, or further may be, a special-purpose logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, a device may include code that constructs the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0133] Computer programs, which may also be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program in question, in a part of a file that holds one or more scripts stored in other programs or data, such as a markup language document, or in a series of collaborative files, such as a file that holds one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one place, or distributed across multiple locations and interconnected by a data communication network.

[0134] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0135] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0136] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Typical elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or integrated into dedicated logic circuits. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operationally connected to receive data from or transfer data to or both of these mass storage devices. However, a computer is not required to have such devices. Furthermore, a computer can be integrated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0137] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks, as well as all forms of non-volatile memory, media, and memory devices.

[0138] Embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device that allows the user to input into the computer, such as a mouse or trackball, in order to provide user interaction. Other types of devices can also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving response messages from the user.

[0139] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for handling the general and computationally intensive parts of machine learning training or production, i.e., inference workloads.

[0140] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0141] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, backend components such as a data server, or middleware components such as an application server, or frontend components such as a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communications of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0143] While this specification includes details of many specific embodiments, these should not be interpreted as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0144] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0145] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield the desired results. As an example, the process shown in the accompanying diagram does not necessarily require to be performed in the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0146] Further aspects of the present invention are defined in the following clauses.

[0147] 1. A computer-implemented method for training a watermarking system including a watermark-generating neural network and a watermark-decoding neural network, the method comprising: processing multiple audio data objects, each containing a representation of an audio signal, such that the audio data objects are generated to produce a spectrogram of the audio data objects; processing the spectrogram using the watermark-generating neural network to produce a watermark of the audio data objects; combining the watermark and the spectrogram to obtain a watermarked spectrogram; and converting the watermarked spectrogram into a watermarked version of the audio data object, thereby perturbing the watermark to produce a perturbed watermarked data object, thereby obtaining either or both of the watermarked version of the audio data object and the watermarked spectrogram. A method comprising: applying an adversarial transformation; applying an adversarial transformation to one or both the audio data object and the spectrogram of the audio data object in order to generate a perturbed data object; processing the perturbed watermarked data object using the watermark-decoder neural network in order to generate a first watermark signal indicating whether the perturbed watermarked data object is predicted to be watermarked; processing the perturbed data object using the watermark-decoder neural network in order to generate a second watermark signal indicating whether the perturbed data object is predicted to be watermarked; and jointly training the watermark-decoder neural network and the watermark-generating neural network using the first watermark signal and the second watermark signal to distinguish between the perturbed watermarked data object and the perturbed data object.

[0148] 2. The method according to Clause 1, wherein jointly training the watermark decoding neural network and the watermark generating neural network involves backpropagating the gradient of a classification-based objective function having values ​​that depend on classifying the perturbed watermarked data object as watermarked and classifying the perturbed watermarked data object as not watermarked.

[0149] 3. The method according to Clause 1 or 2, comprising applying the adversarial transformation to the spectrogram of the audio data object to generate the perturbation data object, wherein applying the adversarial transformation involves changing one or more of the pitch, speed, frequency components, or noise levels in the audio data object or the spectrogram of the audio data object.

[0150] 4. The method according to any one of the claims 1 to 3, wherein each audio data object includes a digital representation of the audio signal obtained by sampling the audio signal in the time domain at a signal sampling rate, and the method further includes adapting the spectrogram to the sampling rate of the audio signal so as to compensate for different sampling rates of different audio data objects.

[0151] 5. The method according to Clause 4, wherein processing the audio data object to generate a spectrogram of the audio data object includes performing a time-frequency domain transformation on a series of frames of the audio data object to generate the spectrogram, each frame defining a time window on the audio data object containing a plurality of audio signal samples, and fitting the spectrogram to the sampling rate of the audio signal includes varying the number of audio signal samples in a frame so that each frame has the same duration for different sampling rates.

[0152] 6. A computer-implementable method for watermarking an audio signal, the method comprising: processing the audio signal to generate a spectrogram of the audio signal; processing the spectrogram using a watermark-generating neural network to generate a watermark for the audio signal; combining the watermark and the spectrogram to obtain a watermarked spectrogram; and converting the watermarked spectrogram into a watermarked version of the audio signal.

[0153] 7. The method according to Clause 6, wherein the audio signal includes a digital representation of the audio signal obtained by sampling the audio signal in the time domain at a signal sampling rate, and the method further includes fitting the spectrogram to the sampling rate of the audio signal.

[0154] 8. The method according to Clause 7, wherein processing the audio signal to generate the spectrogram of the audio signal comprises performing a time-frequency domain transformation on a series of frames of the audio signal to generate the spectrogram, each frame defining a time window on the audio signal containing a plurality of audio signal samples, and fitting the spectrogram to the sampling rate of the audio signal comprises varying the number of audio signal samples in a frame such that each frame has the same duration for different sampling rates.

[0155] 9. The method according to any one of the claims 7 to 8, further comprising: performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal for a range of frequencies when the sampling rate is higher than the reference sampling rate on which the watermark generating neural network was trained; selecting a portion of the frequency domain representation for frequencies up to a reference maximum frequency within the range of frequencies representing the reference sampling rate; generating the spectrogram of the audio signal and the watermark for the audio signal from the time-frequency domain transformation for frequencies up to the reference maximum frequency; combining the watermark and the spectrogram to obtain the watermarked spectrogram; combining the watermarked spectrogram with a portion of the frequency domain representation for frequencies above the reference maximum frequency within the frequency range to determine a composite spectrogram; and converting the composite spectrogram into the watermarked version of the audio signal.

[0156] 10. The method according to any one of the claims 7 to 8, further comprising: performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal for a range of frequencies when the sampling rate is lower than the reference sampling rate on which the watermark generating neural network was trained; padding the frequency domain representation of the audio signal with zeros for frequencies exceeding the maximum frequency defined by the sampling rate up to a reference maximum frequency representing the reference sampling rate to generate a padded frequency domain representation of the audio signal; and generating the spectrogram of the audio signal and the watermark for the audio signal from the padded frequency domain representation of the audio signal.

[0157] 11. A computer-implementable method for identifying the presence or absence of a watermark in an audio signal, the method comprising: processing the audio signal to generate a spectrogram of the audio signal; and processing the spectrogram using a watermark-decoding neural network to generate a watermark signal for the audio signal, wherein the watermark signal predicts whether the audio signal is watermarked or not.

[0158] 12. The method according to clause 11, further comprising: performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal for a range of frequencies when the sampling rate is higher than the reference sampling rate on which the watermark-decoding neural network was trained; selecting a portion of the frequency domain representation for frequencies up to a reference maximum frequency within the range of frequencies representing the reference sampling rate; and generating the spectrogram of the audio signal from the time-frequency domain transformation for frequencies up to the reference maximum frequency for processing using the watermark-decoding neural network.

[0159] 13. The method of Clause 11 or 12, further comprising: performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal for a range of frequencies when the sampling rate is lower than the reference sampling rate on which the watermark generating neural network was trained; padding the frequency domain representation of the audio signal with zeros for frequencies exceeding the maximum frequency defined by the sampling rate up to a reference maximum frequency representing the reference sampling rate to generate a padded frequency domain representation of the audio signal; and generating the spectrogram of the audio signal and the watermark for the audio signal from the padded frequency domain representation of the audio signal.

[0160] 14. A computer-implemented method for training a watermarking system for watermarking digital objects, the watermarking system comprising: a watermarking neural network configured to process digital objects according to watermarking neural network parameters to generate watermarked digital objects; and a watermark decoding neural network configured to process watermarked digital objects according to watermark decoding neural network parameters to generate watermarked signals, the method comprising: processing each of a plurality of training objects using the watermarking neural network to generate a watermarked training object; applying a differentiable adversarial transformation to the watermarked training object to generate an alternative training object; and the alternative training A method comprising processing the alternative training objects using the watermark-decoder neural network to generate the watermark signal for the training objects, and backpropagating the gradient of the objective function through the watermark-decoder neural network, the differentiable adversarial transform, and the watermark-encoder neural network to update the watermark-decoder neural network parameters and the watermark-encoder neural network parameters, and to train the watermark-decoder neural network and the watermark-encoder neural network together to optimize the objective function, wherein the gradient is obtained with respect to the watermark-decoder neural network parameters and the watermark-encoder neural network parameters, and the objective function measures the accuracy of the watermark signal in identifying the watermarked training objects as watermarked.

[0161] 15. The method according to Clause 14, wherein the watermarked training object comprises a plurality of watermarked training object elements, and applying the differentiable adversarial transformation comprises applying one or more perturbations to the values ​​of the watermarked training object elements, the perturbations changing the value of the objective function such that the accuracy of the watermark signal is reduced when identifying the watermarked training object as watermarked.

[0162] 16. The method according to Clause 14 or 15, wherein the differentiable adversarial transformation has one or more parameters that adjust the transformation, and the method further comprises iteratively determining a perturbation of each of the parameters to each of the values ​​of the parameters by adjusting each of the values ​​of the parameters based on the gradient of the objective function with respect to the parameters in one or more iteration steps.

[0163] 17. The method according to any one of the clauses 14 to 16, wherein the objective function includes a first loss term and a second loss term, and the method further includes processing the training object using the watermark decoding neural network to generate the watermark signal for the training object; determining a value of the first loss term depending on the watermark signal of the alternative training object and a first training watermark signal indicating that the watermarked training object is watermarked; determining a value of the second loss term depending on the watermark signal of the training object and a second training watermark signal indicating that the watermarked training object is not watermarked; and combining the value of the first loss term and the value of the second loss term to obtain a value of the objective function that backpropagates the gradient of the objective function.

[0164] 18. The digital object and the training object are each the method described in any one of the clauses 14 to 17, including images.

[0165] 19. The method described in any one of the clauses 14 to 18, wherein the digital object and the training object each include a digital audio object.

[0166] 20. The method according to Clause 19, wherein processing the training object using the watermarking neural network to generate a watermarked training object comprises processing the training object to generate a spectrogram of the training object, processing the spectrogram using the watermarking neural network to generate a watermark for the training object, and combining the watermark and the spectrogram to obtain a watermarked spectrogram, wherein the watermarked training object comprises the watermarked spectrogram, or a watermarked version of the training object obtained from the watermarked spectrogram, and applying the differentiable adversarial transformation to the watermarked training object to generate the alternative training object comprises applying the differentiable adversarial transformation to the watermarked spectrogram, or the watermarked version of the training object obtained from the watermarked spectrogram.

[0167] 21. The method according to Clause 20, wherein the training object includes a digital representation of an audio signal obtained by sampling the audio signal in the time domain at a signal sampling rate, and the method further includes adapting the spectrogram to the sampling rate of the audio signal so as to compensate for different sampling rates of different training objects.

[0168] 22. The method according to any one of the clauses 14 to 21, wherein the watermarking neural network includes a neural network having a U-Net architecture, and / or the watermarking decoding neural network includes a neural network having a convolutional neural network architecture.

[0169] 23. The method according to any one of the clauses 14 to 22, wherein the watermark decoding neural network has more trainable parameters than the watermarking neural network.

[0170] 24. The watermarking signal comprising zero, one, or more watermark message bits, as described in any one of clauses 14 to 23.

[0171] 25. A computer-implemented method for watermarking a digital object, the method comprising: processing the digital object or data representing the digital object using a watermarking neural network trained by the method described in any one of Clauses 14 to 24 to generate a watermark for the digital object; and generating a watermarked version of the digital object from a combination of the watermark and the digital object or data representing the digital object.

[0172] 26. A computer-implemented method for identifying the presence or absence of a watermark in a digital object, the method comprising processing the digital object or data representing the digital object using a watermark-decoding neural network trained by the method described in any one of clauses 14 to 24 to generate a watermark signal for the digital object, the watermark signal predicting whether the digital object is watermarked or not.

[0173] 27. A computer-implemented method for verifying the origin of a digital object, the method comprising maintaining an object verification system, the object verification system comprising: a first interface for receiving a digital object or a request to generate a digital object; a second interface for providing a watermarked digital object for use; an embedding neural network configured to process the digital object and generate an embedding of the digital object; and an object verification database configured to store at least the embedding of the digital object, the method further comprising: receiving a query digital object for verification; processing the query digital object using a watermark-decode neural network to generate a watermark signal for the query digital object; processing the query digital object using the embedding neural network to generate a query embedding of the query digital object; querying the object verification database using the query embedding to determine one or more sets of similarity scores for one or more stored sets of embeddings of corresponding digital objects similar to the query digital object; and verifying the origin of the query digital object based on a combination of the watermark signal and the one or more sets of similarity scores.

[0174] 28. The method according to Clause 27, wherein the query object includes a digital audio object, and processing the query digital object using the watermark-decode neural network to generate a watermark signal for the query digital object, processing the digital audio object to generate a spectrogram of the digital audio object, and processing the spectrogram using the watermark-decode neural network to generate the watermark signal for the query digital object.

[0175] 29. The object verification system is an object generation and verification system, the request includes a request to generate the digital object, and maintaining the object verification system (100) further includes maintaining a generative neural network (108) configured to process the request to generate the digital object (109) in accordance with the request, the method further includes receiving a request to generate the digital object (103), processing the request using the generative neural network to generate the digital object (109), and using the watermarking neural network to generate the watermarked digital object (113). The method according to Clause 27 or 28, comprising processing a subject (109), providing the watermarked digital object for use, processing the digital object (109) using the embedding neural network to generate the embedding (115) of the digital object, storing the embedding (115) of the digital object in the object verification database, and verifying the origin of the query digital object as generated by the object generation and verification system, provided that the query embedding (119) matches the embedding (115) of the digital object stored in the object verification database.

[0176] 30. The method according to Clause 29, wherein the digital object includes a digital audio object, and processing the digital object using the watermarking neural network to generate the watermarked digital object includes processing the digital audio object to generate a spectrogram of the digital audio object, processing the spectrogram using a watermarking neural network to generate a watermark for the digital audio object, combining the watermark and the spectrogram to obtain a watermarked spectrogram, and converting the watermarked spectrogram into the watermarked digital object.

[0177] 31. The method according to clause 29 or 30, further comprising storing the digital object in the object verification database and using the stored digital object to detect attempts to remove a watermark from the query digital object.

[0178] 32. The method according to any one of the clauses 27 to 31, wherein the watermarking neural network and the watermark decoding neural network are jointly trained to generate the watermarked signal under adversarial perturbations of the watermarked digital object.

[0179] 33. A computer-implemented method for generating a digital object having a verifiable provenance, the method comprising maintaining an object generation and verification system, the object generation and verification system comprising: a first interface for receiving requests to generate a digital object; a generation neural network configured to process the requests to generate a digital object in accordance with the requests; a watermarking neural network configured to process the digital object to generate a watermarked digital object; a second interface for providing the watermarked digital object for use; an embedding neural network configured to process the digital object to generate an embedding of the digital object; and an object verification database configured to store at least the embedding of the digital object, the method further comprising receiving requests to generate the digital object and generating the digital object in accordance with the requests The process includes processing the request using the generative neural network, processing the digital object using the watermarking neural network to generate a watermarked digital object, providing the watermarked digital object for use, processing the digital object using the embedding neural network to generate the embedding of the digital object, and storing the embedding of the digital object in the object verification database, wherein the provenance of the digital object includes generating query embeddings for the query digital object using the embedding neural network, querying the object verification database using the query embeddings to determine one or more sets of similarity scores for one or more stored sets of embeddings of corresponding digital objects similar to the query digital object, and a watermark signal indicating the watermark of the query digital object.A method for verifying the origin of a query digital object based on a combination of one or more sets of similarity scores.

[0180] 34. The method according to Clause 33, wherein the digital object includes a digital audio object, the query object includes a digital audio object, and processing the query digital object using the watermark-decode neural network to generate a watermark signal for the query digital object includes processing the digital audio object to generate a spectrogram of the digital audio object, and processing the spectrogram using the watermark-decode neural network to generate a watermark signal for the query digital object, and processing the digital object using the watermark neural network to generate a watermark digital object includes processing the digital audio object to generate a spectrogram of the digital audio object, processing the spectrogram using a watermark-generating neural network to generate a watermark for the digital audio object, combining the watermark and the spectrogram to obtain a watermark spectrogram, and converting the watermark spectrogram into the watermark digital object.

[0181] 35. The digital object is an image, as described in any one of the provisions of Clauses 27 to 34.

[0182] 36. The method described in any one of the clauses 27 to 35, wherein the digital object includes a digital audio object.

[0183] 37. The method according to any one of the clauses 27 to 36, wherein the watermarking neural network includes a neural network having a U-Net architecture, and / or the watermarking-decoding neural network includes a neural network having a convolutional neural network architecture.

[0184] 38. The method according to any one of the clauses 27 to 37, wherein the watermark decoding neural network has more trainable parameters than the watermarking neural network.

[0185] 39. The watermarking signal comprising zero, one, or more watermark message bits, as described in any one of clauses 27 to 38.

[0186] 40. One or more computer storage media for storing instructions, wherein, when executed by one or more computers, the instructions cause one or more computers to perform the operations of each of the methods described in any one of the clauses 1 to 39.

[0187] 41. A system comprising one or more computers and one or more storage devices communicably coupled to the one or more computers, wherein the one or more storage devices, when executed by the one or more computers, store instructions causing the one or more computers to perform the operation of any one of the methods described in any one of the clauses 1 to 39.

Claims

1. A computer-implemented method for training a watermarking system that includes a watermark generation neural network and a watermark decoding neural network, Each of the multiple audio data objects contains a representation of an audio signal, Processing the audio data object to generate a spectrogram of the audio data object, The spectrogram is processed using the watermarking neural network to generate a watermark for the audio data object. Combining the watermark and the spectrogram to obtain a watermarked spectrogram, Applying an adversarial transformation to either or both of the watermarked version of the audio data object and the watermarked spectrogram, which are obtained by converting the watermarked spectrogram into a watermarked version of the audio data object, in order to generate a perturbed watermarked data object by perturbing the watermark, Applying adversarial transformations to either or both the audio data object and the spectrogram of the audio data object in order to generate a perturbed data object, The perturbed watermarked data object is processed using the watermark decoding neural network to generate a first watermark signal indicating whether or not the perturbed watermarked data object is predicted to be watermarked, The perturbation data object is processed using the watermark decoding neural network to generate a second watermark signal indicating whether or not the perturbation data object is predicted to be watermarked, The watermark decoding neural network and the watermark generation neural network are jointly trained using the first watermark signal and the second watermark signal to distinguish between the perturbed watermarked data object and the perturbed data object. Methods that include...

2. The method according to claim 1, wherein jointly training the watermark decoding neural network and the watermark generating neural network involves backpropagating the gradient of a classification-based objective function having values ​​that depend on classifying the perturbed watermarked data object as watermarked by the first watermark signal and classifying the perturbed watermarked data object as not watermarked by the second watermark signal.

3. The method according to claim 1 or 2, comprising applying the adversarial transformation to the spectrogram of the audio data object so as to generate the perturbation data object, wherein applying the adversarial transformation involves changing one or more of the pitch, speed, frequency components, or noise levels in the audio data object or in the spectrogram of the audio data object.

4. Each audio data object includes a digital representation of the audio signal obtained by sampling the audio signal in the time domain at a certain signal sampling rate, and the method further includes The method according to any one of claims 1 to 3, comprising adapting the spectrogram to the sampling rate of the audio signal so as to compensate for different sampling rates of different audio data objects.

5. Processing the audio data object in order to generate a spectrogram of the audio data object is, The process includes performing a time-frequency domain transformation on a series of frames of the audio data object to generate the spectrogram, wherein each frame defines a time window on the audio data object containing a plurality of audio signal samples. The method according to claim 4, wherein fitting the spectrogram to the sampling rate of the audio signal involves varying the number of audio signal samples in a frame so that each frame has the same duration for different sampling rates.

6. A computer-implemented method for watermarking an audio signal, wherein the method is Processing the audio signal to generate a spectrogram of the audio signal, The spectrogram is processed using a watermarking neural network to generate a watermark for the audio signal, Combining the watermark and the spectrogram to obtain a watermarked spectrogram, Converting the watermarked spectrogram into a watermarked version of the audio signal, Methods that include...

7. The audio signal includes a digital representation of the audio signal obtained by sampling the audio signal in the time domain at a certain signal sampling rate, and the method further includes The method according to claim 6, comprising fitting the spectrogram to the sampling rate of the audio signal.

8. Processing the audio signal to generate the spectrogram of the audio signal is, The process includes performing a time-frequency domain transformation on a series of frames of the audio signal to generate the spectrogram, wherein each frame defines a time window on the audio signal containing a plurality of audio signal samples. The method according to claim 7, wherein fitting the spectrogram to the sampling rate of the audio signal involves varying the number of audio signal samples in a frame so that each frame has the same duration for different sampling rates.

9. When the sampling rate is higher than the reference sampling rate on which the watermark generation neural network was trained, Performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal over a certain frequency range, Selecting a portion of the frequency domain representation for frequencies up to the reference maximum frequency within the range of frequencies representing the reference sampling rate, From the time-frequency domain conversion for frequencies up to the aforementioned reference maximum frequency, the spectrogram of the audio signal and the watermark for the audio signal are generated. Combining the watermark and the spectrogram in order to obtain the aforementioned watermarked spectrogram, To determine a composite spectrogram, the watermarked spectrogram is combined with a portion of the frequency domain representation for frequencies exceeding the reference maximum frequency within the range of frequencies, Converting the composite spectrogram into the watermarked version of the audio signal, The method according to claim 7 or claim 8, further comprising:

10. When the sampling rate is lower than the reference sampling rate on which the watermark generation neural network was trained, Performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal over a certain frequency range, To generate a padded frequency domain representation of the audio signal, the frequency domain representation of the audio signal is padded with zeros for frequencies exceeding the maximum frequency defined by the sampling rate and up to a reference maximum frequency representing the reference sampling rate, From the padded frequency domain representation of the audio signal, the spectrogram of the audio signal and the watermark for the audio signal are generated. The method according to any one of claims 7 to 9, further comprising:

11. A computer-implemented method for identifying the presence or absence of a watermark in an audio signal, wherein the method is: Processing the audio signal to generate a spectrogram of the audio signal, The spectrogram is processed using a watermark decoding neural network to generate a watermark signal for the audio signal, Includes, The watermark signal is a method for predicting whether or not the audio signal is watermarked.

12. When the sampling rate is higher than the reference sampling rate on which the watermark decoding neural network was trained, Performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal over a certain frequency range, Selecting a portion of the frequency domain representation for frequencies up to the reference maximum frequency within the range of frequencies representing the reference sampling rate, From the time-frequency domain transformation for frequencies up to the aforementioned reference maximum frequency, the spectrogram of the audio signal is generated for processing using the watermark decoding neural network, The method according to claim 11, further comprising:

13. When the sampling rate is lower than the reference sampling rate on which the watermark generation neural network was trained, Performing a time-frequency domain transformation on the audio signal to generate a frequency domain representation of the audio signal over a certain frequency range, To generate a padded frequency domain representation of the audio signal, the frequency domain representation of the audio signal is padded with zeros for frequencies exceeding the maximum frequency defined by the sampling rate and up to a reference maximum frequency representing the reference sampling rate, From the padded frequency domain representation of the audio signal, the spectrogram of the audio signal and the watermark for the audio signal are generated. The method according to claim 11 or claim 12, further comprising:

14. A computer-implemented method for verifying the provenance of a digital object, wherein the digital object includes a digital audio object, and the method is This includes maintaining an object validation system, the object validation system is A first interface that receives a digital object, or a request to generate a digital object, A second interface is provided for the use of watermarked digital objects, An embedding neural network configured to process the digital object and generate an embedding of the digital object, The method further includes an object verification database configured to store the embedding of at least the digital object, and the method further includes The method includes receiving a query digital object for verification, wherein the query digital object includes a digital audio object, and the method further includes: Processing the query digital object using a watermark-decoded neural network to generate a watermark signal for the query digital object by processing the digital audio object and generating a spectrogram of the digital audio object; and processing the spectrogram using the watermark-decoded neural network to generate the watermark signal for the query digital object. The process involves using the embedding neural network to process the query digital object in order to generate query embeddings for the query digital object, The query embeddings are used to query the object validation database to determine one or more sets of similarity scores for one or more sets of stored embeddings of digital objects similar to the query digital object. The origin of the query digital object is verified based on the combination of the watermark signal and the set of one or more similarity scores, Methods that include...

15. The object verification system is an object generation and verification system, wherein the request includes a request to generate the digital object, and maintaining the object verification system further includes maintaining a generative neural network configured to process the request to generate the digital object in accordance with the request, and the method further includes Receiving a request to generate the aforementioned digital object, Processing the request using the generative neural network to generate the digital object, The process includes processing the digital object using the watermarking neural network to generate the watermarked digital object, and processing the digital object is Processing the digital audio object to generate a spectrogram of the digital audio object, The spectrogram is processed using a watermarking neural network to generate a watermark for the digital audio object, Combining the watermark and the spectrogram to obtain a watermarked spectrogram, This is done by converting the watermarked spectrogram into the watermarked digital object, and the method further, To provide the aforementioned watermarked digital object for use, Processing the digital object using the embedding neural network to generate the embedding of the digital object, The embedding of the digital object is stored in the object verification database, The history of the query digital object is verified as having been generated by the object generation and verification system, provided that the query embedding matches the embedding of the digital object stored in the object verification database. The method according to claim 14, including the method described in claim 14.

16. The digital object is stored in the object verification database, The stored digital object is used to detect an attempt to remove the watermark from the query digital object, The method according to claim 15, further comprising:

17. The method according to any one of claims 14 to 16, wherein the watermarking neural network and the watermark decoding neural network are jointly trained to generate the watermarked signal under adversarial perturbations of the watermarked digital object.

18. The method according to any one of claims 14 to 17, wherein the watermarked neural network comprises a neural network having a U-Net architecture and having more trainable parameters than the watermarked neural network.

Images