An audio tracing device and method based on semi-fragile watermarking

Abstract

This invention discloses an audio source tracing device and method based on semi-fragile audio watermarking. The semi-fragile watermarking of this invention combines robust and fragile watermarking, integrating the advantages of both while eliminating their shortcomings in audio protection and source tracing. The robust portion of the semi-fragile watermark is used for audio source authentication. The invariance of the robust watermark after being attacked by malicious models provides users with a marker to identify the audio source, effectively enabling audio source tracing. Conversely, the fragile watermark is completely destroyed after being attacked by malicious models. Provided the robust watermark is correctly extracted and the audio source is confirmed, the fragile watermark, which cannot be correctly extracted, can be used to prove that the audio has been tampered with by malicious models, thereby achieving efficient audio protection.

Description

Technical Field

[0001] This invention relates to the fields of audio watermarking and audio traceability, specifically to an audio traceability device and method based on semi-fragile audio watermarking. Technical Background

[0002] Existing watermarking technologies can be divided into traditional audio watermarking technologies and deep learning-based audio watermarking technologies. The watermarks generated by these two technologies are mainly robust to transmission loss. With the increasing development of deep learning models, the methods of modifying audio using deep learning models are also increasing. Such modifications may be malicious. Therefore, the demand for robust audio watermarks against deep learning model modifications and for audio traceability under malicious tampering is growing. Specifically, existing technologies lack the following two capabilities: (1) Robustness of watermarks against deep learning models: In the real world, audio is not only subject to transmission losses such as white noise and resampling during transmission, but may also be subject to malicious modifications by various deep learning models. Audio watermarks should not only be robust to simple transmission loss, but should also resist modifications by deep learning models; (2) Audio traceability through watermarks: In the real world, it is not only necessary to determine whether audio has been subjected to malicious attacks, but also to trace the source of the audio. In summary, existing audio watermarks are not robust to deep learning models. In the case of malicious AI model tampering, it is impossible to trace the source of audio through watermarks. Summary of the Invention

[0003] This invention improves upon existing technical solutions by providing an audio tracing device and method based on semi-fragile audio watermarking. It utilizes the fragile and robust watermark portions of the semi-fragile watermark to add watermark protection to synthesized audio, and further trains the robust watermark portion using a malicious AI model perturbation. This ensures that the robust watermark can still be extracted normally even after being tampered with by the malicious AI model. Ultimately, it enables the correct identification of tampered audio content when AIGC is modified using a malicious AI model, thus tracing the original audio back to its source.

[0004] This invention is achieved through the following technical solution:

[0005] This invention discloses an audio tracing device based on semi-fragile watermarking, comprising a watermark embedding portion, a watermark distortion portion, and a watermark extraction portion connected thereto:

[0006] The watermark embedding section includes a watermark pool, a watermark encoder connected to the output of the watermark pool, a watermark embedder connected to the output of the watermark encoder, a spectrum encoder connected to the input of the watermark embedder, a short-time Fourier transform A connected to the input of the spectrum encoder, an inverse short-time Fourier transform connected to the output of the watermark embedder, a training audio dataset connected to the short-time Fourier transform A, an audio discriminator connected to the output of the training audio dataset, a device optimizer A associated with the audio discriminator, and a device optimizer B connected to the inputs of the watermark encoder, the watermark embedder, and the spectrum encoder.

[0007] The watermark distortion part is used to enhance the robustness of the watermark, including audio jammer A connected to the output of the inverse short-time Fourier transform and audio jammer B connected to the output of the inverse short-time Fourier transform. Audio jammer A and audio jammer B are used in different stages of device training.

[0008] The watermark extraction section includes a short-time Fourier transform B connected to the audio jammer, a watermark extractor connected to the output of the short-time Fourier transform B, a watermark decoder connected to the output of the watermark extractor, and a device optimizer C connected to the inputs of both the watermark extractor and the watermark decoder.

[0009] As a further improvement, the audio jammer of the present invention simultaneously includes audio cutting, audio resampling, audio compression, and AutoVC interference, and uses the included interference to attempt to destroy the watermark; the training audio dataset is a subset of LibriSpeech, and the watermark pool is a self-built watermark pool containing multiple different watermark information.

[0010] As a further improvement, the spectrum encoder and watermark encoder described in this invention are feature extractors used to extract features from the original data and input them into the watermark embedder for subsequent training.

[0011] As a further improvement, the watermark embedder, audio discriminator, watermark extractor, and watermark decoder described in this invention are all composed of one or more of CNN and LSTM, and during training, the hidden layers in the model are randomly ignored by dropout to reduce the possibility of model overfitting.

[0012] This invention also discloses an audio tracing method based on a semi-fragile watermarking audio tracing device, specifically including the following steps:

[0013] 1) Obtain multiple different watermark information through the watermark pool. The watermark information is initialized as a 0-1 bit sequence that conforms to a normal distribution. The watermarks obtained from multiple samplings from the watermark pool are spliced ​​together to synthesize a semi-fragile watermark.

[0014] 2) The source audio is obtained by sampling the training audio dataset and input into the short-time Fourier transform A. The output spectrum information is input into the spectrum encoder. The spectrum encoder extracts features from the spectrum information to obtain the spectrum embedding information.

[0015] 3) The semi-fragile watermark constructed in step 1) is feature extracted by the watermark encoder. To increase the time independence of the watermark, the encoded watermark information is repeated on the time axis. The output of the watermark encoder is the watermark embedding information.

[0016] 4) Input the spectrum embedding information obtained in step 2) and the watermark embedding information obtained in step 3) into the watermark embedder, and output the audio spectrum with watermark. The watermark embedder mainly implements watermark embedding through CNN and uses dropout to prevent overfitting during training.

[0017] 5) The watermarked audio spectrum obtained in step 4) is input into the inverse short-time Fourier transform A, which converts the audio information in the frequency domain into time-domain audio information that can be heard by the human ear, and outputs the watermarked audio.

[0018] 6) The source audio or the audio with watermark output by the inverse short-time Fourier transform is used by the audio discriminator to determine whether the input audio contains a watermark. The output of the audio discriminator is used as the input of the device optimizer A. The output of the device optimizer A is used to optimize the discrimination ability of the audio discriminator, alternately improving the embedding ability and the discrimination ability.

[0019] 7) Device optimizer B accepts the input of the inverse short-time Fourier transform, and its output is used as model optimization data input into the watermark embedder, spectrum encoder and watermark encoder;

[0020] 8) The audio with the watermark is input to the audio jammer A. The anti-interference ability of the watermark is enhanced by the pre-implemented audio cutting, audio resampling and audio compression transmission interference methods. The enhanced audio with the watermark to be extracted is output.

[0021] 9) The augmented audio of the watermark to be extracted is input to the short-time Fourier transform B, and the augmented audio spectrum information is output.

[0022] 10) Input the augmented audio spectrum information obtained in step 9) into the watermark extractor, perform feature extraction on the augmented spectrum information in the frequency domain, and obtain the augmented watermark embedding information.

[0023] 11) The data-enhanced watermark embedding information obtained in step 10) is averaged over time by average pooling, and the pooled watermark embedding information is output.

[0024] 12) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information;

[0025] 13) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part.

[0026] 14) After repeating steps 8) to 13) several times, the audio with the watermark is input to the audio jammer B. The watermark's resistance to model interference is enhanced by using the pre-implemented AutoVC model or YoutTTS model, and the enhanced audio with the watermark to be extracted is output.

[0027] 15) The enhanced audio of the watermark to be extracted is input to the short-time Fourier transform B, and the enhanced audio spectrum information is output.

[0028] 16) Input the enhanced audio spectrum information obtained in step 15) into the watermark extractor, perform feature extraction on the enhanced spectrum information in the frequency domain, and obtain the enhanced watermark embedding information.

[0029] 17) The enhanced watermark embedding information obtained in step 16) is averaged over time using average pooling to output the pooled watermark embedding information.

[0030] 18) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information;

[0031] 19) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part to finally obtain a trained watermark embedding device.

[0032] As a further improvement, the device used in step 19) of the present invention has the following optimization issues:

[0033]

[0034] in, Let w be the loss function between the audio with and without watermark w, where w is a robust watermark. robust And fragile watermark W vunlerable Combining these components, D(·) is the audio discriminator, σ(·) is the sigmoid function, and the audio discriminator outputs a confidence score indicating whether the input audio carries a watermark. These represent the bit error rates of the undistorted watermark and the robust portion of the watermark after extraction, respectively. This demonstrates that the vulnerable parts of the watermark are damaged after being attacked by a malicious model. The accuracy of extracting the vulnerable watermark is close to guessing, which is approximately 1 / 2.

[0035] As a further improvement, in step 3) of the present invention, the watermark encoder is used to encode semi-fragile watermark information and align the encoded semi-fragile watermark with the spectrogram. The watermark encoder is based on the Full Connect Layer (FCLayer) neural network architecture.

[0036] As a further improvement, in step 4) of the present invention, the watermark embedder embeds the semi-fragile watermark into the audio spectrogram. The watermark embedder is based on a CNN neural network architecture, and its workflow is as follows: the encoded semi-fragile watermark, the encoded spectrogram and the original spectrogram are concatenated and then input into the watermark embedder. The hidden layer adopts a CNN neural network layer with variable size. In step 10), the watermark extractor model architecture is the opposite of that of the watermark embedder.

[0037] As a further improvement, in step 6) of the present invention, the audio discriminator is used to determine whether the audio contains semi-fragile watermark information. The audio discriminator is based on a CNN neural network architecture. Its workflow is as follows: the audio to be judged is transformed by a short-time Fourier transform to generate audio spectrum information, and the audio spectrum is input into the audio discriminator. The smaller the cross-entropy between the output result and the tensor that is all 1, the higher the probability that the audio contains a watermark.

[0038] The beneficial effects of the present invention are as follows:

[0039] This invention proposes an audio tracing device and method based on semi-fragile watermarking. Existing audio watermarking techniques primarily focus on the inaudibility during the watermark embedding process. Furthermore, existing methods, when constructing watermarking methods, largely consider only transmission losses in terms of attack and loss considerations. However, as AIGC content generation becomes increasingly accessible, watermarked audio also needs to be protected against malicious AI model tampering attacks. Current watermarking methods are not robust against malicious attacks, leaving them completely defenseless against such attacks and making it difficult to effectively protect the generated audio. Meanwhile, the primary goal of current watermarking methods is to build robust watermarks. However, a single robust watermark cannot achieve efficient audio protection and tracing while simultaneously resisting malicious model tampering. This invention innovatively proposes a watermarking method that combines fragile and robust watermarks into a semi-fragile watermark. The robust portion enables accurate audio tracing, while the fragile portion of the watermark is destroyed when subjected to malicious model tampering, indicating that the audio has been maliciously tampered with, thereby achieving efficient protection of the target audio. By combining robust watermarking and fragile watermarking, this invention addresses the shortcomings of existing methods and achieves robustness against attack methods not included in the training process.

[0040] In the evaluation of watermark embedding capability, under the condition of no attack or only transmission attack, the overall extraction accuracy of semi-fragile watermark is 99%, which has good embedding and extraction performance. At the same time, the mean square error between the watermarked audio and the original audio is calculated to be 1.5%, which effectively inputs imperceptible watermark information.

[0041] To meet the requirements of audio source tracing, existing technologies mostly employ signal processing methods. Watermark information generated by signal processing methods can only be robust to various losses during transmission. In recent years, with the increasing capabilities of neural network models, research teams have proposed model watermarking schemes. However, existing model watermarks can only be robust to losses during transmission; when faced with malicious tampering of the model, the watermark is destroyed and cannot be extracted, thus losing its function of protecting the audio. This invention proposes a model training-based approach to enhance the robustness of watermarks against malicious model attacks. Watermarked audio can be modified by audio synthesis models (i.e., the malicious models mentioned above) (such as AutoVC and YourTTS). By introducing various audio synthesis models with different architectures into the watermark embedding and extraction training process, the trained watermark embedding and extraction model can effectively discover and learn the methods by which audio synthesis models modify audio, thus providing better defense against malicious model tampering that occurs during training. This makes the embedded semi-fragile watermark robust against these malicious models. This invention achieves robust watermark extraction accuracy of 97% and 90% (average success rate) under two different attack models, AutoVC and YourTTS, respectively. In contrast, existing technologies only achieve around 50% watermark extraction accuracy against these two attack models. This demonstrates that existing audio watermark embedding models cannot make watermarks robust against malicious models. Therefore, given the low barrier to entry and strong synthesis capabilities of current malicious tampering methods, existing audio watermark embedding methods have significant limitations and cannot effectively protect audio. In contrast, the semi-fragile watermarking method proposed in this invention achieves a high watermark extraction accuracy against malicious tampering models, fully demonstrating that the model training method proposed in this invention effectively improves the robust watermark's ability to resist malicious model tampering.

[0042] To meet the requirements of audio protection, this invention combines fragile watermarks and robust watermarks into a semi-fragile watermark. The extraction accuracy of the fragile watermark is 55% (ideally 50%) under attacks from two different models: AutoVC and YourTTS. The proposed semi-fragile watermark achieves audio source tracing through its robust portion, using the watermark's invariance to different attacks to prove the audio's origin. It provides audio protection against malicious attacks through its fragile portion. When faced with a malicious attack, the fragile portion of the watermark is destroyed and cannot be correctly extracted. Therefore, after the fragile portion of the watermark is destroyed, the audio user can determine that the audio segment has been attacked by a malicious model and differs from the original audio, thus achieving audio protection.

[0043] Semi-fragile watermarking combines robust and fragile watermarking, integrating the advantages of both while mitigating their shortcomings in audio protection and source tracing. The robust portion of a semi-fragile watermark is used for audio source authentication. The invariance of a robust watermark after being attacked by malicious models provides users with a marker to identify the audio source, effectively enabling audio source tracing. Conversely, a fragile watermark is completely destroyed after being attacked by malicious models. Provided the robust watermark is correctly extracted and the audio source is confirmed, the fragile watermark, which cannot be correctly extracted, can be used to prove that the audio has been tampered with by malicious models, thus achieving efficient audio protection. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the device framework of the present invention;

[0045] Figure 2 The figure shows an example of the test results of the robustness of the watermark against different types of interference for the present invention and existing work. Detailed Implementation

[0046] The technical solution of the present invention will be further illustrated below through specific implementation examples:

[0047] This invention discloses an audio tracing device and method based on semi-fragile watermarking, which operates in the following scenario: When a defender generates their own audio, for security reasons, they embed a semi-fragile watermark into the audio using the watermark embedding device disclosed in this invention. The semi-fragile watermark is sampled by the model from the watermark pool constructed by the defender themselves. Assuming the attacker can only obtain a limited amount of audio data generated by the defender, they can only use zero-shot models such as AutoVC and YourTTS to tamper with the obtained audio. After tampering with the defender's audio, the attacker claims the tampered audio belongs to the defender, thereby framing them. It is important to note that the models used by the attacker are all models trained on a large amount of high-quality audio data. During the tampering process, because the amount of audio generated by the defender is too small, it is impossible to fine-tune the model using the limited audio data; the attacker can only modify the audio based on the model's inherent capabilities.

[0048] This invention discloses an audio source tracing device and method based on semi-fragile watermarking. The device includes a watermark embedding part, a watermark distortion part, and a watermark extraction part. Figure 1 This is a schematic diagram of the device framework of the present invention.

[0049] The watermark embedding section includes a watermark pool, a watermark encoder connected to the output of the watermark pool, a watermark embedder connected to the output of the watermark encoder, a spectrum encoder connected to the input of the watermark embedder, a short-time Fourier transform A connected to the input of the spectrum encoder, an inverse short-time Fourier transform connected to the output of the watermark embedder, a training audio dataset connected to the short-time Fourier transform A, an audio discriminator connected to the output of the training audio dataset, a device optimizer A associated with the audio discriminator, and a device optimizer B connected to the inputs of the watermark encoder, the watermark embedder, and the spectrum encoder.

[0050] The original audio is sampled from the training dataset. The sampled original audio is input into a short-time Fourier transform (SFT) A, which outputs overall spectral and phase information. The semi-fragile watermark consists of two parts: a fragile watermark and a robust watermark. It is sampled twice from a self-built watermark pool to obtain the fragile and robust watermarks respectively. These two watermarks are then concatenated to obtain the desired semi-fragile watermark. The semi-fragile watermark is then input into a watermark encoder to obtain the encoded watermark information. Simultaneously, the spectral information output from the SFT A is encoded by a spectral encoder to obtain the encoded spectral information. This encoded spectral information is combined with the encoded watermark information and input into a watermark embedder to output the watermarked audio spectrum. The phase information output from the SFT A and the watermarked audio spectrum are used as input to an inverse SFT to obtain the watermarked audio. The watermarked audio is then used as input to an audio discriminator, which judges whether the audio carries watermark information. The output of the audio discriminator is backpropagated through device optimizer A and used to optimize the audio discriminator. The watermark information and spectrum information are forward propagated in the watermark embedder, and the output is backpropagated through device optimizer B and used to optimize the watermark embedder, audio discriminator, and spectrum encoder.

[0051] The watermark distortion section enhances the watermark's robustness and includes audio jammer A and audio jammer B, both connected to the output of the inverse short-time Fourier transform (ISFT). Audio jammers A and B are used at different stages of device training. The audio jammers incorporate multiple types of audio interference, including audio clipping, audio resampling, audio compression, and AutoVC interference.

[0052] The watermark extraction section includes a short-time Fourier transform (SFT) B connected to the audio jammer, a watermark extractor connected to the output of SFT B, a watermark decoder connected to the output of the watermark extractor, and a device optimizer C connected to the inputs of both the watermark extractor and the watermark decoder. The audio output from the watermark distortion section is processed by SFT B to output an audio spectrum. This audio spectrum is input to the watermark extractor, which outputs an encoded watermark. The encoded watermark is then input to the watermark decoder, which outputs the decoded watermark. The audio with the semi-fragile watermark propagates forward between the watermark decoder and the watermark extractor. The MSE Loss between the extracted watermark information and the original semi-fragile watermark is backpropagated by the device optimizer C and used to optimize the watermark decoder and the watermark extractor.

[0053] The audio source tracing device based on semi-fragile watermarking includes an audio jammer that simultaneously incorporates audio clipping, audio resampling, audio compression, and AutoVC interference. The jammer attempts to disrupt the watermark. The training audio dataset is a subset of LibriSpeech, and the watermark pool is a self-built pool containing multiple different watermark information.

[0054] In an audio tracing device based on semi-fragile watermarking, the spectrum encoder and watermark encoder are feature extractors used to extract features from the raw data and input them into the watermark embedder for subsequent training.

[0055] The audio source tracing device based on semi-fragile watermarking consists of a watermark embedder, an audio discriminator, a watermark extractor, and a watermark decoder, all of which are composed of one or more of CNN and LSTM. During training, the hidden layers in the model are randomly ignored by dropout to reduce the possibility of model overfitting.

[0056] The watermark embedding model generated by training the model with data from the dataset can add a credible semi-fragile watermark to audio, thereby achieving effective protection and accurate traceability of audio.

[0057] This invention discloses an audio tracing device and method based on semi-fragile watermarking. The method mainly consists of a watermark embedding part, a watermark distortion part, and a watermark extraction part, and its implementation steps are as follows:

[0058] 1) Obtain multiple different watermark information through the watermark pool. The watermark information is initialized as a 0-1 bit sequence that conforms to a normal distribution. The watermarks obtained from multiple samplings from the watermark pool are spliced ​​together to synthesize a semi-fragile watermark.

[0059] 2) The source audio is obtained by sampling the training audio dataset and input into the short-time Fourier transform A. The output spectrum information is input into the spectrum encoder. The spectrum encoder extracts features from the spectrum information to obtain the spectrum embedding information.

[0060] 3) The semi-fragile watermark constructed in step 1) is feature extracted by the watermark encoder. To increase the time independence of the watermark, the encoded watermark information is repeated on the time axis. The output of the watermark encoder is the watermark embedding information.

[0061] 4) Input the spectrum embedding information obtained in step 2) and the watermark embedding information obtained in step 3) into the watermark embedder, and output the audio spectrum with watermark. The watermark embedder mainly implements watermark embedding through CNN and uses dropout to prevent overfitting during training.

[0062] 5) The watermarked audio spectrum obtained in step 4) is input into the inverse short-time Fourier transform A, which converts the audio information in the frequency domain into time-domain audio information that can be heard by the human ear, and outputs the watermarked audio.

[0063] 6) The source audio or the audio with watermark output by the inverse short-time Fourier transform is used by the audio discriminator to determine whether the input audio contains a watermark. The output of the audio discriminator is used as the input of the device optimizer A. The output of the device optimizer A is used to optimize the discrimination ability of the audio discriminator, alternately improving the embedding ability and the discrimination ability.

[0064] 7) Device optimizer B accepts the input of the inverse short-time Fourier transform, and its output is used as model optimization data input into the watermark embedder, spectrum encoder and watermark encoder;

[0065] 8) The audio with the watermark is input to the audio jammer A. The anti-interference ability of the watermark is enhanced by the pre-implemented audio cutting, audio resampling and audio compression transmission interference methods. The enhanced audio with the watermark to be extracted is output.

[0066] 9) The augmented audio of the watermark to be extracted is input to the short-time Fourier transform B, and the augmented audio spectrum information is output.

[0067] 10) Input the augmented audio spectrum information obtained in step 9) into the watermark extractor, perform feature extraction on the augmented spectrum information in the frequency domain, and obtain the augmented watermark embedding information.

[0068] 11) The data-enhanced watermark embedding information obtained in step 10) is averaged over time by average pooling, and the pooled watermark embedding information is output.

[0069] 12) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information;

[0070] 13) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part.

[0071] 14) After repeating steps 8) to 13) several times, the audio with the watermark is input to the audio jammer B. The watermark's resistance to model interference is enhanced by using the pre-implemented AutoVC model or YoutTTS model, and the enhanced audio with the watermark to be extracted is output.

[0072] 15) The enhanced audio of the watermark to be extracted is input to the short-time Fourier transform B, and the enhanced audio spectrum information is output.

[0073] 16) Input the enhanced audio spectrum information obtained in step 15) into the watermark extractor, perform feature extraction on the enhanced spectrum information in the frequency domain, and obtain the enhanced watermark embedding information.

[0074] 17) The enhanced watermark embedding information obtained in step 16) is averaged over time using average pooling to output the pooled watermark embedding information.

[0075] 18) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information;

[0076] 19) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part to finally obtain a trained watermark embedding device.

[0077] The following is a detailed explanation of the watermark encoder and watermark decoder. To extend the low-dimensional watermark information to a high dimension similar to the audio spectrum, the watermark encoder extends the watermark from 16 bits to 512 bits, aligning it with the output of the spectrum encoder. To increase the time independence of the watermark, the watermark encoder also repeats the encoded watermark information along the time axis. The watermark encoder is based on a Full Connect Layer (FC Layer) neural network architecture to encode the watermark. The framework of the watermark extractor is symmetrical to the watermark encoder. Before watermark extraction, to improve the extractor's capabilities, it uses pre-implemented transmission interference and malicious model interference to simulate audio distortion under different conditions, thus augmenting the audio features. Specifically, let the watermark information be w = concatenate(W robust W vunlerableThis indicates that the watermark information consists of both robust and vulnerable parts, hence the watermark encoder is called Encoder. w The watermark decoder is Decoder w The high-dimensional watermark information extracted from the watermarked audio by the watermark extractor is f′. w Therefore, based on the model structure, the loss functions for the watermark encoder and watermark decoder are derived as follows:

[0078] w′=Decoder w (f′ w )

[0079]

[0080] In the presence of interference, let f dl For the watermarked audio after being interfered with by an audio jammer, the high-dimensional watermark information extracted by the watermark extractor is then given as follows:

[0081] w′ dl =Decoder w (f dl )

[0082]

[0083] When the device optimizer optimizes the model using this loss function, this loss function will be used to train all trainable models except the audio discriminator during backpropagation.

[0084] The following is a detailed explanation of the watermark embedder. The input to the watermark embedder is the audio spectrum information and the output of the spectrum encoder. To increase the time independence of the watermark, the watermark is repeated on the time axis before being input into the watermark embedder. Let the audio spectrum information be s, and the spectrum encoder be... s The output of the watermark encoder is f w Then the input to the watermark embedder is:

[0085] f + =Concatenate(Encoder) s (s), s, Repeat(f) w ,T))

[0086] The output of the watermark embedder generates audible audio information through inverse short-time Fourier transform. Therefore, the output of the watermark embedder is a watermarked audio spectrogram. Let the watermark embedder be called Embedder. w The process of generating watermarked audio by performing an inverse short-time Fourier transform (ISTFT) is as follows:

[0087] audio w =ISTFT(Embedder) w (f + ))

[0088] The following is a detailed description of the audio watermark extractor. In addition to the watermark embedding part, this invention also discloses a watermark extraction technique to enable defenders to extract watermarks from audio and verify whether it originates from themselves. Specifically, the input to the watermark extractor is a spectrum. Therefore, before a sound sample (audio) is input to the watermark extractor, a corresponding spectrum signal s = STFT(audio) needs to be generated through a short-time Fourier transform. The generated spectrum is then input to the watermark extractor (Extractor). w This leads to the encoded watermark information f′. w The overall process is as follows:

[0089] s, p = STFT(audio)

[0090] f′ w =Extractor w (s)

[0091] The following is a detailed explanation of the audio discriminator. To enhance the listening experience of watermarked audio and improve the watermark's concealment, this invention employs a Generative Adversarial Network (GAN) structure. The basic idea of ​​this method is to generate data that sounds natural and realistic, similar to the original audio, through a generative model (i.e., the audio embedder), and then use a discriminative model (i.e., the audio discriminator) to determine whether the given audio is real or watermarked. Using this method, the watermark embedder can significantly improve the watermark's concealment during the training of the watermark embedding model. Let the audio discriminator be D, and the loss functions for the audio discriminator and the watermark embedder are:

[0092]

[0093] σ(·) is the sigmoid function. By training the audio discriminator with the generated watermarked audio, the embedding capability of the watermark embedder and the discriminator's ability to distinguish whether the audio carries a watermark are spirally increased, ultimately obtaining a watermark embedding model that can efficiently and covertly embed watermarks.

[0094] The following is a detailed explanation of the audio jammer. The audio jammer includes pre-implemented transmission interference such as audio trimming, audio resampling, and audio compression; a pre-implemented AutoVC model for modifying speaker information; and a pre-implemented YourTTS model for modifying speech content. During optimization, the parameters of both the AutoVC and YourTTS models are frozen, and the device optimizer will not optimize these two models.

[0095] The device optimizers are described in detail below. The three device optimizers A, B, and C used in this invention are all Adam optimizers based on the PyTorch platform, and are trained and optimized using mini-batch technology.

[0096] To verify the technical effectiveness of this invention, the semi-fragile watermark embedding and extraction model was trained on an AMAX server (Intel Xeon Silver 4210R, 256GB RAM, NVIDIA RTX A6000) by minimizing the objective function. By default, the audio sampling rate was set to 22050, the window size during the short-time Fourier transform was set to 1024, and the step size for each translation was set to 1024. During training, two mainstream audio processing model architectures (CNN and LSTM) and two audio synthesis models (i.e., malicious tampering models, AutoVC and YourTTS) were used. In addition, two training datasets and one test dataset (from the LibriSpeech dataset) were used.

[0097] The test data during the testing process consisted of two categories: the first category was watermarked audio generated by existing audio watermarking methods. This invention adopted three existing audio watermarking methods: Timber watermark, DCT-watermarking, and FSVC-watermark. The first method is model watermarking, while the latter two are traditional signal processing watermarking. The second category was the semi-fragile watermark embedding model built by this invention. For each type of test data, this invention conducted four tests, which were: (1) the watermark extraction accuracy under interference-free conditions; (2) the watermark extraction accuracy under transmission loss interference (such as resampling, audio clipping, etc.); (3) the watermark extraction accuracy after the AutoVC model modified the speaker of the watermarked audio; and (4) the watermark extraction accuracy after the YourTTS model modified the speech content of the watermarked audio. Figure 2 The results show the watermark extraction accuracy during testing. Compared to existing methods, the robust portion of the semi-fragile watermark embedded in this invention maintains a 97% extraction accuracy even after a malicious model attack. This is a significant improvement in robustness compared to the 50% extraction accuracy of existing watermarking methods after malicious model attacks. Furthermore, the vulnerable portion of the semi-fragile watermark is also destroyed when facing malicious model attacks, achieving an extraction accuracy of 55% during testing (ideally 50%).

[0098] The accuracy of watermark extraction of this invention after being attacked by different malicious models was tested to further evaluate the transferability of this invention in different models. The X-TTS and DiffVC models were used to modify the watermarked audio, and then the watermark extraction model contained in this invention was used to extract the watermark and calculate the extraction accuracy.

[0099] Finally, an objective experiment was conducted to evaluate the degree of perceptual difference between the watermarked audio generated by this invention and the original audio in the human ear. First, the watermarked audio and the original audio were randomly mixed and played in sequence. Volunteers evaluated the quality of the watermarked audio by rating the audio using MOS (1-5, with higher scores indicating more authentic audio) and judging whether it contained a watermark.

[0100] Finally, it should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An audio tracing device based on semi-fragile watermarking, characterized in that, This includes the watermark embedding portion, the watermark distortion portion, and the watermark extraction portion, which are connected in this way: The watermark embedding part includes a watermark pool, a watermark encoder connected to the output of the watermark pool, a watermark embedder connected to the output of the watermark encoder, a spectrum encoder connected to the input of the watermark embedder, a short-time Fourier transform A connected to the input of the spectrum encoder, an inverse short-time Fourier transform connected to the output of the watermark embedder, a training audio dataset connected to the short-time Fourier transform A, an audio discriminator connected to the output of the training audio dataset, a device optimizer A associated with the audio discriminator, and a device optimizer B connected to the inputs of the watermark encoder, the watermark embedder, and the spectrum encoder. The watermark distortion portion is used to enhance the robustness of the watermark, including an audio jammer A connected to the output of the inverse short-time Fourier transform and an audio jammer B connected to the output of the inverse short-time Fourier transform. The audio jammer A and the audio jammer B are used in different stages of device training. The watermark extraction section includes a short-time Fourier transform B connected to the audio jammer, a watermark extractor connected to the output of the short-time Fourier transform B, a watermark decoder connected to the output of the watermark extractor, and a device optimizer C connected to the inputs of both the watermark extractor and the watermark decoder.

2. The audio tracing device based on semi-fragile watermarking according to claim 1, characterized in that, The audio jammer includes audio cutting, audio resampling, audio compression, and AutoVC interference, and attempts to disrupt the watermark using the included interference. The training audio dataset is a subset of LibriSpeech, and the watermark pool is a self-built watermark pool containing multiple different watermark information.

3. The audio tracing device based on semi-fragile watermarking according to claim 1, characterized in that, The spectrum encoder and watermark encoder are feature extractors used to extract features from the raw data and input them into the watermark embedder for subsequent training.

4. The audio tracing device based on semi-fragile watermarking according to claim 1, 2, or 3, characterized in that, The watermark embedder, audio discriminator, watermark extractor, and watermark decoder are all composed of one or more of CNN and LSTM, and during training, the hidden layers in the model are randomly ignored by dropout to reduce the possibility of model overfitting.

5. An audio tracing method based on a semi-fragile watermarking-based audio tracing device, characterized in that, Specifically, the following steps are included: 1) Obtain multiple different watermark information through the watermark pool. The watermark information is initialized as a 0-1 bit sequence that conforms to a normal distribution. The watermarks obtained from multiple samplings from the watermark pool are spliced ​​together to synthesize a semi-fragile watermark. 2) The source audio is obtained by sampling the training audio dataset and input into the short-time Fourier transform A. The output spectrum information is input into the spectrum encoder. The spectrum encoder extracts features from the spectrum information to obtain the spectrum embedding information. 3) The semi-fragile watermark constructed in step 1) is feature extracted by the watermark encoder. To increase the time independence of the watermark, the encoded watermark information is repeated on the time axis. The output of the watermark encoder is the watermark embedding information. 4) Input the spectrum embedding information obtained in step 2) and the watermark embedding information obtained in step 3) into the watermark embedder, and output the audio spectrum with watermark. The watermark embedder mainly implements watermark embedding through CNN and uses dropout to prevent overfitting during training. 5) The watermarked audio spectrum obtained in step 4) is input into the inverse short-time Fourier transform A, which converts the audio information in the frequency domain into time-domain audio information that can be heard by the human ear, and outputs the watermarked audio. 6) The source audio or the audio with watermark output by the inverse short-time Fourier transform is used by the audio discriminator to determine whether the input audio contains a watermark. The output of the audio discriminator is used as the input of the device optimizer A. The output of the device optimizer A is used to optimize the discrimination ability of the audio discriminator, alternately improving the embedding ability and the discrimination ability. 7) Device optimizer B accepts the input of the inverse short-time Fourier transform, and its output is used as model optimization data input into the watermark embedder, spectrum encoder and watermark encoder; 8) The audio with the watermark is input to the audio jammer A. The anti-interference ability of the watermark is enhanced by the pre-implemented audio cutting, audio resampling and audio compression transmission interference methods. The enhanced audio with the watermark to be extracted is output. 9) The augmented audio of the watermark to be extracted is input to the short-time Fourier transform B, and the augmented audio spectrum information is output. 10) Input the augmented audio spectrum information obtained in step 9) into the watermark extractor, perform feature extraction on the augmented spectrum information in the frequency domain, and obtain the augmented watermark embedding information. 11) The data-enhanced watermark embedding information obtained in step 10) is averaged over time by average pooling, and the pooled watermark embedding information is output. 12) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information; 13) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part. 14) After repeating steps 8) to 13) several times, the watermarked audio is input to the audio jammer B. The watermark's resistance to model interference is enhanced by using the pre-implemented AutoVC model or YoutTTS model, and the enhanced audio with the watermark to be extracted is output. 15) The enhanced audio of the watermark to be extracted is input into the short-time Fourier transform B, and the enhanced audio spectrum information is output. 16) Input the enhanced audio spectrum information obtained in step 15) into the watermark extractor, perform feature extraction on the enhanced spectrum information in the frequency domain, and obtain the enhanced watermark embedding information. 17) The enhanced watermark embedding information obtained in step 16) is averaged over time using average pooling to output the pooled watermark embedding information. 18) The pooled watermark embedding information is input into the watermark decoder to obtain the extracted watermark information; 19) Calculate the MSE Loss of the watermark decoding result and the robust and fragile parts of the original watermark respectively. Based on the calculated results, the device optimizer C solves the device optimization problem and iteratively optimizes the models of the watermark extractor, watermark decoder and watermark embedding part to finally obtain a trained watermark embedding device.

6. The audio tracing method based on the semi-fragile watermarking-based audio tracing device according to claim 5, characterized in that, The optimization problem of the device used in step 19) is as follows: ； ； ； ； ； ； in, To embed watermark The loss function between the audio after watermarking and the audio without watermarking, and the watermark Robust watermark and fragile watermark Combining For audio discriminator, The sigmoid function is used by the audio discriminator to output a confidence score indicating whether the input audio carries a watermark. These represent the bit error rates of the undistorted watermark and the robust portion of the watermark after extraction, respectively. This demonstrates that the vulnerable parts of the watermark are damaged after being attacked by a malicious model. The accuracy of extracting the vulnerable watermark is close to guessing, which is approximately 1 / 2.

7. The audio tracing method based on the semi-fragile watermarking-based audio tracing device according to claim 5, characterized in that, In step 3), the watermark encoder is used to encode the semi-fragile watermark information and align the encoded semi-fragile watermark with the spectrogram. The watermark encoder is based on the Full Connect Layer (FC Layer) neural network architecture.

8. The audio tracing method based on the semi-fragile watermarking-based audio tracing device according to claim 5, characterized in that, In step 4), the watermark embedder embeds the semi-fragile watermark into the audio spectrogram. The watermark embedder is based on a CNN neural network architecture. Its workflow is as follows: the encoded semi-fragile watermark, the encoded spectrogram and the original spectrogram are concatenated and then input into the watermark embedder. The hidden layer adopts a CNN neural network layer with variable size. In step 10), the watermark extractor model architecture is the opposite of the watermark embedder.

9. The audio tracing method based on the semi-fragile watermarking-based audio tracing device according to claim 5, characterized in that, In step 6), the audio discriminator is used to determine whether the audio contains semi-fragile watermark information. The audio discriminator is based on a CNN neural network architecture. Its workflow is as follows: the audio to be judged is transformed by a short-time Fourier transform to generate audio spectrum information, and the audio spectrum is input into the audio discriminator. The smaller the cross-entropy between the output result and the tensor that is all 1, the higher the probability that the audio contains a watermark.

Images