Training of an audio watermark model, audio watermark embedding method and apparatus

Abstract

This invention provides a training method and apparatus for an audio watermarking model, an audio watermark embedding method, and an apparatus thereof, belonging to the field of audio and video processing technology. The method includes: acquiring a watermarked audio sample obtained by fusing an initial audio sample with a watermark-encoded sample; inputting the perceptual pair composed of the watermarked audio and the initial audio into an auditory perception evaluation model to obtain an auditory perception quality value; and finally using the perceptual loss determined based on the auditory perception quality value as part of the model loss to update the audio watermarking model. This invention constructs the perceptual loss through a differentiable proxy model, directly using the auditory perception score as the optimization objective. This overcomes the deficiency of traditional mean square error in representing the nonlinear characteristics of the human ear, significantly improving the naturalness, fidelity, and concealment of the watermarked audio, while greatly enhancing the model's robustness against various complex channel distortion attacks, ensuring the accuracy of watermark extraction.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to an audio watermarking model training, audio watermark embedding method and apparatus. Background Technology

[0002] As a widely disseminated information carrier, digital audio is increasingly important for copyright protection and authenticity verification. Digital watermarking technology, by embedding specific hidden information into audio, can effectively identify copyright ownership and prevent illegal tampering. However, due to the high sensitivity of the human auditory system, audio watermarking technology, while ensuring high extraction accuracy and robustness against attacks, faces extremely stringent audio quality requirements. It must ensure that the watermarked audio is completely transparent and imperceptible to the user.

[0003] To meet the aforementioned high-quality audio watermarking embedding requirements, existing deep learning-based audio watermarking processing schemes typically first perform time-frequency transformation on the original audio signal to separate the amplitude spectrum and phase spectrum features. Then, the watermark encoding and amplitude spectrum features are concatenated and input into a neural network to calculate modulation features. Finally, the original phase spectrum is combined to reconstruct the watermarked audio. During the entire network model training phase, to maintain audio quality as much as possible, existing technologies generally use the mean squared error (MSE) or absolute error in the frequency domain features of the original audio and its corresponding watermarked audio as the loss function. The core logic is to continuously minimize this error during model training, forcibly constraining the physical numerical differences between the generated signal and the original signal, thereby guiding the iterative update of the network parameters.

[0004] However, existing technologies have significant limitations in optimizing audio quality. This constraint method, which is based solely on frequency domain numerical approximation, essentially only measures the absolute differences of audio signals in the mathematical and statistical dimension. The optimization direction that relies solely on reducing mathematical and statistical errors is seriously out of touch with the real listening experience, making it difficult for the generated watermarked audio to achieve true high fidelity in subjective perception. Summary of the Invention

[0005] This invention provides a training method and apparatus for an audio watermarking model, an audio watermark embedding method, and an apparatus to address the shortcomings of existing technologies where loss functions based solely on frequency domain numerical approximation cannot characterize the nonlinear perceptual characteristics of the human auditory system, leading to a severe disconnect between the model optimization direction and the actual subjective listening experience. This invention achieves direct driving of model network parameter updates based on auditory perception quality, overcoming the limitations of traditional numerical error optimization, thereby significantly improving the naturalness, fidelity, and concealment of watermarked audio in subjective listening experience.

[0006] This invention provides a training method for an audio watermarking model, comprising: A watermarked audio sample is obtained by fusing an initial audio sample with a watermarked encoded sample. The perceptual audio pair composed of the watermarked audio sample and the initial audio sample is input into the auditory perception evaluation model to obtain the auditory perception quality value output by the auditory perception evaluation model; wherein, the auditory perception evaluation model is trained based on the real labels corresponding to the objective auditory evaluation index; Based on the auditory perception quality value, a perceptual loss function is determined, and the model parameters of the audio watermarking model are updated in combination with the perceptual loss function.

[0007] According to a training method for an audio watermarking model provided by the present invention, the step of updating the model parameters of the audio watermarking model by combining the perceptual loss function includes: Calculate the frequency domain reconstruction loss between the amplitude spectrum features of the watermarked audio sample and the amplitude spectrum features of the initial audio sample; The watermark audio sample is input into the watermark extraction model to obtain the reconstructed watermark code output by the watermark extraction model; Calculate the watermark decoding loss between the reconstructed watermark code and the watermark code sample; The global loss function is obtained by fusing the perceptual loss function, the frequency domain reconstruction loss, and the watermark decoding loss. The model parameters of the audio watermarking model are jointly updated using the global loss function.

[0008] According to a training method for an audio watermarking model provided by the present invention, the step of inputting the watermarked audio sample into a watermark extraction model and obtaining the reconstructed watermark code output by the watermark extraction model includes: The watermarked audio sample is input into the simulation attack layer of the watermark extraction model; In the current training batch, a target attack method is randomly selected from the preset set of attack methods to interfere with the watermarked audio sample, resulting in an attacked audio sample. The attacked audio sample is input into the feature extraction layer of the watermark extraction model, and the reconstructed watermark code is output.

[0009] According to a training method for an audio watermarking model provided by the present invention, the step of determining the perceptual loss function based on the auditory perception quality value includes: From the auditory perception quality value, obtain the current quality evaluation score of the perceived audio pair under different auditory evaluation dimensions, wherein the auditory evaluation dimensions include at least the speech quality perception dimension and the objective intelligibility dimension; Obtain the target quality label corresponding to each of the auditory evaluation dimensions for the perceived audio pair; Calculate the degree of quality deviation between each current quality evaluation score and the corresponding target quality label; The perceptual loss function is obtained by fusing and calculating the degree of quality deviation under each of the aforementioned auditory evaluation dimensions.

[0010] According to the training method of the audio watermarking model provided by the present invention, the auditory perception evaluation model is obtained by iteratively performing the following pre-training steps until a preset convergence condition is met: Obtain an evaluation training sample pair, which consists of any initial training audio sample and a watermarked audio sample obtained by adding a watermark to the initial training audio sample. The evaluation training sample pairs are input into the auditory perception evaluation model to be trained, and the predicted values ​​of the objective auditory evaluation index output by the auditory perception evaluation model to be trained are obtained. Calculate the prediction difference loss between the predicted value of the objective auditory evaluation index and the actual label; The model parameters of the auditory perception evaluation model to be trained are updated based on the predicted difference loss.

[0011] According to the training method of the audio watermarking model provided by the present invention, the true label is determined based on the following steps: Calculate the speech quality perception evaluation value and short-term objective intelligibility value for the evaluation training sample pairs respectively; The speech quality perception evaluation value is subjected to numerical transformation processing to map its numerical range to the same target interval as the short-term objective intelligibility value, thus obtaining the mapped evaluation value. The mapping evaluation value and the short-term objective understandability value are jointly determined as the true label.

[0012] According to a training method for an audio watermarking model provided by the present invention, the step of obtaining the evaluation training sample pairs includes: When the training rounds of the audio watermarking model reach a preset number of rounds, the pre-trained audio watermarking model is obtained. Using the pre-trained audio watermarking model, the initial training audio samples in the training dataset are processed to obtain watermarked audio samples corresponding to the initial training audio samples. The initial training audio samples and the corresponding watermarked audio samples are combined to obtain the evaluation training sample pair used to train the auditory perception evaluation model.

[0013] According to a training method for an audio watermarking model provided by the present invention, the step of obtaining a watermarked audio sample obtained by fusing an initial audio sample and a watermarked encoded sample includes: Obtain the amplitude spectrum features and phase spectrum features of the initial audio sample; Determine the time dimension and frequency dimension of the amplitude spectrum feature; The watermarked coded sample is extended along the time dimension to obtain an extended watermark feature, and the time dimension of the extended watermark feature matches the time dimension of the amplitude spectrum feature. The extended watermark feature and the amplitude spectrum feature are concatenated along the direction of the frequency dimension to obtain the fused feature; The fused features are input into the audio watermarking model to obtain the modulation features output by the audio watermarking model; The watermarked audio sample is reconstructed based on the modulation features and the phase spectrum features.

[0014] According to a training method for an audio watermarking model provided by the present invention, the step of reconstructing the watermarked audio sample based on the modulation features and the phase spectrum features includes: The modulation features are determined as the target amplitude spectrum features; The target amplitude spectrum features and phase spectrum features are converted from the frequency domain to the time domain using the inverse short-time Fourier transform to obtain the watermarked audio sample.

[0015] The present invention also provides an audio watermark embedding method, comprising: Based on any of the above training methods for audio watermarking models, an audio watermarking model is trained. Obtain the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark encoding to be embedded; The target watermark code and the amplitude spectrum feature are fused to obtain the target fused feature; The target fusion feature is input into the audio watermarking model to obtain the target modulation feature output by the audio watermarking model; Based on the target modulation features and the phase spectrum features, the target watermark audio embedded with the target watermark code is reconstructed.

[0016] The present invention also provides a training device for an audio watermarking model, comprising: The audio reconstruction module is used to obtain a watermarked audio sample obtained by fusing an initial audio sample with a watermarked encoded sample. The perception evaluation module is used to input the perception audio pair composed of the watermarked audio sample and the initial audio sample into the auditory perception evaluation model, and obtain the auditory perception quality value output by the auditory perception evaluation model; wherein, the auditory perception evaluation model is trained based on the real labels corresponding to the objective auditory evaluation index; The model update module is used to determine the perceptual loss function based on the auditory perception quality value, and to update the model parameters of the audio watermarking model in combination with the perceptual loss function.

[0017] The present invention also provides an audio watermark embedding device, comprising: The model training module is used to train an audio watermarking model based on any of the above-mentioned audio watermarking model training methods. The information acquisition module is used to acquire the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark encoding to be embedded. The feature fusion module is used to fuse the target watermark code with the amplitude spectrum feature to obtain the target fused feature; A modulation processing module is used to input the target fusion feature into the audio watermarking model and obtain the target modulation feature output by the audio watermarking model. The audio reconstruction module is used to reconstruct the target watermark audio embedded with the target watermark code based on the target modulation features and the phase spectrum features.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training of the audio watermarking model and the audio watermark embedding method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training of the audio watermarking model and the audio watermark embedding method as described above.

[0020] The audio watermarking model training, audio watermarking embedding method and device provided by this invention constructs perceptual loss through a differentiable proxy model, directly using the listening score as the optimization target, overcoming the defect that traditional mean square error cannot characterize the nonlinear characteristics of the human ear, significantly improving the naturalness, fidelity and concealment of the watermarked audio, while greatly enhancing the robustness of the model against various complex channel distortion attacks, and ensuring the accuracy of watermark extraction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the training method for the audio watermarking model provided by this invention.

[0023] Figure 2 This is a schematic diagram of the training process of the audio watermarking model combined with the perceptual loss function provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the process for obtaining the reconstructed watermark code provided by the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the process of determining the perceptual loss function provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the training process of the auditory perception assessment model provided by the present invention.

[0027] Figure 6 This is a schematic diagram illustrating the process of constructing real labels for training the auditory perception assessment model provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the process for obtaining evaluation training sample pairs provided by the present invention.

[0029] Figure 8 This is a flowchart illustrating the audio watermark embedding method provided by the present invention.

[0030] Figure 9 This is a schematic diagram of the structure of the training device for the audio watermarking model provided by the present invention.

[0031] Figure 10 This is a schematic diagram of the audio watermark embedding device provided by the present invention.

[0032] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0035] With the increasing prevalence of digital multimedia (encompassing various forms such as audio, images, and video), the protection of copyright and the verification of authenticity of this digital multimedia content have become increasingly crucial. Existing watermarking methods, commonly used in technology based on neural networks, typically attempt to maintain audio quality by limiting signal differences solely through minimizing the mean square error between the short-time Fourier transform (STFT) features of the original audio and the watermarked audio.

[0036] However, this method, which relies solely on frequency domain numerical approximation, has significant limitations, primarily in that it fails to consider the nonlinear perceptual characteristics of the human auditory system. For example, in the low-energy frequency band, even slight signal changes can lead to significant differences in perceived sound; while in the high-energy frequency band, the same signal changes may be masked. This means that simply reducing the MSE is insufficient to guarantee high fidelity in subjective listening experience.

[0037] To address this issue, this invention proposes a training method and apparatus for an audio watermarking model, as well as an audio watermark embedding method. It employs a dynamic fusion optimization strategy based on multiple auditory quality assessment metrics, enabling direct use of objective auditory perception to drive the training of the audio watermarking model. Specifically, this invention constructs an auditory perception assessment model to fit non-differentiable objective evaluation metrics such as Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). Using the trained auditory perception assessment model, a differentiable perceptual loss function can be constructed to guide the online parameter updates of the audio watermarking model. This effectively overcomes the limitation of traditional MSE (Perceptual Evaluation of Speech Quality) in representing real auditory perception, ensuring that the trained audio watermarking model can generate watermarked audio with superior auditory perception quality. The following section combines... Figures 1-11 This invention provides a detailed description of the training of the audio watermark model, the audio watermark embedding method, and the apparatus thereof.

[0038] It should be noted in advance that the execution entity of the audio watermarking model training method provided in the subsequent embodiments of the present invention can be an electronic device with powerful data processing and deep learning model training capabilities, such as a data processing server, model training cluster, artificial intelligence computing center, or high-performance workstation deployed in the cloud. This electronic device typically includes a processor and a memory, and can implement the end-to-end training logic described in this embodiment by executing a computer program stored in the memory. Subsequent embodiments will use an electronic device as the execution entity for detailed description.

[0039] Figure 1 This is a flowchart illustrating the training method for the audio watermarking model provided by this invention, as shown below. Figure 1 As shown, including but not limited to the following steps: Step 11: Obtain a watermarked audio sample obtained by fusing an initial audio sample with a watermarked encoded sample.

[0040] Optionally, this embodiment provides a specific implementation method for a watermarked audio sample: First, obtain the amplitude spectrum features and phase spectrum features of an initial audio sample.

[0041] In this embodiment, the initial audio samples specifically refer to a large number of raw digital audio signals that are not embedded with any hidden watermark information and are used to train the network model. Their content can cover various forms of audio data such as speech segments, music, and ambient sounds.

[0042] In the field of deep learning audio processing, embedding hidden information directly into the time-domain waveform often struggles to balance fidelity and robustness. Therefore, this embodiment first performs time-frequency domain analysis on the initial audio samples in the one-dimensional time domain.

[0043] In practice, electronic devices can use time-frequency analysis algorithms such as Short Time Fourier Transform (STFT), Wavelet Transform, or Discrete Cosine Transform to obtain frequency domain characteristics.

[0044] Taking the short-time Fourier transform as an example, by performing frame segmentation, windowing (such as Hanning window, Hamming window, etc.), and fast Fourier transform on each initial audio sample, the one-dimensional time-domain signal is decomposed into a two-dimensional frequency-domain representation. Based on this, the magnitude of each frequency point in the complex spectrum data is calculated to extract the amplitude spectrum features, and the phase angle of each frequency point is calculated to extract the phase spectrum features.

[0045] Among them, the extracted amplitude spectrum features can intuitively reflect the energy distribution of the initial audio sample at different times and frequencies, and are the main carriers for embedding watermark information; while the phase spectrum features preserve the auditory coherence, beat and microscopic physical structure of the initial audio sample, providing key support for subsequent lossless audio reconstruction.

[0046] Next, the fused features obtained by fusing a randomly selected watermarked coded sample with the amplitude spectrum features are input into the audio watermarking model to obtain the modulation features output by the audio watermarking model.

[0047] Watermark-encoded samples refer to sequence data representing specific hidden information such as copyright ownership, identity authentication, and anti-counterfeiting tracking. During the training phase, this embodiment typically constructs a large watermark codebook library, from which a string of binary bit streams (e.g., a bit sequence containing 0s and 1s) is randomly selected. This stream can be converted into a multi-dimensional initial feature vector through linear mapping operations such as fully connected layers, serving as the watermark-encoded sample for this training.

[0048] Because watermark-encoded samples typically differ in dimensionality (e.g., a one-dimensional vector) from two-dimensional amplitude spectrum features (which include time and frequency dimensions), they cannot be directly input into the audio watermarking model. Therefore, this embodiment may first perform feature fusion between the watermark-encoded samples and the amplitude spectrum features.

[0049] Specifically, the fusion methods employed may include: First, determine the time dimension and frequency dimension of the amplitude spectrum feature. Then, expand the watermark-encoded sample along the time or frequency dimension (e.g., using Repeat or Expand operations) to match its feature dimension with the amplitude spectrum feature. Finally, combine the expanded watermark-encoded sample with the amplitude spectrum feature using tensor computation methods such as concatenate, add, or multiply to obtain the fused feature.

[0050] The fused features can then be input into the audio watermarking model to be trained. The audio watermarking model can be any deep neural network architecture with feature extraction and reconstruction capabilities, such as one of the following: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Transformer architecture, or a deep residual network that can effectively avoid gradient vanishing and preserve fine-grained features.

[0051] The core task of an audio watermarking model is to learn how to adaptively embed a watermark on amplitude spectrum features. After forward propagation calculations through multiple hidden layers within the audio watermarking model, a modulation feature is output. In practical applications, to ensure high fidelity of the initial audio sample, this modulation feature can be either the residual mask output by the audio watermarking model or the watermark modulation weight matrix.

[0052] A new amplitude spectrum representation containing watermark information, called the modulation feature, can be obtained by multiplying or weighting the watermark modulation weight matrix with the original amplitude spectrum features.

[0053] Finally, based on the modulation features and the phase spectrum features, the watermarked audio sample is reconstructed.

[0054] After obtaining the frequency-domain modulation features carrying the watermark information, this embodiment converts them back to a time-domain waveform signal to verify the actual perceived sound quality in subsequent steps. In this step, the modulation features are used as the target amplitude spectrum features and combined with the unaltered phase spectrum features retained in step 11. Specifically, the modulation features and phase spectrum features can be reverse-analyzed from the frequency domain to the time domain using audio reconstruction methods such as inverse short-time Fourier transform (iSTFT), Griffin-Lim algorithm, or vocoder based on neural networks. Because the true phase information of the initial audio sample is strictly preserved during the reconstruction process, phase distortion in the time sequence is avoided to the greatest extent, generating a time-domain watermarked audio sample with a complete physical structure.

[0055] Step 12: Input the perceptual audio pair composed of the watermarked audio sample and the initial audio sample into the auditory perception evaluation model, and obtain the auditory perception quality value output by the auditory perception evaluation model.

[0056] Evaluating auditory perception quality is a core evaluation step in achieving high-fidelity audio watermark embedding. Existing watermark generation models often rely solely on calculating the mean squared error or absolute error of the feature matrix between the original audio sample and the resulting watermarked audio sample for model training. However, this purely mathematical statistical numerical approximation has serious limitations: it fails to consider the nonlinear perceptual characteristics of the human auditory system (such as frequency domain masking and time domain masking effects). For example, small numerical variations in low-energy frequencies may result in noticeable noise to the human ear, while larger numerical variations in high-energy frequencies may be ignored.

[0057] To overcome this deficiency, this embodiment innovatively combines the reconstructed watermarked audio sample with the initial audio sample (e.g., by splicing and pairing in the channel dimension of the tensor) to construct a perceptual audio pair, and inputs it into a pre-built auditory perception evaluation model to objectively evaluate the auditory perception quality value using the auditory perception evaluation model.

[0058] It should be noted that the auditory perception evaluation model used in this embodiment is essentially a differentiable surrogate neural network, such as an evaluation network built based on a multi-layer Transformer.

[0059] In addition, the auditory perception assessment model is trained based on the real labels corresponding to the objective auditory evaluation indicators. That is, the auditory perception assessment model has been fully trained in the pre-training stage using massive amounts of audio data and the real labels corresponding to the objective auditory evaluation indicators.

[0060] Objective auditory evaluation metrics refer to precise but non-differentiable evaluation algorithms used in traditional signal processing, such as the Perceptual Evaluation of Speech Quality (PESQ) or Short-Time Objective Intelligibility (STOI). By fitting these non-differentiable objective evaluation metrics, the trained auditory perception evaluation model can output a continuous and differentiable auditory perception quality value during forward inference for the input perceptual audio pair, such as a comprehensive auditory perception score mapped between 0 and 1.

[0061] Step 13: Determine the perceptual loss function based on the auditory perception quality value, and update the model parameters of the audio watermarking model in combination with the perceptual loss function.

[0062] After obtaining an auditory perception quality value that accurately reflects human hearing evaluation, it can be transformed into a supervisory signal for model optimization. Specifically, a perceptual loss function can be constructed based on this auditory perception quality value, with the objective of maximizing the quality value or minimizing the difference between the quality value and the perfect score.

[0063] Since the auditory perception evaluation model is fully differentiable, the gradient signal of the perceptual loss function relative to the parameters of each network layer in the audio watermarking model can be accurately calculated during the backpropagation process of the network. Subsequently, using deep learning optimizers such as stochastic gradient descent and Adam, the model parameters such as weights and biases of the audio watermarking model are iteratively updated online along the direction of gradient descent. Through this training strategy, the audio watermarking model can actively learn how to hide watermark information in frequency bands that are not sensitive to the human ear during continuous training, thereby avoiding feature modifications that would degrade subjective listening experience.

[0064] The audio watermarking model training method provided by this invention introduces an auditory perception evaluation model pre-trained based on objective auditory evaluation indicators. A differentiable perceptual loss function is constructed in the end-to-end network training link, completely overcoming the technical bottleneck of traditional loss functions such as mean square error being severely deviating from subjective human auditory perception. During the training process of the audio watermarking model, the perceptual loss function can directly provide gradient signals that accurately reflect auditory quality, driving network parameters to evolve in a direction that conforms to the nonlinear auditory masking characteristics of humans. This not only achieves a high degree of unity between the training objective and real auditory perception in the underlying optimization logic, but also endows the audio watermarking model with the ability to intelligently avoid modifications to sensitive frequency bands in practical applications. Thus, while ensuring the effective embedding and extraction of watermark information, it greatly improves the naturalness, clarity, and concealment of the watermarked audio, effectively preventing perceptual distortion caused by deterioration in sound quality.

[0065] Based on the above embodiments, as an optional embodiment, the joint update mechanism of the audio watermark model parameters will be further described in detail below.

[0066] This embodiment takes into account that in the training of deep learning multi-task models, a single loss function often struggles to balance complex interrelationships. If only the perceptual loss function is relied upon during training, the model may lack basic physical feature constraints in the initial training phase, leading to slow convergence or causing the generated watermarked audio to deviate from the basic energy distribution of the initial audio. Therefore, this embodiment innovatively provides a strategy based on joint optimization of multi-dimensional loss functions.

[0067] Figure 2 This is a schematic diagram of the training process of the audio watermarking model combined with the perceptual loss function provided by the present invention, as shown below. Figure 2 As shown, updating the model parameters of the audio watermarking model using the perceptual loss function includes, but is not limited to, specific implementation steps: Step 21: Calculate the frequency domain reconstruction loss between the amplitude spectrum features of the watermarked audio sample and the amplitude spectrum features of the initial audio sample; Step 22: Input the watermark audio sample into the watermark extraction model and obtain the reconstructed watermark code output by the watermark extraction model; Step 23: Calculate the watermark decoding loss between the reconstructed watermark code and the watermark code sample; Step 24: The perceptual loss function, the frequency domain reconstruction loss, and the watermark decoding loss are fused and calculated to obtain the global loss function; Step 25: Use the global loss function to jointly update the model parameters of the audio watermarking model.

[0068] In this embodiment, in order to increase the basic physical constraints, the amplitude spectrum features of the reconstructed watermark audio sample are first obtained, and the amplitude spectrum features of the corresponding initial audio sample are extracted, and then the numerical difference between the two at the feature matrix level is calculated.

[0069] As a feasible implementation method, measurement algorithms such as mean square error or mean absolute error (MAE) can be used. For example, the mean square error of energy of the initial audio sample and the watermarked audio sample at each time frequency point can be calculated as the frequency domain reconstruction loss, thereby constraining the watermarked audio sample from undergoing severe distortion in the macroscopic spectral morphology and maintaining basic signal consistency.

[0070] After completing the forward processing at the embedding end, it is also necessary to ensure that the embedded watermark information can be accurately extracted. To this end, this embodiment inputs the reconstructed watermark audio sample into a pre-configured watermark extraction model.

[0071] Optionally, the watermark extraction model may include feature parsing structures such as a deep residual network with a depth of 18 and average pooling layers. Its main function is to reverse-engineer the hidden watermark information from audio features containing complex background signals. After forward propagation parsing of the watermark extraction model, the reconstructed watermark code output by it can be obtained. This reconstructed watermark code represents the bitstream or feature vector that the extraction end can actually recover from the watermarked audio sample under the current parameter state.

[0072] To measure the accuracy of the extracted watermark information, this embodiment further calculates the similarity between the output reconstructed watermark code and the original watermark code sample used as the embedding target, as the watermark decoding loss.

[0073] Considering that simple Euclidean distance (ED) may not be sensitive enough to the measurement of directional similarity when the feature dimension is high, this embodiment chooses to use cosine distance (CD) and other methods.

[0074] Specifically, the cosine similarity between the reconstructed watermark code and the watermark code sample is calculated, and the watermark decoding loss is constructed based on this similarity. For example, the loss value of the watermark decoding loss is obtained by subtracting the cosine similarity from 1, so as to accurately characterize the information matching degree between the two in the feature space.

[0075] Finally, the global loss function can be used to calculate the comprehensive gradient under the multi-task objective through the backpropagation algorithm, and the weights, biases and other model parameters of the audio watermarking model (which may also include the watermark extraction model under the joint training mechanism) can be synchronously iterated and jointly updated.

[0076] The audio watermarking model training method provided by this invention effectively overcomes the defect of model optimization skew caused by a single loss function by combining frequency domain reconstruction loss, watermark decoding loss, and perceptual loss function. Specifically, the frequency domain reconstruction loss ensures the basic physical fidelity of the watermarked audio samples, the watermark decoding loss ensures high accuracy in extracting watermarked encoded samples, and the perceptual loss function accurately compensates for the blind spots of pure mathematical statistical errors at the human subjective auditory level. This multi-objective joint optimization mechanism enables the audio watermarking model to find the optimal balance among the three mutually restrictive core indicators of high fidelity, high concealment, and high extraction accuracy, giving the trained audio watermarking model strong robustness in complex real-world industrial application scenarios.

[0077] Based on the above embodiments, as an optional embodiment, considering that in real digital communication environments, such as compression by social networking software, microphone recording, and channel transmission, audio signals may suffer from uncorrectable distortion and severe distortion, if the audio watermarking model is trained only in a complete data environment, it may cause it to reproduce interference and fail to effectively extract information when actually used. Therefore, this embodiment may selectively adopt the method of introducing dynamic interference simulation inside the audio watermarking model.

[0078] Figure 3 This is a schematic diagram of the process for obtaining the reconstructed watermark code provided by the present invention, as shown below. Figure 3 As shown, the step of inputting the watermark audio sample into the watermark extraction model and obtaining the reconstructed watermark code output by the watermark extraction model includes, but is not limited to, the following specific implementation steps: Step 31: Input the watermarked audio sample into the simulation attack layer of the watermark extraction model; Step 32: In the current training batch, randomly select a target attack method from the preset attack method set to interfere with the watermarked audio sample to obtain the attacked audio sample. Step 33: Input the attacked audio sample into the feature extraction layer of the watermark extraction model and output the reconstructed watermark code.

[0079] To adapt to complex distortion environments in advance during the end-to-end training phase of the model, this embodiment first feeds the reconstructed watermarked audio samples into the pre-module of the watermark extraction model, namely the simulation attack layer. This simulation attack layer is not a traditional learnable network weight layer used for feature extraction, but a perturbation module specifically designed to apply various mathematical and digital signal processing transformations to the input tensor. Its main function is to simulate the distortion effects that may occur in real channel transmission at the feature level or waveform level.

[0080] When training an audio watermarking model, data is typically fed in batches. When processing each current training batch, attack strategies are extracted from a pre-defined set of attack methods established within the system.

[0081] The preset set of attack methods covers a variety of common signal processing interference techniques, including but not limited to: random sampling point dropping and Gaussian noise (GN) injection operations at the time domain level, and high-pass filtering (HPF) or low-pass filtering (LPF) operations at the frequency domain level.

[0082] In this embodiment, a target attack method is randomly selected from the above attack method set through random number generation and other methods. This target attack method is then uniformly applied to all watermarked audio samples in the current training batch, thereby performing numerical interference processing on the original tensor data and finally generating attacked audio samples that simulate a real damaged environment.

[0083] To ensure that the audio watermarking model faces different challenges in each iteration and to prevent overfitting to a single attack, when the next training batch is executed, a different target attack method will be randomly switched to interfere with the data in the new batch.

[0084] After interference injection, the attacked audio sample with distorted information is further input into the core feature extraction layer of the watermark extraction model. This feature extraction layer can consist of multiple convolutional structures, such as a deep residual network with a depth of 18, and has corresponding pooling operations (such as average pooling layers) at the end. Its main function is to peel away irrelevant redundant interference layer by layer from the feature matrix containing complex background signals and various noise interferences (such as noisy amplitude spectrum features extracted after re-time-frequency transformation), and reverse analyze and recover the hidden feature weights. After forward propagation calculation by this feature extraction layer, the final output is a reconstructed watermark code that can survive in harsh environments.

[0085] The audio watermarking model training method provided by this invention effectively internalizes the channel distortion environment in the real world into an adversarial challenge during model training by embedding a simulation attack layer in the watermark extraction link and randomly applying different target attack methods in each training batch. This dynamic and random interference mechanism forces the feature extraction layer to continuously learn and refine the most anti-interference watermark feature representation under harsh conditions full of severe noise and signal distortion. This not only greatly improves the robustness of the trained audio watermarking model to complex signal processing attacks such as random sampling point dropping, various environmental noise injections, and filtering operations, but also enables the trained audio watermarking model to maintain a very high watermark extraction accuracy in actual complex industrial transmission links.

[0086] Considering that the human auditory system's perception is highly complex and multi-dimensional when evaluating the impact of audio watermarking on sound quality, if the audio watermarking model is trained based on only a single auditory evaluation index, it may cause auditory degradation in specific frequency bands or features when actually facing complex audio signals, and cannot comprehensively and objectively reflect the real auditory experience. Therefore, this embodiment chooses to use a multi-dimensional fusion auditory perception evaluation mechanism to determine the perception loss function.

[0087] Figure 4 This is a schematic diagram of the process for determining the perceptual loss function provided by the present invention, as shown below. Figure 4 As shown, determining the perceptual loss function based on the auditory perception quality value includes, but is not limited to, the following specific implementation steps: Step 41: Obtain the current quality evaluation score of the perceived audio pair under different auditory evaluation dimensions from the auditory perception quality value. The auditory evaluation dimensions include at least the speech quality perception dimension and the objective intelligibility dimension. Step 42: Obtain the target quality label corresponding to each of the perceived audio pairs in each auditory evaluation dimension; Step 43: Calculate the degree of quality deviation between each current quality evaluation score and the corresponding target quality label; Step 44: Perform a fusion calculation on the quality deviation degree under each of the auditory evaluation dimensions to obtain the perceptual loss function.

[0088] To comprehensively and accurately quantify the differences in human ear perception of watermarked audio, this embodiment first analyzes the auditory perception quality values ​​output by the auditory perception evaluation model. Since this auditory perception evaluation model (such as a network built based on multi-layer Transformers) is designed with multi-node or multi-channel output capabilities, it can simultaneously extract the current quality evaluation scores of the perceived audio pair under different auditory evaluation dimensions from the overall auditory perception quality values.

[0089] It should be noted that these auditory evaluation dimensions at least cover the speech quality perception dimension, which reflects the naturalness and overall clarity of the sound, and the objective comprehensibility dimension, which reflects the comprehensibility of the speech content.

[0090] Subsequently, the target quality label corresponding to each auditory evaluation dimension of the perceived audio pair is obtained. This target quality label typically refers to the true score obtained offline by using a traditional, non-differentiable objective auditory evaluation algorithm to calculate the initial audio sample and the reconstructed watermarked audio sample in the perceived audio pair.

[0091] Specifically, the speech quality perception dimension can be represented by the speech quality perception assessment value (PESQ), and the objective intelligibility dimension can be represented by the short-time objective intelligibility index (STOI). These real scores represent the objective evaluation standards of audio signals on specific auditory evaluation dimensions.

[0092] After obtaining the current quality assessment score predicted by the model and the target quality label used as a benchmark, the degree of quality deviation between each current quality assessment score and the corresponding target quality label is calculated. During this process, mathematical metrics such as mean squared error can be selectively used to calculate the difference loss between the predicted score and the actual PESQ label in the speech quality perception dimension, and the difference loss between the predicted score and the actual STOI label in the objective intelligibility dimension.

[0093] Finally, the quality deviations calculated under each auditory evaluation dimension are fused together. For example, this can be done by simply adding the values ​​or by assigning specific weight coefficients to the deviations of different auditory evaluation dimensions and then performing a weighted summation, thereby integrating the quality deviations of multiple dimensions into a unified scalar value to obtain the final perceptual loss function.

[0094] The audio watermarking model training method provided by this invention effectively internalizes the complex perception of overall sound quality and local content comprehensibility in the human auditory system into a direct driving force for model optimization by introducing a multi-dimensional evaluation mechanism, including speech quality perception and objective intelligibility dimensions, during the construction of the perceptual loss function. This multi-dimensional fusion loss constraint mechanism forces the audio watermarking model to maintain not only the naturalness of the macroscopic acoustic features of the audio but also strictly ensure the clarity and intelligibility of the microscopic speech information when embedding the watermark. This not only greatly improves the high-fidelity sound quality maintenance capability of the trained audio watermarking model in complex dynamic generation processes but also enables the trained audio watermarking model to output high-quality audio watermarks that closely match the multi-dimensional real hearing experience of humans in practical applications.

[0095] Since traditional speech quality assessment metrics are usually mathematically non-differentiable, directly using them in gradient descent-based neural network training may cause gradient chain breaks during backpropagation, preventing effective optimization of network parameters. As an alternative implementation, an end-to-end differentiable neural network surrogate model (i.e., an auditory perception assessment model) is constructed and pre-trained to fit these objective evaluation processes.

[0096] Figure 5 This is a schematic diagram of the training process of the auditory perception assessment model provided by the present invention, as shown below. Figure 5 As shown, the auditory perception evaluation model is obtained by iteratively executing the following pre-training steps until a preset convergence condition is met, including but not limited to the following specific implementation steps: Step 51: Obtain an evaluation training sample pair, which consists of any initial training audio sample and a watermarked audio sample obtained by adding a watermark to the initial training audio sample. Step 52: Input the evaluation training sample pair into the auditory perception evaluation model to be trained, and obtain the predicted value of the objective auditory evaluation index output by the auditory perception evaluation model to be trained. Step 53: Calculate the prediction difference loss between the predicted value of the objective auditory evaluation index and the actual label; Step 54: Update the model parameters of the auditory perception evaluation model to be trained based on the predicted difference loss.

[0097] Before training the auditory perception evaluation model, sufficient and high-quality data is prepared to teach it how to analyze differences in sound. This embodiment acquires any clean initial training audio sample and embeds hidden watermark features into it using a pre-trained watermarking network or basic algorithm, thereby generating a corresponding watermarked audio sample. Subsequently, these two samples are combined to construct the evaluation training sample pair used for training the evaluation model. The combination method can be through channel concatenation along the feature dimension; this paired sample structure allows the model to intuitively compare subtle waveform or spectral variations between the original audio and the watermarked audio.

[0098] Subsequently, the constructed evaluation training samples are input into the auditory perception evaluation model to be trained. The auditory perception evaluation model to be trained can be a deep learning architecture composed of multiple transformer network layers stacked together.

[0099] This embodiment utilizes the self-attention mechanism within the auditory perception evaluation model to fully extract the deep spatiotemporal correlation features between the input evaluation training samples, and predicts one or more predicted values ​​at the output node of the auditory perception evaluation model. These predicted values ​​represent the quantitative scores that the auditory perception evaluation model currently gives to the watermarked audio samples on specific objective auditory evaluation indicators (such as the degree of sound quality impairment).

[0100] To further measure the accuracy of the auditory perception assessment model's predictions, this embodiment determines the accuracy by calculating the prediction difference loss between the predicted values ​​of the objective auditory evaluation index and the actual labels.

[0101] The true label can be a baseline score pre-calculated offline using a standard and non-differentiable auditory assessment algorithm. Loss functions such as mean squared error or mean absolute error can be used to calculate the numerical distance between the output predicted value and the true label representing objective reality, thereby quantifying the current prediction bias as the prediction difference loss.

[0102] After obtaining the predicted difference loss, the gradient of the model parameters of the auditory perception evaluation model can be calculated using the backpropagation algorithm, and the internal weight matrix can be iteratively updated by the optimizer.

[0103] Steps 51 to 54 above will be executed iteratively until a preset convergence condition is met. The preset convergence condition may refer to the predicted difference loss decreasing and stabilizing within a very small threshold range, or reaching a pre-set maximum number of training rounds. When this preset convergence condition is met, it indicates that the auditory perception assessment model has the ability to highly fit traditional complex auditory evaluation algorithms.

[0104] The audio watermarking model training method provided by this invention effectively transforms the originally black-box, mathematically non-differentiable auditory evaluation process into an end-to-end differentiable computational graph by designing and pre-training a dedicated neural network surrogate model. This data-driven surrogate evaluation mechanism forces the model to fully learn and internalize the complex laws of human auditory perception during the pre-training stage. This not only greatly improves the accuracy and generalization ability of the trained auditory perception evaluation model in scoring various types of distorted audio, but also provides accurate, stable, and fully differentiable gradient signals for directly guiding the parameter updates of the backbone audio watermarking model, fundamentally clearing away the technical obstacles to model optimization based on auditory perception.

[0105] When extracting various objective listening evaluation indicators as supervision signals for training audio watermarking models, different indicators often have completely different physical dimensions and numerical distribution ranges. If these raw scores without uniform scale processing are directly input into the audio watermarking model as labels, it may cause the model to over-focus on indicators with larger numerical ranges during the learning process, while ignoring indicators with smaller numerical ranges, leading to an imbalance in network gradient updates during multi-task learning. Therefore, this embodiment uses a method of numerical transformation and interval mapping of specific indicators to determine the true labels.

[0106] Figure 6 This is a schematic diagram illustrating the process of constructing real labels for training the auditory perception assessment model provided by the present invention, as shown below. Figure 6 As shown, the true label is determined based on the following steps, including but not limited to the following specific implementation steps: Step 61: Calculate the speech quality perception evaluation value and short-term objective intelligibility value of the evaluation training sample pair respectively.

[0107] In this embodiment, the constructed evaluation training sample pairs, consisting of initial training audio samples and watermarked audio samples, will be scored offline.

[0108] Specifically, traditional objective audio evaluation algorithms can be used to calculate the Perceptual Speech Quality (PESQ) value and the Objective Intelligibility (STOI) value between the initial training audio samples and the watermarked audio samples, respectively.

[0109] As a preferred embodiment, the PESQ value is mainly used to measure the overall subjective auditory quality degradation of the audio signal after watermark embedding, while the STOI value focuses on assessing whether the semantic features contained in the audio signal can still be clearly identified. These two constitute the core data foundation for comprehensively evaluating the concealment of audio watermarks.

[0110] It should be noted that those skilled in the art will understand that, in addition to the PESQ and STOI values ​​mentioned above, the perceptual evaluation values ​​of speech quality in the embodiments of the present invention can be replaced with or further combined with other objective indicators used to measure audio quality, such as Perceptual Objective Listening Quality Analysis (POLQA) values, Virtual Speech Quality Objective Listener (ViSQOL) values, Perceptual Evaluation of Audio Quality (PEAQ) values, Segmental Signal-to-Noise Ratio (SegSNR), Log-Spectral Distance (LSD), or Cepstral Distance (CD), etc. Similarly, the STOI value can be replaced with or further combined with other indicators used to measure speech intelligibility, such as Extended Short-Time Objective Intelligibility (ESTOI) values, Speech Intelligibility Index (SII), or Normalized Covariance Measure (NCM), etc. Using any one or more of the above-mentioned indicators for subsequent numerical mapping and model training falls within the scope of the implementation scheme of this embodiment.

[0111] Step 62: Perform numerical transformation processing on the speech quality perception evaluation value, and map its numerical range to the same target interval as the short-term objective intelligibility value to obtain the mapped evaluation value.

[0112] In practical evaluation systems, the typical output range of PESQ values ​​is between -0.5 and 4.5, while the output range of STOI values ​​is strictly limited to 0 to 1. To eliminate this significant difference in units, this embodiment performs linear or nonlinear numerical transformation on the obtained PESQ values. For example, using a linear normalization formula, the original PESQ values ​​are uniformly increased by 0.5 and then divided by 5, thereby accurately mapping their numerical range to the target interval of 0 to 1, obtaining a mapped evaluation value that is completely consistent with the STOI value scale. This processing makes the scores of the two evaluation dimensions directly comparable within the same numerical space.

[0113] In practical evaluation systems, the typical output range of PESQ values ​​is between -0.5 and 4.5, while the output range of STOI values ​​is strictly limited to 0 to 1. To eliminate this significant difference in units, the obtained PESQ values ​​can be subjected to linear or nonlinear numerical transformations. For example, selectively using a linear normalization formula, the original PESQ values ​​can be uniformly increased by 0.5 and then divided by 5, thereby accurately mapping their numerical range to the target interval of 0 to 1, resulting in a mapped evaluation value that is completely consistent with the STOI value scale. This processing makes the scores of the two evaluation dimensions directly comparable within the same numerical space.

[0114] Step 63: The mapping evaluation value and the short-term objective understandability value are jointly determined as the true label.

[0115] After standardizing the numerical scale, the PESQ values ​​processed by the normalization standard can be combined with the originally calculated STOI values. For example, these two values, which are in the range of 0 to 1, can be used to construct a label vector containing two elements, which can then be used to determine the true label that the auditory perception evaluation model needs to fit during the pre-training phase. This multi-dimensional true label will be directly used to guide the joint regression training of the agent network output nodes.

[0116] The audio watermarking model training method provided by this invention effectively unifies multi-dimensional auditory evaluation standards with vastly different dimensions to the same feature scale by introducing a numerical transformation and mapping alignment mechanism for specific evaluation indicators before constructing supervised labels. This label construction mechanism based on numerical range mapping forces the auditory perception evaluation model to give balanced attention to speech quality perception and short-term objective intelligibility during multi-task joint learning. This not only greatly improves the convergence speed and numerical stability of the trained evaluation model when predicting multiple auditory indicators, but also makes the final supervised signal, which serves as the true label, more scientific and balanced, effectively avoiding network optimization deviation caused by differences in evaluation scales from the underlying data structure.

[0117] In the pre-training stage of the auditory perception evaluation model, if the audio watermarking model that has not been trained is randomly initialized to generate training data, the generated audio samples may be completely meaningless extreme noise, causing the auditory perception evaluation model to be unable to learn the specific and subtle distortion features brought about by the real watermark embedding. Therefore, this embodiment adopts a linkage mechanism of phased warm-up and data feedback to obtain high-quality evaluation training sample pairs.

[0118] Figure 7 This is a schematic diagram of the process for obtaining evaluation training sample pairs provided by the present invention, as shown below. Figure 7As shown, the steps for obtaining the evaluation training sample pairs include, but are not limited to, the following specific implementation steps: Step 71: When the training rounds of the audio watermark model reach the preset number of rounds, obtain the audio watermark model after preliminary training. Step 72: Using the pre-trained audio watermarking model, process the initial training audio samples in the training dataset to obtain watermarked audio samples corresponding to the initial training audio samples. Step 73: Combine the initial training audio samples and the corresponding watermarked audio samples to obtain the evaluation training sample pair used to train the auditory perception evaluation model.

[0119] In the initial training phase of the overall network architecture, the backbone audio watermarking model is typically trained independently and iteratively. To determine whether the audio watermarking model possesses adequate data generation capabilities, this embodiment sets a preset number of training rounds, for example, 50 rounds. When the system monitors that the audio watermarking model has reached this preset number of training rounds, it indicates that the audio watermarking model has passed the random exploration phase and has initially acquired the ability to embed hidden information while maintaining a certain level of audio quality. At this point, the network weights for this phase are saved or temporarily fixed to obtain the preliminarily trained audio watermarking model.

[0120] Subsequently, the pre-trained audio watermarking model was used as a generator specifically for creating distorted samples. A large training dataset (Dataset) identical to the backbone network training dataset was retrieved, and for each initial training audio sample, a watermark code containing specific bit information was randomly matched. Using the pre-trained audio watermarking model, a complete forward inference process was performed on these initial training audio samples, sequentially completing time-frequency feature analysis, feature fusion modulation, and final waveform reconstruction. This resulted in batches of watermarked audio samples corresponding to each initial training audio sample, containing microscopic defects in the actual network.

[0121] It should be noted that after generating the large batch of distortion data, this embodiment strictly follows the one-to-one mapping relationship of data origins, pairing and combining the initial training audio samples with their corresponding watermarked audio samples. In specific implementation, a method of splicing and aligning along the channel dimension of the tensor can be further adopted, so that the original reference features and the distortion features to be evaluated can be presented side by side in the same data structure, ultimately obtaining evaluation training sample pairs for specifically training the auditory perception evaluation model.

[0122] The audio watermarking model training method provided by this invention introduces a phased warm-up strategy into the overall architecture. It uses the pre-trained audio watermarking model to generate high-quality evaluation training sample pairs in reverse, effectively using the specific defect patterns generated by the real network at a specific training stage as the learning material for the auditory perception evaluation model. This dynamic, closed-loop strategy for preparing training data for the auditory perception evaluation model forces the model to stop aimlessly fitting random noise and instead perform highly targeted deep learning for the specific distortion types generated by the audio watermarking model at the current stage. This not only greatly improves the scoring accuracy and sensitivity of the trained auditory perception evaluation model in real model adversarial scenarios, but also enables the trained auditory perception evaluation model to provide more accurate perceptual gradient signals.

[0123] Based on the above embodiments, as an optional embodiment, the step of reconstructing the watermarked audio sample based on the modulation features and the phase spectrum features includes: The modulation features are determined as the target amplitude spectrum features; The target amplitude spectrum features and phase spectrum features are converted from the frequency domain to the time domain using the inverse short-time Fourier transform to obtain the watermarked audio sample.

[0124] After the audio watermarking model completes forward inference, its output modulation feature is essentially a frequency domain energy distribution matrix that has been adaptively adjusted by the neural network and has incorporated the hidden watermark sequence. In this embodiment, before signal reconstruction, this modulation feature is first physically defined as the target amplitude spectrum feature. This means that in the subsequent signal reconstruction stage, this target amplitude spectrum feature will completely replace the original amplitude spectrum of the initial audio sample, serving as the basic frequency domain energy benchmark for constructing the new audio signal and carrying all the embedded invisible watermark information.

[0125] After identifying the target amplitude spectrum features containing watermark information, they are combined with the initial phase spectrum features to recover the micro-temporal and periodic structure of the audio. This includes: extracting the phase spectrum features separated and retained in the initial stage, and using the Inverse Short-Time Fourier Transform (iSTFT) algorithm to perform frequency domain to time domain conversion processing on the target amplitude spectrum features and phase spectrum features.

[0126] Specifically, the target amplitude spectrum, which represents energy, and the phase spectrum, which represents structure, are recombined into complete complex spectrum data. Then, through low-level signal processing operations such as inverse discrete Fourier transform and overlap-add, the two-dimensional frequency domain feature matrix is ​​smoothly inversely converted into a one-dimensional time domain sound waveform, ultimately obtaining a watermarked audio sample that is audible to humans and has a complete physical structure.

[0127] As an optional embodiment, an alternative method is provided for obtaining a watermarked audio sample obtained by fusing an initial audio sample with a watermarked encoded sample, which mainly includes, but is not limited to: Obtain the amplitude spectrum features and phase spectrum features of the initial audio sample; Determine the time dimension and frequency dimension of the amplitude spectrum feature; The watermarked coded sample is extended along the time dimension to obtain an extended watermark feature, and the time dimension of the extended watermark feature matches the time dimension of the amplitude spectrum feature. The extended watermark feature and the amplitude spectrum feature are concatenated along the direction of the frequency dimension to obtain the fused feature; The fused features are input into the audio watermarking model to obtain the modulation features output by the audio watermarking model; The watermarked audio sample is reconstructed based on the modulation features and the phase spectrum features.

[0128] The foregoing embodiments have detailed the acquisition of amplitude spectrum features and phase spectrum features of the initial audio sample provided by the embodiments of the present invention. Then, after determining the time dimension and frequency dimension of the amplitude spectrum feature, the watermark-encoded sample is extended along the time dimension to obtain an extended watermark feature, the time dimension of which matches the time dimension of the amplitude spectrum feature. The extended watermark feature and the amplitude spectrum feature are then concatenated along the direction of the frequency dimension to obtain a fused feature. The fused feature is input to an audio watermarking model to obtain the modulation features output by the audio watermarking model. This is at least one optional implementation method for feature fusion using a time-dimension-based frame-level extension and a frequency-dimension-based channel concatenation mechanism. Based on this, this embodiment further elaborates on how to fuse a randomly selected watermark-encoded sample with the amplitude spectrum feature, which may include, but is not limited to, the following specific implementation steps: Determine the time dimension and frequency dimension of the amplitude spectrum feature; The watermarked coded sample is extended along the time dimension to obtain an extended watermark feature, and the time dimension of the extended watermark feature matches the time dimension of the amplitude spectrum feature. The extended watermark feature and the amplitude spectrum feature are spliced ​​together along the direction of the frequency dimension to obtain the fused feature.

[0129] First, the extracted two-dimensional spectrum matrix is ​​analyzed using tensor shape analysis. The amplitude spectrum feature typically includes a time dimension representing the number of audio frames, and a frequency dimension representing the different frequency channels or feature dimensions within each frame. The specific sizes of these two dimensions can be extracted using tensor analysis functions.

[0130] The watermarked encoding sample is typically a fixed-length discrete bitstream or a one-dimensional feature vector. To ensure that this one-dimensional feature vector is perfectly aligned with the two-dimensional amplitude spectrum feature on the time axis, this embodiment expands the watermarked encoding sample along the time dimension direction as determined above. In practice, operations such as retiling or tensor expansion in deep learning frameworks can be used to perform frame-level copying of the watermark encoding according to the frame number of the amplitude spectrum feature. After expansion, the original one-dimensional vector with only the feature dimension is stretched into a two-dimensional matrix with both time and feature dimensions, thus obtaining the expanded watermark feature. This ensures that the expanded watermark feature achieves a perfect size match with the amplitude spectrum feature on the time axis.

[0131] After completing the size alignment in the time dimension, the two can be merged in another orthogonal dimension, including merging the extended watermark feature with the original amplitude spectrum feature along the direction of the frequency dimension (i.e., the feature channel dimension).

[0132] For example, tensor splicing can be used to combine the original frequency feature channels with the expanded watermark feature channels to obtain a fused feature. This fused feature preserves the complete time-frequency energy distribution of the original audio on independent channels, while also tightly attaching the same watermark information features to each time frame.

[0133] The audio watermarking model training method provided by this invention effectively maps and flattens a one-dimensional discrete watermark sequence into a two-dimensional continuous acoustic feature space by introducing independent extension processing for the time dimension and a splicing method for the frequency dimension at the network input. This allows the hidden information to be uniformly and continuously distributed throughout each time frame of the audio without destroying the independence of the original frequency features. This not only greatly improves the ability of the trained audio watermarking model to capture redundant watermarks when dealing with local audio clipping, but also enables the audio watermarking model to learn a highly robust modulation strategy from more standardized time-frequency joint features.

[0134] Figure 8 This is a flowchart illustrating the audio watermark embedding method provided by the present invention, as shown below. Figure 8 As shown, the main implementation steps include, but are not limited to, the following: Step 81: Based on the training method of the audio watermarking model, train the audio watermarking model to obtain the audio watermarking model. The training method for the audio watermarking model can be any of the methods provided in the above embodiments.

[0135] Step 82: Obtain the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark encoding to be embedded. Step 83: Perform feature fusion between the target watermark code and the amplitude spectrum feature to obtain the target fused feature; Step 84: Input the target fusion feature into the audio watermarking model to obtain the target modulation feature output by the audio watermarking model; Step 85: Based on the target modulation features and the phase spectrum features, reconstruct the target watermark audio embedded with the target watermark code.

[0136] Considering that directly using a basic model that has not been fully jointly optimized for watermark embedding in actual business may result in low extraction rate of the generated audio after transmission through complex channels, or produce obvious noise that can be perceived by the human ear, it cannot meet the stringent requirements for high fidelity and high robustness in industrial applications. Therefore, this invention adopts the multi-stage joint optimization training mechanism provided in any of the aforementioned embodiments to obtain a usable audio watermark model, and then uses the audio watermark model for forward watermark embedding.

[0137] First, the specific implementation steps for end-to-end training and actual embedding of this audio watermarking model are as follows: Step a): After windowing and short-time Fourier transform processing of the initial audio sample, obtain the amplitude spectrum features and phase spectrum features of the initial audio sample. The feature dimension size is 1. ,in f This indicates the time dimension (i.e., the number of feature frames). d This represents the frequency dimension (i.e., the feature dimension).

[0138] Step b): Randomly select a watermark-encoded sample from the watermark codebook library and extend it along the time dimension of the amplitude spectrum feature (e.g., frame-level copying). The size of the extended watermark-encoded sample (i.e., the extended watermark feature) is... ,in This indicates the number of bits in the watermark encoding sample.

[0139] Step c): Concatenate the amplitude spectrum feature and extended watermark feature obtained in steps a) and b) along the direction of the frequency dimension to obtain the fused feature, the feature size of which is... .

[0140] Step d): Input the obtained fused features into the audio watermarking model, such as a Deep Residual Network (DRN) with a network depth of 34, and obtain the watermark modulation weight matrix output by the audio watermarking model.

[0141] Step e): Multiply the amplitude spectrum feature from step a) and the watermark modulation weight matrix from step d) to obtain the modulation feature; Step f): Combine the phase spectrum features from step a) and the modulation features from step e), and use the inverse short-time Fourier transform to perform frequency domain to time domain conversion processing to reconstruct the watermarked audio sample.

[0142] Step g): Using the watermarked audio sample obtained in step f) and the initial audio sample in step a), calculate the mean square error between the amplitude spectrum features as the frequency domain reconstruction loss.

[0143] Step h): Input each watermarked audio sample into the simulation attack layer in the watermark extraction model. In the current training batch, randomly select a target attack method from the preset attack method set to interfere with all audio samples in the current training batch, and obtain the attacked audio sample.

[0144] Step i): The attacked audio sample obtained in step h) is first input into a deep residual network with a network depth of 18, and then passed through an average pooling layer to output the reconstructed watermark code. Then, the extracted reconstructed watermark code is calculated. and the input watermark code sample The cosine distance between the two watermarks is used to obtain the watermark decoding loss between the two watermark encodings. : .

[0145] Step j): The loss functions from steps g) and i) above are fused to obtain the global loss function for the audio watermarking model and the watermark extraction network. Then, this global loss function is used to jointly update the model parameters based on the data in the current training batch.

[0146] Step k): For the next current training batch, repeat steps a) to j). When continuing with step h), randomly switch to a different target attack method to interfere with all audio samples in the current training batch, ensuring that all audio samples in the current training batch suffer the same target attack. Similarly, use the calculated global loss function to update the network gradient based on the data in the current training batch.

[0147] Step 1): After the audio watermarking model has undergone a preset number of training rounds (e.g., 50 rounds) using steps a) to k), a pre-trained audio watermarking model is obtained. Then, this pre-trained audio watermarking model is used to process all initial training audio samples in the training dataset using steps a) to f), obtaining their corresponding watermarked audio samples. During this process, a watermark encoding sample is randomly selected for each initial training audio sample.

[0148] Step m): Take the initial training audio samples obtained in step l). and the corresponding watermarked audio sample By combining the samples, we can obtain the evaluation training sample pairs. The PESQ and STOI values ​​were calculated separately. Since the PESQ value ranges from -0.5 to 4.5 and the STOI value ranges from 0 to 1, the calculated PESQ value was first transformed to map it to the target range of 0 to 1 to obtain the mapped evaluation value. The mapped evaluation value and the STOI value were then used together to determine the true label.

[0149] Step n): Pair this evaluation training sample with... After channel concatenation, the input is fed into the auditory perception evaluation model to be trained, such as a 6-layer Transformer network. The output of this network consists of two nodes: one node corresponds to the predicted mapping evaluation value, and the other node corresponds to the predicted STOI value.

[0150] Step o): Calculate the predicted difference loss between the predicted mapping evaluation value and STOI value of the objective auditory evaluation index obtained through step n) and the true label determined in step m).

[0151] Step p): Based on the predicted difference loss and evaluation training sample pairs calculated in step o), update the model parameters of the auditory perception evaluation model to be trained until the preset convergence condition is met.

[0152] Step q): Add the auditory perception assessment model trained in step p) to step j), while fixing the model parameters of the auditory perception assessment model.

[0153] Step r): Load the audio watermarking model preliminarily trained in step l), and obtain the auditory perception quality value output by the auditory perception evaluation model. Determine the perceptual loss function based on the auditory perception quality value, and add the perceptual loss function to the global loss function calculation in step j). Use the fused global loss function to complete the joint update of model parameters based on the data in the current training batch.

[0154] Step s): Repeat steps a) to k) above, and replace the global loss function in step j) with the global loss function containing the perceptual loss function in step r). After iterating through the preset number of rounds, the audio watermark model is finally trained.

[0155] After obtaining a high-quality audio watermark model through the above steps, the electronic device can then perform the actual audio watermark embedding, specifically as follows: First, the electronic device acquires the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark code to be embedded. Subsequently, the target watermark code and the amplitude spectrum feature are fused to obtain the target fused feature; Next, the target fusion feature is input into the audio watermarking model to obtain the target modulation feature output by the audio watermarking model; Finally, based on the target modulation features and the phase spectrum features, the target watermark audio embedded with the target watermark code is reconstructed.

[0156] The audio watermarking embedding method provided in this invention, through the above-mentioned multi-stage joint optimization network training process, not only internalizes the nonlinear auditory masking law of humans and the complex real channel distortion into the driving force of network evolution during the model training stage, but also ensures the absolute transparency of the target watermark audio in the subjective listening experience of the human ear and the extremely high robustness in the physical transmission process during actual embedding, significantly improving the overall service quality and security of audio watermarking technology in industrial environments.

[0157] Figure 9 This is a schematic diagram of the structure of the training device for the audio watermarking model provided by the present invention, as shown below. Figure 9 As shown, it mainly includes, but is not limited to: The audio reconstruction module 91 is used to obtain a watermarked audio sample obtained by fusing an initial audio sample with a watermarked encoded sample. The perception evaluation module 92 is used to input the perception audio pair composed of the watermarked audio sample and the initial audio sample into the auditory perception evaluation model, and obtain the auditory perception quality value output by the auditory perception evaluation model; wherein, the auditory perception evaluation model is trained based on the real labels corresponding to the objective auditory evaluation index; The model update module 93 is used to determine the perceptual loss function based on the auditory perception quality value, and update the model parameters of the audio watermark model in combination with the perceptual loss function.

[0158] It should be noted that the audio watermarking model training device provided by the present invention can execute the audio watermarking model training method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0159] Figure 10 This is a schematic diagram of the audio watermark embedding device provided by the present invention, as shown below. Figure 10 As shown, it mainly includes, but is not limited to: The model training module 101 is used to train the audio watermark model based on the training method of the audio watermark model provided in any of the above embodiments to obtain the audio watermark model. The information acquisition module 102 is used to acquire the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark code to be embedded. Feature fusion module 103 is used to fuse the target watermark code with the amplitude spectrum feature to obtain the target fused feature; Modulation processing module 104 is used to input the target fusion feature into the audio watermarking model and obtain the target modulation feature output by the audio watermarking model; The audio reconstruction module 105 is used to reconstruct the target watermark audio embedded with the target watermark code based on the target modulation features and the phase spectrum features.

[0160] It should be noted that the audio watermark embedding device provided by the present invention can execute the audio watermark embedding method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0161] The audio watermark embedding device provided by this invention constructs perceptual loss through a differentiable proxy model and directly uses the listening score as the optimization target. This overcomes the shortcomings of traditional mean square error in representing the nonlinear characteristics of the human ear, significantly improves the naturalness, fidelity and concealment of the watermarked audio, and greatly enhances the robustness of the model against various complex channel distortion attacks, ensuring the accuracy of watermark extraction.

[0162] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 11As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 may call logical instructions in the memory 1130 to execute the training method or the audio watermarking embedding method of the audio watermarking model provided in any of the above embodiments.

[0163] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the training method of the audio watermarking model or the audio watermarking embedding method provided in any of the above embodiments.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the training method or audio watermark embedding method of the audio watermark model provided in any of the above embodiments.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an audio watermarking model, characterized in that, include: A watermarked audio sample is obtained by fusing an initial audio sample with a watermarked encoded sample. The perceptual audio pair composed of the watermarked audio sample and the initial audio sample is input into the auditory perception evaluation model to obtain the auditory perception quality value output by the auditory perception evaluation model; wherein, the auditory perception evaluation model is trained based on the real labels corresponding to the objective auditory evaluation index; Based on the auditory perception quality value, a perceptual loss function is determined, and the model parameters of the audio watermarking model are updated in combination with the perceptual loss function.

2. The training method for the audio watermarking model according to claim 1, characterized in that, The step of updating the model parameters of the audio watermarking model by incorporating the perceptual loss function includes: Calculate the frequency domain reconstruction loss between the amplitude spectrum features of the watermarked audio sample and the amplitude spectrum features of the initial audio sample; The watermark audio sample is input into the watermark extraction model to obtain the reconstructed watermark code output by the watermark extraction model; Calculate the watermark decoding loss between the reconstructed watermark code and the watermark code sample; The global loss function is obtained by fusing the perceptual loss function, the frequency domain reconstruction loss, and the watermark decoding loss. The model parameters of the audio watermarking model are jointly updated using the global loss function.

3. The training method for the audio watermarking model according to claim 2, characterized in that, The step of inputting the watermark audio sample into the watermark extraction model and obtaining the reconstructed watermark code output by the watermark extraction model includes: The watermarked audio sample is input into the simulation attack layer of the watermark extraction model; In the current training batch, a target attack method is randomly selected from the preset set of attack methods to interfere with the watermarked audio sample, resulting in an attacked audio sample. The attacked audio sample is input into the feature extraction layer of the watermark extraction model, and the reconstructed watermark code is output.

4. The training method for the audio watermarking model according to claim 1, characterized in that, The determination of the perceptual loss function based on the auditory perception quality value includes: From the auditory perception quality value, obtain the current quality evaluation score of the perceived audio pair under different auditory evaluation dimensions, wherein the auditory evaluation dimensions include at least the speech quality perception dimension and the objective intelligibility dimension; Obtain the target quality label corresponding to each of the auditory evaluation dimensions for the perceived audio pair; Calculate the degree of quality deviation between each current quality evaluation score and the corresponding target quality label; The perceptual loss function is obtained by fusing and calculating the degree of quality deviation under each of the aforementioned auditory evaluation dimensions.

5. The training method for the audio watermarking model according to claim 1, characterized in that, The auditory perception evaluation model is obtained by iteratively performing the following pre-training steps until a preset convergence condition is met: Obtain an evaluation training sample pair, which consists of any initial training audio sample and a watermarked audio sample obtained by adding a watermark to the initial training audio sample. The evaluation training sample pairs are input into the auditory perception evaluation model to be trained, and the predicted values ​​of the objective auditory evaluation index output by the auditory perception evaluation model to be trained are obtained. Calculate the prediction difference loss between the predicted value of the objective auditory evaluation index and the actual label; The model parameters of the auditory perception evaluation model to be trained are updated based on the predicted difference loss.

6. The training method for the audio watermarking model according to claim 5, characterized in that, The true label is determined based on the following steps: Calculate the speech quality perception evaluation value and short-term objective intelligibility value for the evaluation training sample pairs respectively; The speech quality perception evaluation value is subjected to numerical transformation processing to map its numerical range to the same target interval as the short-term objective intelligibility value, thus obtaining the mapped evaluation value. The mapping evaluation value and the short-term objective understandability value are jointly determined as the true label.

7. The training method for the audio watermarking model according to claim 5, characterized in that, The step of obtaining the evaluation training sample pairs includes: When the training rounds of the audio watermarking model reach a preset number of rounds, the pre-trained audio watermarking model is obtained. Using the pre-trained audio watermarking model, the initial training audio samples in the training dataset are processed to obtain watermarked audio samples corresponding to the initial training audio samples. The initial training audio samples and the corresponding watermarked audio samples are combined to obtain the evaluation training sample pair used to train the auditory perception evaluation model.

8. The training method for the audio watermarking model according to claim 1, characterized in that, The process of obtaining a watermarked audio sample by fusing an initial audio sample with a watermarked encoded sample includes: Obtain the amplitude spectrum features and phase spectrum features of the initial audio sample; Determine the time dimension and frequency dimension of the amplitude spectrum feature; The watermarked coded sample is extended along the time dimension to obtain an extended watermark feature, and the time dimension of the extended watermark feature matches the time dimension of the amplitude spectrum feature. The extended watermark feature and the amplitude spectrum feature are concatenated along the direction of the frequency dimension to obtain the fused feature; The fused features are input into the audio watermarking model to obtain the modulation features output by the audio watermarking model; The watermarked audio sample is reconstructed based on the modulation features and the phase spectrum features.

9. The training method for the audio watermarking model according to claim 8, characterized in that, The process of reconstructing the watermarked audio sample based on the modulation features and the phase spectrum features includes: The modulation features are determined as the target amplitude spectrum features; The target amplitude spectrum features and phase spectrum features are converted from the frequency domain to the time domain using the inverse short-time Fourier transform to obtain the watermarked audio sample.

10. An audio watermark embedding method, characterized in that, include: An audio watermark model is trained based on the training method of the audio watermark model according to any one of claims 1-9; Obtain the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark encoding to be embedded; The target watermark code and the amplitude spectrum feature are fused to obtain the target fused feature; The target fusion feature is input into the audio watermarking model to obtain the target modulation feature output by the audio watermarking model; Based on the target modulation features and the phase spectrum features, the target watermark audio embedded with the target watermark code is reconstructed.

11. A training device for an audio watermarking model, characterized in that, include: The audio reconstruction module is used to obtain a watermarked audio sample obtained by fusing an initial audio sample with a watermarked encoded sample. The perception evaluation module is used to input the perception audio pair composed of the watermarked audio sample and the initial audio sample into the auditory perception evaluation model, and obtain the auditory perception quality value output by the auditory perception evaluation model; wherein, the auditory perception evaluation model is trained based on the real labels corresponding to the objective auditory evaluation index; The model update module is used to determine the perceptual loss function based on the auditory perception quality value, and to update the model parameters of the audio watermarking model in combination with the perceptual loss function.

12. An audio watermark embedding device, characterized in that, include: The model training module is used to train an audio watermark model based on the training method of the audio watermark model according to any one of claims 1-9. The information acquisition module is used to acquire the amplitude spectrum features and phase spectrum features of the audio to be processed, as well as the target watermark encoding to be embedded. The feature fusion module is used to fuse the target watermark code with the amplitude spectrum feature to obtain the target fused feature; A modulation processing module is used to input the target fusion feature into the audio watermarking model and obtain the target modulation feature output by the audio watermarking model. The audio reconstruction module is used to reconstruct the target watermark audio embedded with the target watermark code based on the target modulation features and the phase spectrum features.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the audio watermarking model as described in any one of claims 1 to 9, or the audio watermarking embedding method as described in claim 10.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the audio watermarking model as described in any one of claims 1 to 9, or the audio watermarking embedding method as described in claim 10.

Images