A deep learning-based audio watermark embedding and extraction method

Abstract

The application provides a kind of audio watermark embedding and extraction method based on deep learning, comprising: the discrete wavelet transform is carried out to the audio signal to be processed, and low-frequency sub-signal and high-frequency sub-signal are obtained;Based on the pre-trained watermark embedding model, the watermark information to be embedded is embedded into the high-frequency sub-signal, and the high-frequency sub-signal embedded with watermark is obtained;The watermark embedding model adjusts the received high-frequency sub-signal with attention, then splices and embeds the watermark information in channel dimension, and implicitly encodes the watermark information into the time-frequency structure of the high-frequency sub-signal through convolution operation and jump connection;The high-frequency sub-signal embedded with watermark and the low-frequency sub-signal are inversely discrete wavelet transformed, and the audio signal with watermark is obtained.The application solves the problem that the watermark embedding method in the prior art has poor concealment, affects the audio quality, and is easily attacked by signal processing, thereby affecting its reliability and effectiveness.

Description

Technical Field

[0001] This invention belongs to the field of audio processing technology, and in particular relates to a method for embedding and extracting audio watermarks based on deep learning. Background Technology

[0002] With the widespread application of audio data in multimedia communication, digital entertainment, online education, and remote work, ensuring its security and integrity has become an increasingly important issue that urgently needs to be addressed. The illegal recording, alteration, and dissemination of audio data infringes copyright, disrupts market order, and even poses a risk of leaking important audio files. Therefore, an efficient and reliable audio protection mechanism has become a crucial research topic in the digital audio field.

[0003] Digital watermarking technology embeds specific watermark information (such as copyright notices, user IDs, timestamps, etc.) into audio files without significantly affecting the original audio's auditory quality. This embedding method is difficult to detect and can be extracted using specific algorithms when needed, thus achieving multiple purposes such as copyright protection, content integrity verification, information tracking, and anti-counterfeiting authentication.

[0004] However, traditional audio watermarking embedding methods, such as the least significant bit method, echo hiding method, and frequency domain method, each have their own advantages and disadvantages, and all have different deficiencies in terms of concealment and robustness. In recent years, deep learning, with its powerful feature extraction, pattern recognition, and automatic learning capabilities, has provided strong support for designing more efficient, robust, and adaptive audio watermarking algorithms; however, the concealment performance of existing watermarking embedding methods is still limited to a certain extent, causing varying degrees of interference to audio quality, and is susceptible to signal processing attacks, thus affecting their reliability and effectiveness. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a deep learning-based audio watermark embedding and extraction method to solve the problems of poor concealment, poor audio quality, and vulnerability to signal processing attacks in existing watermark embedding methods, thereby affecting their reliability and effectiveness.

[0006] The objective of this invention is mainly achieved through the following technical solutions:

[0007] On one hand, this invention provides a deep learning-based audio watermark embedding method, comprising:

[0008] The audio signal to be processed is subjected to discrete wavelet transform to obtain low-frequency sub-signals and high-frequency sub-signals;

[0009] Based on a pre-trained watermark embedding model, the watermark information to be embedded is embedded into a high-frequency sub-signal to obtain a high-frequency sub-signal with embedded watermark. After the watermark embedding model performs attention adjustment on the received high-frequency sub-signal, it splices and embeds the watermark information in the channel dimension, and implicitly encodes the watermark information into the time-frequency structure of the high-frequency sub-signal through convolution operations and skip connections.

[0010] The high-frequency sub-signal with embedded watermark and the low-frequency sub-signal are subjected to inverse discrete wavelet transform to obtain the watermarked audio signal.

[0011] Furthermore, the watermark embedding model is a model based on the U-net network, and the multi-channel attention module is set after the first three downsampling units of the U-net network to perform attention adjustment on the received signal;

[0012] After concatenating the watermark information with the audio features obtained from the last layer downsampling unit of the U-net network, the watermarked high-frequency sub-signal is encoded by skipping connections between multiple upsampling units in the decoder of the U-net network and the corresponding downsampling units in the encoder, and then encoding the high-frequency sub-signal with the embedded watermark.

[0013] Furthermore, the multi-channel attention module adjusts the attention of the received signal using the following method:

[0014] The multi-channel attention module divides the received downsampled signal X into n sub-feature groups {X1, X2, ... X} in the channel dimension. i ...,X n};

[0015] For each sub-feature group X i Perform average pooling operations along the W and H directions respectively to obtain W feature map data and H feature map data;

[0016] The W feature mapping data is transposed and then concatenated with the H feature mapping data. The concatenated feature mapping data is then compressed in dimension by a 1×1 convolutional layer.

[0017] After the n feature mapping data with dimensionality compression are processed by the activation function, channel merging is performed.

[0018] The convolutional layer restores the feature map data after channel merging to the original channels, and the attention weight A is obtained by the Sigmoid activation function.

[0019] Multiply the downsampled signal X by the corresponding attention weight A to obtain the attention-adjusted feature X′.

[0020] Furthermore, the H feature mapping data is represented as follows:

[0021]

[0022] The W feature mapping data is represented as follows:

[0023]

[0024] Among them, F i,H For the H feature mapping data corresponding to the i-th sub-feature group, F i,W Let W be the feature mapping data corresponding to the i-th sub-feature group, and X be the feature mapping data. i Let i be the i-th sub-feature group.

[0025] Furthermore, the discrete wavelet transform of the audio signal to be processed includes:

[0026] S1: Convert the audio signal from the time domain to the frequency domain;

[0027] S2: Convolve the frequency-domain transformed audio signal using a low-pass filter and a high-pass filter of the db4 wavelet, respectively, to generate approximate coefficients c. j+1 [k] and detail factor d j+1 [k];

[0028] S3: Perform a double downsampling on the approximation coefficients, expressed as:

[0029]

[0030] S4: Use the approximation coefficients obtained from downsampling as a new audio signal, and repeat steps S2-S3 until the preset number of decomposition layers is reached;

[0031] S5: Reconstruct the signals from the final approximation coefficients and detail coefficients to obtain the low-frequency sub-signals corresponding to the approximation coefficients and the high-frequency sub-signals corresponding to the detail coefficients.

[0032] Furthermore, before embedding the watermark information to be embedded into the high-frequency sub-signal, the process further includes:

[0033] The watermark information is preprocessed into a byte array and then BCH encoded to generate an error correction code.

[0034] The watermark information is concatenated with the error correction code, and a scaling factor is set for upsampling to expand the concatenated watermark information into a watermark feature with the same size as the high-frequency sub-signal, which is then used for watermark embedding.

[0035] Furthermore, the audio signal is the audio signal corresponding to the speech segment after the audio file to be processed is determined by speech segment and non-speech segment;

[0036] After embedding a watermark into the audio signal corresponding to the speech segment, the process further includes splicing the watermarked speech segment and the non-speech segment together to obtain a watermarked audio file.

[0037] On the other hand, a deep learning-based audio watermark extraction method is also disclosed, including:

[0038] S1: The audio file with the watermark embedded by the aforementioned method is preprocessed to obtain a speech segment;

[0039] S2: Perform discrete wavelet transform on the audio signal of the speech segment to obtain low-frequency sub-signals and high-frequency sub-signals;

[0040] S3: The high-frequency sub-signal is decoded by a pre-trained audio watermark extraction network, and the decoding result is binarized to obtain the predicted watermark information;

[0041] S4: The predicted watermark information is corrected by BHC decoding to obtain the final watermark information.

[0042] Furthermore, the audio watermark extraction network includes four downsampling units arranged in sequence, each downsampling unit including a convolutional layer, a pooling layer, a normalization layer and a ReLU activation function arranged in sequence;

[0043] The last downsampling unit also includes a fully connected layer and a Sigmoid activation function; after passing through the Sigmoid activation function, a floating-point value between 0 and 1 is obtained, and the floating-point value is binarized to obtain the predicted watermark information.

[0044] Furthermore, the audio watermark extraction network is trained on a ripped audio dataset after the watermark has been embedded using the aforementioned audio watermark embedding method.

[0045] The data in the audio dataset is labeled with binary original embedded watermark information; during training, the mean square error of the original embedded watermark information and the corresponding decoded information obtained by decoding through the audio watermark extraction network is used as the decoding loss to optimize the training of the audio watermark extraction network.

[0046] The beneficial effects of this technical solution are:

[0047] 1. This invention decomposes the original audio into low-frequency and high-frequency sub-signals by performing discrete wavelet transform. Since the low-frequency sub-signals usually contain the main content of the audio, while the high-frequency sub-signals contain more details and redundant information, they are suitable for embedding watermark information. Furthermore, based on the relationship between human hearing and sound frequency perception, fixing the frequency of the embedded watermark signal in the high-frequency range can enhance the concealment and anti-attack capabilities of the watermark. While maintaining the concealment of the watermark, it ensures that the accuracy of watermark extraction reaches more than 90% in medium to large conference room scenarios.

[0048] 2. This invention designs a multi-channel attention mechanism that effectively captures global and local features in audio data by dividing the input feature mapping data into multiple sub-feature groups and performing independent feature learning on each sub-feature group. This mechanism can adaptively emphasize important parts of the audio signal during the encoding process and overcome the limitations of single-scale attention mechanisms, better preserving the quality and details of the audio.

[0049] 3. The audio watermark embedding method of the present invention performs speech activity detection on the original audio, segments the speech segment and the non-speech segment, embeds the watermark within the speech segment, and then concatenates the speech segment and the non-speech segment according to the time sequence to restore the original speech length. This effectively improves the transparency of the watermark while ensuring that the watermark information is only embedded in the part containing the actual speech content, thereby reducing the impact on background noise or silent parts. Attached Figure Description

[0050] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0051] Figure 1 This is a flowchart of the deep learning-based audio watermark embedding method according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the watermark embedding model structure according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the audio watermark embedding and extraction system according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the overall process of audio watermark embedding and extraction according to an embodiment of the present invention. Detailed Implementation

[0055] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0056] One embodiment of the present invention provides an audio watermark embedding method based on deep learning, such as... Figure 1 As shown, it includes:

[0057] S1: Perform discrete wavelet transform on the audio signal to be processed to obtain low-frequency sub-signals and high-frequency sub-signals;

[0058] Specifically, in this embodiment, the audio signal used for watermark embedding is the audio signal corresponding to the speech segment in the audio file to be processed; preferably, a speech activity detection algorithm can be used to determine whether the audio file contains speech segments or non-speech segments, dividing the audio into speech segments and non-speech segments, specifically including:

[0059] The audio file to be processed is read and segmented into frames. The original audio sampling rate used in this invention is 44.1kHz, single channel, 16-bit bit depth, and 15 seconds.

[0060] Each frame is classified using a speech activity detection algorithm to determine whether it contains speech. Based on the detection results, the audio is segmented into speech segments and non-speech segments, and the segmented audio files are saved. This embodiment uses the WebRTC speech activity detection algorithm, which works by dividing the input spectrum into six sub-bands based on the spectral range of human voice, calculating the energy of each of the six sub-bands, and then using the probability density function of a Gaussian model to obtain a log-likelihood ratio function. The log-likelihood ratio is divided into global and local components. The global component is the weighted sum of the six sub-bands, while the local component refers to each sub-band. Therefore, speech decision-making first judges the sub-bands, and sometimes judges the global component if either component passes the test. If either component passes the test, the segment is judged as a speech segment containing speech; otherwise, it is a non-speech segment.

[0061] Furthermore, the audio signal identified as a speech segment undergoes discrete wavelet transform, including:

[0062] S101: Convert the audio signal from the time domain to the frequency domain;

[0063] S102: Convolve the frequency-domain transformed audio signal using a low-pass filter and a high-pass filter of the db4 wavelet, respectively, to generate approximate coefficients c. j+1 [k] and detail factor d j+1 [k];

[0064] S103: Perform a double downsampling on the approximation coefficients;

[0065] S104: Use the approximate coefficients obtained by downsampling as a new audio signal, and repeat steps S102-S103 until the preset number of decomposition layers is reached;

[0066] S105: Reconstruct the signals from the final approximation coefficients and detail coefficients to obtain the low-frequency sub-signals corresponding to the approximation coefficients and the high-frequency sub-signals corresponding to the detail coefficients.

[0067] More specifically, the audio signal read from the audio file of the speech segment is a one-dimensional array, which needs to be converted from the time domain to the frequency domain. In this embodiment, the speech segment is extracted into a 5-second speech fragment, denoted as a(n), where n is the time index. Discrete wavelet transform is used to convert the signal to the frequency domain. The Daubechies wavelet (db4) is selected. This wavelet has a filter length of 8 and 4 vanishing moments, which can effectively capture the polynomial components in the signal and has a good smoothing effect for low-order polynomials such as linear and quadratic functions.

[0068] The signal was convolved using a db4 low-pass filter and a high-pass filter to generate approximate coefficients c. j+1 [k] and detail factor d j+1 [k] is represented as:

[0069]

[0070] Where h(n-2k) represents the impulse response of the low-pass filter and g(n-2k) represents the impulse response of the high-pass filter.

[0071] The results obtained from the low-pass and high-pass filters are downsampled by a factor of two, meaning that only data from every other sample point is retained to reduce the amount of data, and processing continues in the next layer, as shown below:

[0072]

[0073] The approximation coefficients obtained in the previous step are regarded as new signals. Convolution and downsampling are repeated until the required number of decomposition layers is reached. In this embodiment, the number of decomposition layers is set to 5. Finally, the signal is reconstructed based on the obtained approximation coefficients and detail coefficients to obtain the corresponding low-frequency sub-signals and high-frequency sub-signals.

[0074] S2: Based on a pre-trained watermark embedding model, the watermark information to be embedded is embedded into a high-frequency sub-signal to obtain a high-frequency sub-signal with embedded watermark; the watermark embedding model adjusts the attention of the received high-frequency sub-signal through a multi-channel attention module, then splices and embeds the watermark information in the channel dimension, and implicitly encodes the watermark information into the time-frequency structure of the high-frequency sub-signal through convolution operations and skip connections.

[0075] The watermark embedding model in this embodiment is based on the U-net network. The multi-channel attention module is set after the first three downsampling units of the U-net network to adjust the attention of the received high-frequency sub-signals.

[0076] After concatenating the watermark information with the audio features obtained from the last layer downsampling unit of the U-net network, multiple upsampling units in the decoder of the U-net network are connected to the corresponding downsampling units in the encoder for multiple upsampling operations, and the high-frequency sub-signal with the embedded watermark is encoded.

[0077] Specifically, the high-frequency sub-signal obtained in step S1 is used as the input to the U-net network, and watermark information is added for encoding. This embodiment's watermark embedding model incorporates the MCAM multi-channel attention mechanism based on the U-net network structure. The network structure framework is as follows: Figure 2 As shown, the U-net model consists of an encoder and a decoder. The encoder contains four downsampling units, and multi-channel attention modules are added after the first three downsampling units. This multi-channel attention mechanism better preserves key information in the audio signal. During the downsampling stage, some detailed information is lost in the audio signal; this attention mechanism can adaptively adjust the importance of different channels and positions, effectively improving the loss of audio information and reducing audio quality issues. Skip connections are used to concatenate the features from the corresponding encoder parts to the decoder, facilitating feature reconstruction and making the watermark embedding intensity distribution more natural in the time-frequency domain.

[0078] The special multi-channel attention module is designed as follows:

[0079] Input: The downsampled feature map data is used as the input to the multi-channel attention module. The input feature map data is X∈R. C×H×W Where C is the number of channels, H is the height, and W is the width.

[0080] Multi-scale channels: The input feature map X is divided into 16 sub-feature groups {X1, X2, ..., X...} 16}, each sub-feature group Each group undergoes feature learning separately.

[0081] Mean pooling: For each sub-feature group X i Perform average pooling operations along the W and H directions respectively to obtain W feature map data and H feature map data;

[0082] Where: H feature mapping data F i,H Represented as:

[0083]

[0084] W feature mapping data F i,W Represented as

[0085]

[0086] Among them, F i,H For the H feature mapping data corresponding to the i-th sub-feature group, F i,W Let W be the feature mapping data corresponding to the i-th sub-feature group, and X be the feature mapping data. i Let i be the i-th sub-feature group.

[0087] Concatenation and Convolution: The W feature map data is transposed and then concatenated with the H feature map data to obtain... Then, the dimensions are compressed using a 1×1 convolutional layer to obtain F′. i =Conv 1×1 (F i ).

[0088] Normalization and ReLU activation function: Batch normalization and the non-linear activation function ReLU are applied to further process the feature map data to obtain F". i .

[0089] Convolution and Sigmoid activation function: This converts 16 sub-feature groups {F"1,F"2,...,F"} into a single feature set. 16} Perform channel merging, and then use a convolutional layer to convert it back to the original channels to obtain F′∈R C×1×1 Applying the Sigmoid activation function to the merged features yields the attention weights A∈R. C×1×1 This restricts its value range to (0,1).

[0090] Reweighting: Multiply the original input feature X by the attention weight A to obtain the reweighted feature X′∈R. C×H×W .

[0091] Output: The final result is the feature adjusted by the attention mechanism, represented as:

[0092] X′=X×σ(Conv 1×1 (Concat(AvgPool(X,H),AvgPool(X,W))));

[0093] Where X′ is the attention-adjusted feature representation; σ is the Sigmoid activation function;

[0094] Concat() represents a channel merging operation.

[0095] This attention mechanism employs a multi-channel strategy, which overcomes the limitations of single-channel attention, connects information in the audio over a wider range, and enhances the model's understanding of global and local features.

[0096] To improve the imperceptibility of watermark information, the human ear can hear sound frequencies from 20 Hz to 20 kHz, while the human ear's sensitivity range is from 1 kHz to 4 kHz. Within the high-frequency range, the higher the frequency, the lower the human ear's sensitivity to sound. Therefore, this invention chooses to embed the watermark within a fixed high-frequency range. Watermark information is added after the last downsampling unit; specifically, the watermark information is preprocessed into a byte array and BCH encoded to generate an error correction code; the watermark information and the error correction code are concatenated, and a scaling factor is set for upsampling to expand the concatenated watermark information into a watermark feature with the same magnitude as the high-frequency sub-signal; and the audio feature and the watermark feature are concatenated along the channel dimension.

[0097] Furthermore, the decoder also includes four upsampling units. The upsampling and downsampling modules in the model structure employ the same number of convolutional operations, and a skip connection structure connects the downsampling and upsampling units, allowing the features extracted by the downsampling units to be directly passed to the upsampling units, resulting in more accurate feature refinement in the U-net network. The input to the upsampling unit is the concatenation of the output of the last downsampling unit along the channel dimension with the binary watermark information. For the two upsampling units at the beginning, a dropout layer with a 50% probability is applied after activation to prevent overfitting and increase the learning's robustness to attacks. During training, an encoding loss function, MSE, is introduced into the embedding network to calculate the audio signal loss before and after watermark embedding.

[0098] The watermark information is introduced through channel concatenation in the last layer of the encoder, and then encoded and integrated into the high-frequency sub-signal during the upsampling process of the decoder. Furthermore, the output of each upsampling unit in the decoder is skipped and connected to the corresponding layer's output in the encoder, which is adjusted through a multi-channel attention mechanism. This implicitly encodes the watermark position into the time-frequency structure of the high-frequency sub-signal optimized by the multi-channel attention mechanism, optimizing the watermark embedding position, strength, and compatibility with the host audio. Ultimately, this achieves a covert and robust embedding effect. This solves the problems of traditional U-net networks, such as the loss of key information during downsampling, the lack of precise location and dynamic adjustment capabilities for key regions, and dependence on local features, leading to insufficient robustness, perceptible distortion, and weakened anti-attack capabilities, thus affecting the overall performance of watermarking technology.

[0099] S3: Perform inverse discrete wavelet transform on the high-frequency sub-signal with embedded watermark and the low-frequency sub-signal to obtain the watermarked audio signal.

[0100] Specifically, the inverse discrete wavelet transform is a signal reconstructed from wavelet coefficients obtained by the discrete wavelet transform, that is, the inverse discrete wavelet transform is performed based on the approximation coefficients and detail coefficients, expressed as:

[0101]

[0102] Where s is the signal reconstructed by inverse discrete wavelet transform, and cA j The approximate coefficients obtained after decomposition at the j-th level represent the smoothed portion of the signal at that level, cD j denoted as the detail coefficients obtained after decomposition at level j, representing the high-frequency components of the signal at that level; J is the number of decomposition levels; k is the index of the filter coefficients; h(k-2n) is the impulse response of the reconstructed low-pass filter; g(k-2n) is the impulse response of the reconstructed high-pass filter.

[0103] For example, the watermark embedding model in this embodiment is trained using supervised learning. The training dataset for embedding is human speech audio, collected from the internet and recorded in real-world locations. The distribution of the watermark embedding model dataset is shown in Table 1. The data is labeled with binary watermark information. During training, an encoding loss is introduced, which is calculated by taking the mean square error of the high-frequency sub-signals before and after watermark embedding to evaluate the performance of the watermark embedding model and ensure that the embedded watermark information does not significantly affect the audio quality.

[0104] Table 1. Watermark Embedding Model Dataset

[0105]

[0106] A second embodiment of the present invention further includes a deep learning-based audio watermark extraction method, comprising:

[0107] S1: The audio file with the watermark embedded by the aforementioned method is preprocessed to obtain a speech segment;

[0108] S2: Perform discrete wavelet transform on the audio signal of the speech segment to obtain low-frequency sub-signals and high-frequency sub-signals;

[0109] S3: The high-frequency sub-signal is decoded by a pre-trained audio watermark extraction network, and the decoding result is binarized to obtain the predicted watermark information;

[0110] S4: The predicted watermark information is corrected by BHC decoding to obtain the final watermark information.

[0111] Furthermore, the audio watermark extraction network includes four downsampling units arranged in sequence, each downsampling unit including a convolutional layer, a pooling layer, a normalization layer and a ReLU activation function arranged in sequence;

[0112] The last downsampling unit also includes a fully connected layer and a Sigmoid activation function; after passing through the Sigmoid activation function, a floating-point value between 0 and 1 is obtained, and the floating-point value is binarized to obtain the predicted watermark information.

[0113] The audio watermark extraction network is trained on a ripped audio dataset in which audio watermarks are embedded using the aforementioned method.

[0114] The data in the audio dataset is labeled with binary original embedded watermark information; during training, the mean square error of the original embedded watermark information and the corresponding decoded information obtained by decoding through the audio watermark extraction network is used as the decoding loss to optimize the training of the audio watermark extraction network.

[0115] For example, when extracting audio watermarks, the watermarked audio file is first preprocessed, including converting the audio format to .wav, the sampling rate to 44.1kHz, mono, and the sample width to 16 bits, and then extracting the speech segments from the original audio.

[0116] Discrete wavelet transform (DWT) is performed on the audio signal of a speech segment. DWT converts the audio signal from the time domain to the frequency domain. Low-pass and high-pass filters are used to convolve the signal, yielding approximation and detail coefficients. The results from both filters are downsampled by a factor of two, and the resulting approximation coefficients are then treated as a new signal. This process is recursively repeated until the desired decomposition level is reached. Finally, the wavelet coefficients are used to reconstruct the signal, yielding low-frequency and high-frequency sub-signals.

[0117] The high-frequency sub-signal is decoded using an audio watermark extraction network, and the decoding result is binarized to obtain the watermark information. Specifically, the audio watermark extraction network structure in this invention consists of four downsampling units, each composed of a convolutional layer, a pooling layer, a normalization layer, and a ReLU activation function. The stride in the convolutional layer is 2, and the convolutional kernel is 5×5. The high-frequency sub-signal with the watermark is input into this network structure. The features obtained after passing through the first three downsampling units are fed by the output of the last unit to a fully connected layer and a sigmoid activation function to predict the watermark information. The output value is a floating-point value between 0 and 1, and then the output floating-point value is binarized.

[0118] During the training of the audio watermark extraction network, a watermark loss function MSE is introduced to calculate the mean square error between the original watermark and the extracted watermark.

[0119] The formula for the ReLU function is as follows:

[0120] f(x) = max(0, x);

[0121] Using the ReLU function can overcome the vanishing gradient problem during training and speed up the training process.

[0122] The formula for the Sigmoid function is as follows:

[0123]

[0124] The formula for the loss function is as follows:

[0125]

[0126] Where Loss represents the size of the loss function, x i The tensor representing watermark extraction, y i This refers to the tensor of the original watermark.

[0127] This invention performs BCH encoding on the watermark information before embedding it. BCH is a type of encoding that can correct multiple random errors and is used in the binary field, which can improve the robustness of the watermark information after it suffers various distortions.

[0128] In the BCH encoding process, a suitable generator polynomial y(x) is first selected. The generator polynomial is the core of the BCH code and is composed of the product of minimal polynomials. In this invention, BCH_POLYNOMIAL = 137, and the corresponding generator polynomial is y(x) = x. 7 +x 3 +1. Convert the byte sequence into a polynomial, perform polynomial division, and the remainder is the error correction code.

[0129] In the BHC decoding process, the coefficients of the error position polynomial are first calculated using the received codeword. Then, the location of the error is determined by solving the root of the error position polynomial. Finally, the bit value at the error position is flipped to correct the error, and the extracted watermark information is obtained.

[0130] The audio watermark extraction network was trained using supervised learning. The training dataset consisted of ripped audio recordings of human voices with watermarks embedded using the aforementioned method. The data source was audio recordings with watermarks made using a voice recorder. The distribution of the watermark extraction model dataset is shown in Table 2. The data is labeled with the original embedded watermark information in binary format. A decoding loss was introduced during training. The decoding loss was calculated as the mean square error between the original watermark and the decoded information obtained by the extraction network after ripping, to evaluate the performance of the audio watermark extraction network and compare the extracted watermark information with the original watermark information.

[0131] Table 2 Dataset for Watermark Extraction Model

[0132]

[0133] The above method can encode and accurately extract watermark information from rigging attacks, and the audio watermark extraction network has good robustness against rigging.

[0134] Based on the aforementioned audio watermark embedding and extraction methods, this invention also provides a deep learning-based audio watermarking system, which consists of five core modules: a configuration management module, an audio preprocessing module, a watermark embedding module, a watermark extraction module, and a watermark evaluation module. A schematic diagram of the audio watermarking system is shown below. Figure 3 As shown, the functions of each module are as follows:

[0135] Configuration Management Module: This module is responsible for configuring the parameters of the entire system, covering key settings such as watermark embedding strength, supported audio formats, deep learning model paths, and input / output file paths. It provides a user-friendly interface, allowing users to easily modify configuration parameters and conveniently save and load configuration files, thus ensuring the system's flexibility and customizability.

[0136] Audio preprocessing module: This module primarily performs three tasks: format conversion, noise reduction, and normalization. Format conversion ensures that the input audio meets system requirements, supporting conversion of various audio formats and sampling rates. Noise reduction aims to remove or reduce noise in the audio to improve the effectiveness of watermark embedding and extraction. Normalization adjusts the audio signal to a suitable amplitude range, laying the foundation for subsequent processing.

[0137] Watermark Embedding Module: This module generates a watermark signal based on user input or predefined information. It employs the deep learning-based watermark embedding method mentioned in this invention, performing discrete wavelet transform on the preprocessed audio signal and encoding the high-frequency signal and watermark using an encoder model. The encoded high-frequency signal is then subjected to inverse discrete wavelet transform with the low-frequency signal to obtain the watermarked audio. Furthermore, this module optimizes the embedding process using an improved U-Net model to enhance the robustness and imperceptibility of the watermark.

[0138] Watermark Extraction Module: This module is responsible for detecting and extracting the watermark signal from the watermarked audio. It also employs the deep learning-based watermark extraction method mentioned in this invention, performing discrete wavelet transform on the audio signal to be detected and using a decoder to decode the high-frequency signal. The decoded result is then binarized and processed with error correction codes to obtain the watermark information. This process ensures accurate watermark extraction and recognition.

[0139] Watermark Evaluation Module: This module primarily evaluates the impact of watermark embedding on audio quality and the robustness of the watermark. It uses subjective evaluation (MOS) and objective evaluation (SNR) to ensure the watermark's imperceptibility. The robustness evaluation tests the watermark's survival ability under conditions such as ripping attacks and cropping. Furthermore, this module also evaluates the maximum capacity of watermark information that the system can embed and provides a quantitative assessment of the overall system performance, including key indicators such as processing speed and resource consumption.

[0140] In summary, the overall process of the deep learning-based audio watermark embedding and extraction method of the present invention is as follows: Figure 4 As shown, during watermark embedding, the original audio is first subjected to discrete wavelet transform, decomposing it into low-frequency and high-frequency sub-signals. Since low-frequency sub-signals typically contain the main content of the audio, while high-frequency sub-signals contain more details and redundant information, they are suitable for watermark embedding. Furthermore, based on the relationship between human hearing and sound frequency perception, fixing the embedded watermark signal frequency within the high-frequency range enhances the watermark's concealment and resistance to attacks. While maintaining watermark concealment, the accuracy of watermark extraction in medium to large conference room scenarios reaches over 90%. This invention also designs a multi-channel attention mechanism. By dividing the input feature mapping data into multiple sub-feature groups and performing independent feature learning on each sub-feature group, it can effectively capture global and local features in the audio data. This mechanism can adaptively emphasize important parts of the audio signal during encoding and overcome the limitations of single-scale attention mechanisms, better preserving audio quality and detail.

[0141] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0142] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-based audio watermark embedding method, characterized in that, include: The audio signal to be processed is subjected to discrete wavelet transform to obtain low-frequency sub-signals and high-frequency sub-signals; Based on a pre-trained watermark embedding model, the watermark information to be embedded is embedded into a high-frequency sub-signal to obtain a high-frequency sub-signal with embedded watermark. The watermark embedding model adjusts the attention of the received high-frequency sub-signal through a multi-channel attention module, then splices and embeds the watermark information in the channel dimension, and implicitly encodes the watermark information into the time-frequency structure of the high-frequency sub-signal through convolution operations and skip connections. The high-frequency sub-signal with embedded watermark and the low-frequency sub-signal are subjected to inverse discrete wavelet transform to obtain the watermarked audio signal.

2. The deep learning-based audio watermark embedding method according to claim 1, characterized in that, The watermark embedding model is based on the U-net network. There are three multi-channel attention modules, which are respectively set after the first three downsampling units of the U-net network to adjust the attention of the received signal. After concatenating the watermark information with the audio features obtained from the last layer downsampling unit of the U-net network, the watermarked high-frequency sub-signal is encoded by skipping connections between multiple upsampling units in the decoder of the U-net network and the corresponding downsampling units in the encoder, and then encoding the high-frequency sub-signal with the embedded watermark.

3. The deep learning-based audio watermark embedding method according to claim 2, characterized in that, The multi-channel attention module adjusts the attention of the received high-frequency sub-signals using the following method: The multi-channel attention module divides the received downsampled signal X into n sub-feature groups {X1, X2, ... X} in the channel dimension. i ...,X n }; For each sub-feature group X i Perform average pooling operations along the W and H directions respectively to obtain W feature map data and H feature map data; The W feature mapping data is transposed and then concatenated with the H feature mapping data. The concatenated feature mapping data is then compressed in dimension by a 1×1 convolutional layer. After the n feature mapping data with dimensionality compression are processed by the activation function, channel merging is performed. The convolutional layer restores the feature map data after channel merging to the original channels, and the attention weight A is obtained by the Sigmoid activation function. Multiply the downsampled signal X by the corresponding attention weight A to obtain the attention-adjusted feature X′.

4. The deep learning-based audio watermark embedding method according to claim 3, characterized in that, The H feature mapping data is represented as follows: The W feature mapping data is represented as follows: Among them, F i,H For the H feature mapping data corresponding to the i-th sub-feature group, F i,W Let W be the feature mapping data corresponding to the i-th sub-feature group, and X be the feature mapping data. i Let i be the i-th sub-feature group.

5. The deep learning-based audio watermark embedding method according to claim 1, characterized in that, The discrete wavelet transform of the audio signal to be processed includes: S1: Convert the audio signal from the time domain to the frequency domain; S2: Convolve the frequency-domain transformed audio signal using a low-pass filter and a high-pass filter of the db4 wavelet, respectively, to generate approximate coefficients c. j+1 [k] and detail factor d j+1 [k]; S3: Perform a double downsampling on the approximation coefficients and detail coefficients, expressed as: S4: Use the approximation coefficients obtained from downsampling as a new audio signal, and repeat steps S2-S3 until the preset number of decomposition layers is reached; S5: Reconstruct the signals from the final approximation coefficients and detail coefficients to obtain the low-frequency sub-signals corresponding to the approximation coefficients and the high-frequency sub-signals corresponding to the detail coefficients.

6. The deep learning-based audio watermark embedding method according to claim 2, characterized in that, Before embedding the watermark information to be embedded into the high-frequency sub-signal, the method further includes: The watermark information is preprocessed into a byte array and then BCH encoded to generate an error correction code; The watermark information is concatenated with the error correction code, and a scaling factor is set for upsampling to expand the concatenated watermark information into a watermark feature with the same size as the high-frequency sub-signal, which is then used for watermark embedding.

7. The deep learning-based audio watermark embedding method according to claim 1, characterized in that, The audio signal is the audio signal corresponding to the speech segment after the audio file to be processed is determined by speech segment and non-speech segment; After embedding a watermark into the audio signal corresponding to the speech segment, the process further includes splicing the watermarked speech segment and the non-speech segment together to obtain a watermarked audio file.

8. A deep learning-based audio watermark extraction method, characterized in that, include: S1: Preprocess the audio file with the watermark embedded by the method described in any one of claims 1-7 to obtain a speech segment; S2: Perform discrete wavelet transform on the audio signal of the speech segment to obtain low-frequency sub-signals and high-frequency sub-signals; S3: The high-frequency sub-signal is decoded by a pre-trained audio watermark extraction network, and the decoding result is binarized to obtain the predicted watermark information; S4: The predicted watermark information is corrected by BHC decoding to obtain the final watermark information.

9. The deep learning-based audio watermark extraction method according to claim 8, characterized in that, The audio watermark extraction network includes four downsampling units arranged in sequence. Each downsampling unit includes a convolutional layer, a pooling layer, a normalization layer, and a ReLU activation function arranged in sequence. The last downsampling unit also includes a fully connected layer and a Sigmoid activation function; after passing through the Sigmoid activation function, a floating-point value between 0 and 1 is obtained, and the floating-point value is binarized to obtain the predicted watermark information.

10. The deep learning-based audio watermark extraction method according to claim 8, characterized in that, The audio watermark extraction network is trained on a ripped audio dataset after watermarking is embedded using the audio watermark embedding method described in any one of claims 1-7. The data in the audio dataset is labeled with binary original embedded watermark information; During training, the mean square error between the original embedded watermark information and the corresponding decoded information obtained by decoding through the audio watermark extraction network is used as the decoding loss to optimize the training of the audio watermark extraction network.

Images